blocks|key|471269|text|下面是一个下载图片(使用requests库)并上传到s3，而不写入本地文件的示例：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|471270|import+boto
from+boto.s3.key+import+Key
import+requests

#setup+the+bucket
c+=+boto.connect_s3(your_s3_key,+your_s3_key_secret)
b+=+c.get_bucket(bucket,+validate=False)

#download+the+file
url+=+"http://en.wikipedia.org/static/images/project-logos/enwiki.png"
r+=+requests.get(url)
if+r.status_code+==+200:
++++#upload+the+file
++++k+=+Key(b)
++++k.key+=+"image1.png"
++++k.content_type+=+r.headers['content-type']
++++k.set_contents_from_string(r.content)|code-block|syntax|javascript|471271|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

Here is an example downloading an image (using requests library) and uploading it to s3, without writing to a local file:

<pre><code>import boto
from boto.s3.key import Key
import requests

#setup the bucket
c = boto.connect_s3(your_s3_key, your_s3_key_secret)
b = c.get_bucket(bucket, validate=False)

#download the file
url = "http://en.wikipedia.org/static/images/project-logos/enwiki.png"
r = requests.get(url)
if r.status_code == 200:
 #upload the file
 k = Key(b)
 k.key = "image1.png"
 k.content_type = r.headers['content-type']
 k.set_contents_from_string(r.content)
</code></pre>

blocks|key|471218|text|您可以使用Python标准库中的BytesIO。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|471219|from+io+import+BytesIO
bytesIO+=+BytesIO()
bytesIO.write('whee')
bytesIO.seek(0)
s3_file.set_contents_from_file(bytesIO)|code-block|syntax|javascript|471220|entityMap|0|LINK|mutability|MUTABLE|url|https://docs.python.org/2/library/io.html#io.BytesIO^0|G|7|0|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@$A|R|B|S|1|T]]|C|$]]|$1|D|3|E|5|F|7|U|8|@]|9|@]|C|$G|H]]|$1|I|3|-4|5|6|7|V|8|@]|9|@]|C|$]]]|J|$K|$5|L|M|N|C|$O|P]]]]

You could use <a href="https://docs.python.org/2/library/io.html#io.BytesIO" rel="noreferrer">BytesIO</a> from the Python standard library.

<pre><code>from io import BytesIO
bytesIO = BytesIO()
bytesIO.write('whee')
bytesIO.seek(0)
s3_file.set_contents_from_file(bytesIO)
</code></pre>

blocks|key|471188|text|我假设您使用的是boto。boto的Bucket.set_contents_from_file()将接受StringIO对象，并且您编写的任何用于将数据写入文件的代码都应易于修改以写入StringIO对象。或者，如果您生成一个字符串，则可以使用set_contents_from_string()。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|471189|entityMap^0|8|4|D|4|I|V|1G|8|2K|8|3E|Q|0^^$0|@$1|2|3|4|5|6|7|H|8|@$9|I|A|J|B|C]|$9|K|A|L|B|C]|$9|M|A|N|B|C]|$9|O|A|P|B|C]|$9|Q|A|R|B|C]|$9|S|A|T|B|C]]|D|@]|E|$]]|$1|F|3|-4|5|6|7|U|8|@]|D|@]|E|$]]]|G|$]]

I assume you're using <code>boto</code>. <code>boto</code>'s <code>Bucket.set_contents_from_file()</code> will accept a <code>StringIO</code> object, and any code you have written to write data to a file should be easily adaptable to write to a <code>StringIO</code> object. Or if you generate a string, you can use <code>set_contents_from_string()</code>.

blocks|key|254368|text|def+upload_to_s3(url,+**kwargs):
++++'''
++++:param+url:+url+of+image+which+have+to+upload+or+resize+to+upload
++++:return:+url+of+image+stored+on+aws+s3+bucket
++++'''

++++r+=+requests.get(url)
++++if+r.status_code+==+200:
++++++++#+credentials+stored+in+settings+AWS_ACCESS_KEY_ID+and+AWS_SECRET_ACCESS_KEY
++++++++conn+=+boto.connect_s3(AWS_ACCESS_KEY_ID,+AWS_SECRET_ACCESS_KEY,+host=AWS_HOST)

++++++++#+Connect+to+bucket+and+create+key
++++++++b+=+conn.get_bucket(AWS_Bucket_Name)
++++++++k+=+b.new_key("{folder_name}/{filename}".format(**kwargs))

++++++++k.set_contents_from_string(r.content,+replace=True,
+++++++++++++++++++++++++++++++++++headers={'Content-Type':+'application/%25s'+%25+(FILE_FORMAT)},
+++++++++++++++++++++++++++++++++++policy='authenticated-read',
+++++++++++++++++++++++++++++++++++reduced_redundancy=True)

++++++++#+TODO+Change+AWS_EXPIRY
++++++++return+k.generate_url(expires_in=AWS_EXPIRY,+force_http=True)|type|code-block|depth|inlineStyleRanges|entityRanges|data|syntax|javascript|254369|unstyled|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|G|8|@]|9|@]|A|$B|C]]|$1|D|3|-4|5|E|7|H|8|@]|9|@]|A|$]]]|F|$]]

<pre><code>def upload_to_s3(url, **kwargs):
 '''
 :param url: url of image which have to upload or resize to upload
 :return: url of image stored on aws s3 bucket
 '''

 r = requests.get(url)
 if r.status_code == 200:
 # credentials stored in settings AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
 conn = boto.connect_s3(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, host=AWS_HOST)

 # Connect to bucket and create key
 b = conn.get_bucket(AWS_Bucket_Name)
 k = b.new_key("{folder_name}/{filename}".format(**kwargs))

 k.set_contents_from_string(r.content, replace=True,
 headers={'Content-Type': 'application/%s' % (FILE_FORMAT)},
 policy='authenticated-read',
 reduced_redundancy=True)

 # TODO Change AWS_EXPIRY
 return k.generate_url(expires_in=AWS_EXPIRY, force_http=True)
</code></pre>

blocks|key|254418|text|我有一个dict对象，我想将它作为json文件存储在S3上，而不是创建本地文件。下面的代码适用于我：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|254419|from+smart_open+import+smart_open

with+smart_open('s3://access-key:secret-key@bucket-name/file.json',+'wb')+as+fout:
++++fout.write(json.dumps(dict_object).encode('utf8'))|code-block|syntax|javascript|254420|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

I had a dict object which I wanted to store as a json file on S3, without creating a local file. The below code worked for me:

<pre><code>from smart_open import smart_open

with smart_open('s3://access-key:secret-key@bucket-name/file.json', 'wb') as fout:
 fout.write(json.dumps(dict_object).encode('utf8'))
</code></pre>

blocks|key|471429|text|在boto3中，有一种简单的方法可以上传文件内容，而无需使用以下代码创建本地文件。我已经为boto3修改了JimJty示例代码|type|unstyled|depth|inlineStyleRanges|entityRanges|data|471430|import+boto3
from+botocore.retries+import+bucket
import+requests
from+io+import+BytesIO
#+set+the+values
aws_access_key_id=""
aws_secret_access_key=""
region_name=""
bucket=""
key=""

session+=+boto3.session.Session(aws_access_key_id=aws_access_key_id,aws_secret_access_key=aws_secret_access_key,+region_name=region_name)
s3_client+=+session.client('s3')
#download+the+file
url+=+"http://en.wikipedia.org/static/images/project-logos/enwiki.png"
r+=+requests.get(url)
if+r.status_code+==+200:++++
++++#convert+content+to+bytes,+since+upload_fileobj+requires+file+like+obj
++++bytesIO+=+BytesIO(bytes(r.content))++++
++++with+bytesIO+as+data:
++++++++s3_client.upload_fileobj(data,+bucket,+key)|code-block|syntax|javascript|471431|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

In boto3, there is a simple way to upload a file content, without creating a local file using following code. I have modified JimJty example code for boto3
<pre><code>import boto3
from botocore.retries import bucket
import requests
from io import BytesIO
# set the values
aws_access_key_id=&quot;&quot;
aws_secret_access_key=&quot;&quot;
region_name=&quot;&quot;
bucket=&quot;&quot;
key=&quot;&quot;

session = boto3.session.Session(aws_access_key_id=aws_access_key_id,aws_secret_access_key=aws_secret_access_key, region_name=region_name)
s3_client = session.client('s3')
#download the file
url = &quot;http://en.wikipedia.org/static/images/project-logos/enwiki.png&quot;
r = requests.get(url)
if r.status_code == 200: 
 #convert content to bytes, since upload_fileobj requires file like obj
 bytesIO = BytesIO(bytes(r.content)) 
 with bytesIO as data:
 s3_client.upload_fileobj(data, bucket, key)
</code></pre>

blocks|key|471339|text|您可以尝试使用smart_open+(https://pypi.org/project/smart_open/)。我使用它的目的就是:直接用S3编写文件。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|471340|entityMap|0|LINK|mutability|MUTABLE|url|https://pypi.org/project/smart_open/^0|7|A|J|10|0|0^^$0|@$1|2|3|4|5|6|7|N|8|@$9|O|A|P|B|C]]|D|@$9|Q|A|R|1|S]]|E|$]]|$1|F|3|-4|5|6|7|T|8|@]|D|@]|E|$]]]|G|$H|$5|I|J|K|E|$L|M]]]]

You can try using <code>smart_open</code> (<a href="https://pypi.org/project/smart_open/" rel="nofollow noreferrer">https://pypi.org/project/smart_open/</a>). I used it exactly for that: writing files directly in S3.

blocks|key|471375|text|鉴于静态加密现在是一种非常理想的数据标准，smart_open不支持此afaik|type|unstyled|depth|inlineStyleRanges|entityRanges|data|471376|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

Given that encryption at rest is a much desired data standard now, smart_open does not support this afaik

blocks|key|254458|text|Boto3更新：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|254459|aws_session+=+boto3.Session('my_access_key_id',+'my_secret_access_key')
s3+=+aws_session.resource('s3')
s3.Bucket('my_bucket').put_object(Key='file_name.txt',+Body=my_file)|code-block|syntax|javascript|254460|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

Update for boto3:

<pre><code>aws_session = boto3.Session('my_access_key_id', 'my_secret_access_key')
s3 = aws_session.resource('s3')
s3.Bucket('my_bucket').put_object(Key='file_name.txt', Body=my_file)
</code></pre>

blocks|key|471405|text|我有一个类似的问题，想知道是否有最终的答案，因为在我下面的代码中，"starwars.json“一直保存在本地，但我只想将每个循环的.json文件推送到S3中，并且没有文件存储在本地。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|471406|for+key,+value+in+star_wars_actors.items():

response+=+requests.get('http:starwarsapi/'+%2B+value)



data+=+response.json()


with+open("starwars.json",+"w%2B")+as+d:
++++json.dump(data,+d,+ensure_ascii=False,+indent=4)



s3.upload_file('starwars.json',+'test-bucket',
+++++++++++++++'%25s/%25s'+%25+('test',+str(key)+%2B+'.json'))|code-block|syntax|javascript|471407|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

I am having a similar issue, was wondering if there was a final answer, because with my code below , the "starwars.json" keeps on saving locally but I just want to push through each looped .json file into S3 and have no file stored locally. 

<pre><code>for key, value in star_wars_actors.items():

response = requests.get('http:starwarsapi/' + value)



data = response.json()


with open("starwars.json", "w+") as d:
 json.dump(data, d, ensure_ascii=False, indent=4)



s3.upload_file('starwars.json', 'test-bucket',
 '%s/%s' % ('test', str(key) + '.json'))
</code></pre>

Is there any feasible way to upload a file which is generated dynamically to amazon s3 directly without first create a local file and then upload to the s3 server? I use python. Thanks

How to upload a file to S3 without creating a temporary local file

文件存储

服务器

Python

有没有可行的方法将动态生成的文件直接上传到亚马逊s3，而无需首先创建本地文件，然后上传到s3服务器？我使用python。谢谢

问如何在不创建临时本地文件的情况下将文件上传到S3
EN

回答 10

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何在不创建临时本地文件的情况下将文件上传到S3EN

回答 10

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何在不创建临时本地文件的情况下将文件上传到S3
EN