使用boto3从S3存储桶中读取文件内容
使用boto3从S3存储桶中读取文件内容
提问于 2016-03-25 00:41:02
我通过执行以下操作读取S3存储桶中的文件名
objs = boto3.client.list_objects(Bucket='my_bucket')
while 'Contents' in objs.keys():
objs_contents = objs['Contents']
for i in range(len(objs_contents)):
filename = objs_contents[i]['Key']现在，我需要获取文件的实际内容，类似于open(filename).readlines()。最好的方法是什么？
回答 6
Stack Overflow用户
回答已采纳
发布于 2016-03-25 00:57:04
boto3提供了一个资源模型，使遍历对象等任务变得更容易。不幸的是，StreamingBody没有提供readline或readlines。
s3 = boto3.resource('s3')
bucket = s3.Bucket('test-bucket')
# Iterates through all the objects, doing the pagination for you. Each obj
# is an ObjectSummary, so it doesn't contain the body. You'll need to call
# get to get the whole body.
for obj in bucket.objects.all():
key = obj.key
body = obj.get()['Body'].read()Stack Overflow用户
发布于 2018-12-15 02:30:51
您还可以考虑smart_open模块，它支持迭代器：
from smart_open import smart_open
# stream lines from an S3 object
for line in smart_open('s3://mybucket/mykey.txt', 'rb'):
print(line.decode('utf8'))和上下文管理器：
with smart_open('s3://mybucket/mykey.txt', 'rb') as s3_source:
for line in s3_source:
print(line.decode('utf8'))
s3_source.seek(0) # seek to the beginning
b1000 = s3_source.read(1000) # read 1000 bytes在https://pypi.org/project/smart_open/上查找smart_open
Stack Overflow用户
发布于 2021-01-28 02:08:59
使用客户端而不是资源：
s3 = boto3.client('s3')
bucket='bucket_name'
result = s3.list_objects(Bucket = bucket, Prefix='/something/')
for o in result.get('Contents'):
data = s3.get_object(Bucket=bucket, Key=o.get('Key'))
contents = data['Body'].read()
print(contents.decode("utf-8"))页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/36205481
复制相关文章
相似问题
腾讯云开发者
