Amazon S3 分布式存储的 python 接口实现

用户2434869

发布于 2018-09-12 10:41:04

2.1K0

发布于 2018-09-12 10:41:04

文章被收录于专栏：yl 成长笔记

Amazon s3 是一种分布式的对象存储。用键值对的方式，来存储数据。其中，存入的所有数据都是一个对象（object），每一个对象都有一个键（key）存在。

具有非常方便的 web 访问接口，以及权限控制。

下面是具体读、写、判断三个接口的具体实现

1. 创建一个 bucket

class ImageFeatIO:
    __read_singleton = None
    __write_singleton = None
    __read_count = 0
    __write_count = 0

    def __init__(self):
        pass

    @staticmethod
    def get_write_instance():  #创建一个 bucket 的单例
        if ImageFeatIO.__write_singleton is None:
            paras = get_config('config')
            access_key = paras['access_key']
            secret_key = paras['secret_key']
            write_host = paras['file_write_host']
            conn = boto.connect_s3(
                aws_access_key_id=access_key,
                aws_secret_access_key=secret_key,
                host=write_host, is_secure=False,
                calling_format=boto.s3.connection.OrdinaryCallingFormat()
            )
            bucket_name = paras['bucket_name']
            bucket = conn.get_bucket(bucket_name)
            ImageFeatIO.__write_singleton = bucket
        return ImageFeatIO.__write_singleton

2. 写入文件

写入文件的时候，可以进行权限控制

官方说明如下：

Create a custom ACL that grants specific rights to specific users. At the moment, the users that are specified within grants have to be registered users of Amazon Web Services so this isn’t as useful or as general as it could be.
Use a “canned” access control policy. There are four canned policies defined:
1. private: Owner gets FULL_CONTROL. No one else has any access rights.
2. public-read: Owners gets FULL_CONTROL and the anonymous principal is granted READ access.
3. public-read-write: Owner gets FULL_CONTROL and the anonymous principal is granted READ and WRITE access.
4. authenticated-read: Owner gets FULL_CONTROL and any principal authenticated as a registered Amazon S3 user is granted READ access

def write_image_feature_to_file(id, imageFeaturestring):
    bucket = ImageFeatIO.get_write_instance()
    k = Key(bucket)
    k.key = id
    res = k.set_contents_from_string(imageFeaturestring)
    k.set_acl('authenticated-read')
    return res

3.读取（下载）文件

s3 提供了一种非常方便的 web 服务接口，可以从任何地方以 http 协议获取数据

def read_image_feature_from_file(id):
    url_head = ImageFeatIO.get_read_instance()
    url = url_head + '/' + id  #url= http://img.3weijia.com/bucket_name/key
    request = urllib2.Request(url=url)
    response = urllib2.urlopen(request, timeout=20)
    res_unicode = unicode(response.read())
    res = res_unicode.encode('unicode-escape').decode('string_escape')
    return res

4. 判断键值是否存在

def if_image_feature_exist(id):
    bucket = ImageFeatIO.get_write_instance()
    key = bucket.get_key(id)

    return key is not None

参考：

http://boto.readthedocs.io/en/latest/s3_tut.html

https://www.cnblogs.com/web424/p/6840207.html

本文参与腾讯云自媒体同步曝光计划，分享自作者个人站点/博客。

原始发表：2018-05-26 ，如有侵权请联系 cloudcommunity@tencent.com 删除

其他

本文分享自作者个人站点/博客前往查看

如有侵权，请联系 cloudcommunity@tencent.com 删除。

本文参与腾讯云自媒体同步曝光计划，欢迎热爱写作的你一起参与！

其他

登录后参与评论

0 条评论

热度

Amazon S3 分布式存储的 python 接口实现

Amazon S3 分布式存储的 python 接口实现

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐