前往小程序,Get更优阅读体验!
立即前往
首页
学习
活动
专区
工具
TVP
发布
社区首页 >专栏 >Amazon S3 分布式存储的 python 接口实现

Amazon S3 分布式存储的 python 接口实现

作者头像
用户2434869
发布2018-09-12 10:41:04
2K0
发布2018-09-12 10:41:04
举报
文章被收录于专栏:yl 成长笔记

Amazon s3 是一种分布式的对象存储。用键值对的方式,来存储数据。其中,存入的所有数据都是一个对象(object),每一个对象都有一个键(key)存在。

具有非常方便的 web 访问接口,以及权限控制。

下面是具体读、写、判断三个接口的具体实现

1. 创建一个 bucket

代码语言:javascript
复制
class ImageFeatIO:
    __read_singleton = None
    __write_singleton = None
    __read_count = 0
    __write_count = 0

    def __init__(self):
        pass

    @staticmethod
    def get_write_instance():  #创建一个 bucket 的单例
        if ImageFeatIO.__write_singleton is None:
            paras = get_config('config')
            access_key = paras['access_key']
            secret_key = paras['secret_key']
            write_host = paras['file_write_host']
            conn = boto.connect_s3(
                aws_access_key_id=access_key,
                aws_secret_access_key=secret_key,
                host=write_host, is_secure=False,
                calling_format=boto.s3.connection.OrdinaryCallingFormat()
            )
            bucket_name = paras['bucket_name']
            bucket = conn.get_bucket(bucket_name)
            ImageFeatIO.__write_singleton = bucket
        return ImageFeatIO.__write_singleton

2. 写入文件

写入文件的时候,可以进行权限控制

官方说明如下:

  1. Create a custom ACL that grants specific rights to specific users. At the moment, the users that are specified within grants have to be registered users of Amazon Web Services so this isn’t as useful or as general as it could be.
  2. Use a “canned” access control policy. There are four canned policies defined:
    1. private: Owner gets FULL_CONTROL. No one else has any access rights.
    2. public-read: Owners gets FULL_CONTROL and the anonymous principal is granted READ access.
    3. public-read-write: Owner gets FULL_CONTROL and the anonymous principal is granted READ and WRITE access.
    4. authenticated-read: Owner gets FULL_CONTROL and any principal authenticated as a registered Amazon S3 user is granted READ access
代码语言:javascript
复制
def write_image_feature_to_file(id, imageFeaturestring):
    bucket = ImageFeatIO.get_write_instance()
    k = Key(bucket)
    k.key = id
    res = k.set_contents_from_string(imageFeaturestring)
    k.set_acl('authenticated-read')
    return res

3.读取(下载)文件

s3 提供了一种非常方便的 web 服务接口,可以从任何地方以 http 协议获取数据

代码语言:javascript
复制
def read_image_feature_from_file(id):
    url_head = ImageFeatIO.get_read_instance()
    url = url_head + '/' + id  #url= http://img.3weijia.com/bucket_name/key
    request = urllib2.Request(url=url)
    response = urllib2.urlopen(request, timeout=20)
    res_unicode = unicode(response.read())
    res = res_unicode.encode('unicode-escape').decode('string_escape')
    return res

4. 判断键值是否存在

代码语言:javascript
复制
def if_image_feature_exist(id):
    bucket = ImageFeatIO.get_write_instance()
    key = bucket.get_key(id)

    return key is not None

参考:

http://boto.readthedocs.io/en/latest/s3_tut.html

https://www.cnblogs.com/web424/p/6840207.html

本文参与 腾讯云自媒体同步曝光计划,分享自作者个人站点/博客。
原始发表:2018-05-26 ,如有侵权请联系 cloudcommunity@tencent.com 删除

本文分享自 作者个人站点/博客 前往查看

如有侵权,请联系 cloudcommunity@tencent.com 删除。

本文参与 腾讯云自媒体同步曝光计划  ,欢迎热爱写作的你一起参与!

评论
登录后参与评论
0 条评论
热度
最新
推荐阅读
相关产品与服务
对象存储
对象存储(Cloud Object Storage,COS)是由腾讯云推出的无目录层次结构、无数据格式限制,可容纳海量数据且支持 HTTP/HTTPS 协议访问的分布式存储服务。腾讯云 COS 的存储桶空间无容量上限,无需分区管理,适用于 CDN 数据分发、数据万象处理或大数据计算与分析的数据湖等多种场景。
领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档