SpringBoot开发符合S3协议的文件存储服务

原创

code2roc

发布于 2023-07-23 14:03:10

5220

发布于 2023-07-23 14:03:10

背景

公司最近的业务大量涉及安可项目，要求避免使用第三方组件，原有开发框架支持本地文件存储/Minio/各类云存储，现在要求文件独立存储且文件服务需要自研，经调研评估后决定基于SpringBoot开发文件存储服务，使用s3协议标准，这样可以直接使用aws-sdk接入无需再开发客户端，且安全安全性方面可以得到足够的保证（签名验证部分参考我的博文《Java实现AWS S3 V4 Authorization自定义验证》）

项目地址：https://gitee.com/code2roc/local-s3

运行jar包，默认信息如下

api地址：http://localhost:8001/s3

用户名：admin

密码：abcd@1234

概述

s3协议无标准说明文档，为rest风格，创建/删除/详情方法通过PUT/DELETE/HEAD表述

很多方法共用一个路由，通过head参数区分（例如putObject和copyObject）

参考aws的最新api文档：https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html

实现以下基础功能

Bucket创建

  @PutMapping("/{bucketName}")
  public ResponseEntity<String> createBucket(@PathVariable String bucketName)

Bucket删除

  @DeleteMapping("/{bucketName}")
  public ResponseEntity<String> deleteBucket(@PathVariable String bucketName)

文件上传

  @PutMapping("/{bucketName}/**")
  public ResponseEntity<String> putObject(@PathVariable String bucketName, HttpServletRequest request)

文件删除

  @DeleteMapping("/{bucketName}/**")
  public ResponseEntity<String> deleteObject(@PathVariable String bucketName, HttpServletRequest request)

文件下载

  @GetMapping("/{bucketName}/**")
  public void getObject(@PathVariable String bucketName, HttpServletRequest request, HttpServletResponse response)

文件分片操作（初始化/分片上传/合并）

  @RequestMapping(value = "/{bucketName}/**", method = RequestMethod.POST, params = "uploads")
  public ResponseEntity<Object> createMultipartUpload(@PathVariable String bucketName, HttpServletRequest request)
  
  @RequestMapping(value = "/{bucketName}/**", method = RequestMethod.PUT, params = {"partNumber", "uploadId"})
  public ResponseEntity<String> uploadPart(@PathVariable String bucketName, HttpServletRequest request, HttpServletResponse response)
  
  @RequestMapping(value = "/{bucketName}/**", method = RequestMethod.POST, params = "uploadId")
  public ResponseEntity<String> completeMultipartUpload(@PathVariable String bucketName, HttpServletRequest request)

实现以下扩展功能（兼容s3 browser使用）

Bucket详情

  @RequestMapping(value = "/{bucketName}", method = RequestMethod.HEAD)
  public ResponseEntity<Object> headBucket(@PathVariable(value = "bucketName") String bucketName)

Buckent列表

  @GetMapping("/")
  public ResponseEntity<String> listBuckets()

文件详情

  @RequestMapping(value = "/{bucketName}/**", method = RequestMethod.HEAD)
  public ResponseEntity<Object> headObject(@PathVariable String bucketName, HttpServletRequest request, HttpServletResponse response)

文件列表

  @GetMapping("/{bucketName}")
  public ResponseEntity<String> listObjects(@PathVariable String bucketName, HttpServletRequest request)

项目接入

maven引用

        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3</artifactId>
            <version>2.20.45</version>
        </dependency>

客户端连接

    private S3Client getClient() {
        S3Client s3 = S3Client.builder()
                .credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create(systemConfig.getUsername(), systemConfig.getPassword())))
                .endpointOverride(URI.create(CommonUtil.getApiPath() + "s3/"))
                .serviceConfiguration(S3Configuration.builder().pathStyleAccessEnabled(true).chunkedEncodingEnabled(false).build())
                .region(Region.US_EAST_1)
                .build();
        return s3;
    }

文件操作

    public void upload(String bucketName, String key, InputStream inputStream) throws Exception {
        S3Client s3Client = getClient();
        PutObjectRequest request = PutObjectRequest.builder().bucket(bucketName).key(key).build();
        RequestBody requestBody = RequestBody.fromBytes(FileUtil.convertStreamToByte(inputStream));
        s3Client.putObject(request, requestBody);
        s3Client.close();
    }