经验首页 前端设计 程序设计 Java相关 移动开发 数据库/运维 软件/图像 大数据/云计算 其他经验
当前位置:技术经验 » Java相关 » Java » 查看文章
Amazon S3 对象存储Java API操作记录(Minio与S3 SDK两种实现)
来源:cnblogs  作者:东北小狐狸  时间:2023/4/28 9:28:19  对本文有异议

缘起

今年(2023年) 2月的时候做了个适配Amazon S3对象存储接口的需求,由于4月份自学考试临近,一直在备考就拖着没总结记录下,开发联调过程中也出现过一些奇葩的问题,最近人刚从考试缓过来顺手记录一下。

S3对象存储的基本概念

S3是什么?

Amazon S3(Simple Storage Service)对象存储出现得比较早且使用简单的RESTful API,于是成为了对象存储服务(Object Storage Service,OSS)业内的标准接口规范。

S3的逻辑模型

如下图,我们可以把S3的存储空间想象成无限的,想存储一个任意格式的文件到S3服务中,只需要知道要把它放到哪个桶(Bucket)中,它的名字(Object Id)应该是什么。

按图中的模型,可简单理解为S3是由若干个桶(Bucket)组成,每个桶中包含若干个不同标识的对象(Object),还有就是统一的访问入口(RESTful API),这样基本就足够了。

Minio客户端方式操作S3

详细API文档:https://min.io/docs/minio/linux/developers/java/API.html

以下代码异常处理做了简化,真实使用时请注意捕获异常做处理。

引入依赖

Maven:

  1. <dependency>
  2. <groupId>io.minio</groupId>
  3. <artifactId>minio</artifactId>
  4. <version>8.5.2</version>
  5. </dependency>

Gradle:

  1. dependencies {
  2. implementation("io.minio:minio:8.5.2")
  3. }

初始化客户端

  1. private static final String HTTP_PROTOCOL = "http";
  2. private MinioClient minioClient;
  3. private String endpoint = "http://192.168.0.8:9200";
  4. private String accessKey = "testKey";
  5. private String secretKey = "testSecretKey";
  6. public void init() throws MalformedURLException {
  7. URL endpointUrl = new URL(endpoint);
  8. try {
  9. // url上无端口号时,识别http为80端口,https为443端口
  10. int port = endpointUrl.getPort() != -1 ? endpointUrl.getPort() : endpointUrl.getDefaultPort();
  11. boolean security = HTTP_PROTOCOL.equals(endpointUrl.getProtocol()) ? false : true;
  12. //@formatter:off
  13. this.minioClient = MinioClient.builder().endpoint(endpointUrl.getHost(), port, security)
  14. .credentials(accessKey, secretKey).build();
  15. //@formatter:on
  16. // 忽略证书校验,防止自签名证书校验失败导致无法建立连接
  17. this.minioClient.ignoreCertCheck();
  18. } catch (Exception e) {
  19. e.printStackTrace();
  20. }
  21. }

建桶

  1. public boolean createBucket(String bucket) {
  2. try {
  3. minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucket).build());
  4. } catch (Exception e) {
  5. e.printStackTrace();
  6. return false;
  7. }
  8. return true;
  9. }

删桶

  1. public boolean deleteBucket(String bucket) {
  2. try {
  3. minioClient.removeBucket(RemoveBucketArgs.builder().bucket(bucket).build());
  4. logger.info("删除桶[{}]成功", bucket);
  5. } catch (Exception e) {
  6. e.printStackTrace();
  7. return false;
  8. }
  9. return true;
  10. }

判断桶是否存在

  1. public boolean bucketExists(String bucket) {
  2. try {
  3. return minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucket).build());
  4. } catch (Exception e) {
  5. e.printStackTrace();
  6. return false;
  7. }
  8. }

上传对象

  1. public void upload(String bucket, String objectId, InputStream input) {
  2. try {
  3. //@formatter:off
  4. minioClient.putObject(PutObjectArgs.builder().bucket(bucket).object(objectId)
  5. .stream(input, input.available(), -1)
  6. .build());
  7. //@formatter:on
  8. } catch (Exception e) {
  9. e.printStackTrace();
  10. }
  11. }

下载对象

提供两个下载方法,一个将输入流返回,另一个用参数输出流写出

  1. public InputStream download(String bucket, String objectId) {
  2. try {
  3. return minioClient.getObject(GetObjectArgs.builder().bucket(bucket).object(objectId).build());
  4. } catch (Exception e) {
  5. e.printStackTrace();
  6. }
  7. return null;
  8. }
  9. public void download(String bucket, String objectId, OutputStream output) {
  10. //@formatter:off
  11. try (InputStream input = minioClient.getObject(
  12. GetObjectArgs.builder().bucket(bucket).object(objectId).build())) {
  13. IOUtils.copyLarge(input, output);
  14. } catch (Exception e) {
  15. e.printStackTrace();
  16. }
  17. //@formatter:on
  18. }

删除对象

  1. public boolean deleteObject(String bucket, String objectId) {
  2. //@formatter:off
  3. try {
  4. minioClient.removeObject(RemoveObjectArgs.builder()
  5. .bucket(bucket).object(objectId).build());
  6. } catch (Exception e) {
  7. e.printStackTrace();
  8. }
  9. //@formatter:on
  10. return true;
  11. }

判断对象是否存在

  1. public boolean objectExists(String bucket, String key) {
  2. //@formatter:off
  3. try {
  4. // minio客户端未提供判断对象是否存在的方法,此方法中调用出现异常时说明对象不存在
  5. minioClient.statObject(StatObjectArgs.builder()
  6. .bucket(bucket).object(key).build());
  7. } catch (Exception e) {
  8. return false;
  9. }
  10. //@formatter:on
  11. return true;
  12. }

完整代码

  1. import java.io.InputStream;
  2. import java.io.OutputStream;
  3. import java.net.MalformedURLException;
  4. import java.net.URL;
  5. import org.apache.tomcat.util.http.fileupload.IOUtils;
  6. import org.slf4j.Logger;
  7. import org.slf4j.LoggerFactory;
  8. import io.minio.BucketExistsArgs;
  9. import io.minio.GetObjectArgs;
  10. import io.minio.MakeBucketArgs;
  11. import io.minio.MinioClient;
  12. import io.minio.PutObjectArgs;
  13. import io.minio.RemoveBucketArgs;
  14. import io.minio.RemoveObjectArgs;
  15. import io.minio.StatObjectArgs;
  16. public class S3MinioClientDemo {
  17. private static final Logger logger = LoggerFactory.getLogger(S3MinioClientDemo.class);
  18. private static final String HTTP_PROTOCOL = "http";
  19. private MinioClient minioClient;
  20. private String endpoint = "http://192.168.0.8:9200";
  21. private String accessKey = "testKey";
  22. private String secretKey = "testSecretKey";
  23. public void init() throws MalformedURLException {
  24. URL endpointUrl = new URL(endpoint);
  25. try {
  26. // url上无端口号时,识别http为80端口,https为443端口
  27. int port = endpointUrl.getPort() != -1 ? endpointUrl.getPort() : endpointUrl.getDefaultPort();
  28. boolean security = HTTP_PROTOCOL.equals(endpointUrl.getProtocol()) ? false : true;
  29. //@formatter:off
  30. this.minioClient = MinioClient.builder().endpoint(endpointUrl.getHost(), port, security)
  31. .credentials(accessKey, secretKey).build();
  32. //@formatter:on
  33. // 忽略证书校验,防止自签名证书校验失败导致无法建立连接
  34. this.minioClient.ignoreCertCheck();
  35. } catch (Exception e) {
  36. e.printStackTrace();
  37. }
  38. }
  39. public boolean createBucket(String bucket) {
  40. try {
  41. boolean found = minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucket).build());
  42. if (found) {
  43. logger.info("桶名[{}]已存在", bucket);
  44. return false;
  45. }
  46. minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucket).build());
  47. } catch (Exception e) {
  48. e.printStackTrace();
  49. }
  50. return true;
  51. }
  52. public boolean deleteBucket(String bucket) {
  53. try {
  54. minioClient.removeBucket(RemoveBucketArgs.builder().bucket(bucket).build());
  55. logger.info("删除桶[{}]成功", bucket);
  56. } catch (Exception e) {
  57. e.printStackTrace();
  58. return false;
  59. }
  60. return true;
  61. }
  62. public boolean bucketExists(String bucket) {
  63. try {
  64. return minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucket).build());
  65. } catch (Exception e) {
  66. e.printStackTrace();
  67. return false;
  68. }
  69. }
  70. public void upload(String bucket, String objectId, InputStream input) {
  71. try {
  72. //@formatter:off
  73. minioClient.putObject(PutObjectArgs.builder().bucket(bucket).object(objectId)
  74. .stream(input, input.available(), -1)
  75. .build());
  76. //@formatter:on
  77. } catch (Exception e) {
  78. e.printStackTrace();
  79. }
  80. }
  81. public InputStream download(String bucket, String objectId) {
  82. try {
  83. return minioClient.getObject(GetObjectArgs.builder().bucket(bucket).object(objectId).build());
  84. } catch (Exception e) {
  85. e.printStackTrace();
  86. }
  87. return null;
  88. }
  89. public void download(String bucket, String objectId, OutputStream output) {
  90. //@formatter:off
  91. try (InputStream input = minioClient.getObject(
  92. GetObjectArgs.builder().bucket(bucket).object(objectId).build())) {
  93. IOUtils.copyLarge(input, output);
  94. } catch (Exception e) {
  95. e.printStackTrace();
  96. }
  97. //@formatter:on
  98. }
  99. public boolean objectExists(String bucket, String objectId) {
  100. //@formatter:off
  101. try {
  102. // minio客户端未提供判断对象是否存在的方法,此方法中调用出现异常时说明对象不存在
  103. minioClient.statObject(StatObjectArgs.builder()
  104. .bucket(bucket).object(objectId).build());
  105. } catch (Exception e) {
  106. return false;
  107. }
  108. //@formatter:on
  109. return true;
  110. }
  111. public boolean deleteObject(String bucket, String objectId) {
  112. //@formatter:off
  113. try {
  114. minioClient.removeObject(RemoveObjectArgs.builder()
  115. .bucket(bucket).object(objectId).build());
  116. } catch (Exception e) {
  117. e.printStackTrace();
  118. }
  119. //@formatter:on
  120. return true;
  121. }
  122. public void close() {
  123. minioClient = null;
  124. }
  125. }

Amazon S3 SDK方式操作S3

官方API文档:https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html

这里由于项目上提供的SDK和文档都是1.x的,这里就暂时只提供1.x的代码

引入依赖

Maven:

  1. <dependency>
  2. <groupId>com.amazonaws</groupId>
  3. <artifactId>aws-java-sdk-s3</artifactId>
  4. <version>1.11.300</version>
  5. </dependency>

Gradle:

  1. dependencies {
  2. implementation 'com.amazonaws:aws-java-sdk-s3:1.11.300'
  3. }

初始化客户端

  1. private static final Logger logger = LoggerFactory.getLogger(S3SdkDemo.class);
  2. private AmazonS3 s3client;
  3. private String endpoint = "http://192.168.0.8:9200";
  4. private String accessKey = "testKey";
  5. private String secretKey = "testSecretKey";
  6. public void init() throws MalformedURLException {
  7. URL endpointUrl = new URL(endpoint);
  8. String protocol = endpointUrl.getProtocol();
  9. int port = endpointUrl.getPort() == -1 ? endpointUrl.getDefaultPort() : endpointUrl.getPort();
  10. ClientConfiguration clientConfig = new ClientConfiguration();
  11. clientConfig.setSignerOverride("S3SignerType");
  12. clientConfig.setProtocol(Protocol.valueOf(protocol.toUpperCase()));
  13. // 禁用证书检查,避免https自签证书校验失败
  14. System.setProperty("com.amazonaws.sdk.disableCertChecking", "true");
  15. // 屏蔽 AWS 的 MD5 校验,避免校验导致的下载抛出异常问题
  16. System.setProperty("com.amazonaws.services.s3.disableGetObjectMD5Validation", "true");
  17. AWSCredentials awsCredentials = new BasicAWSCredentials(accessKey, secretKey);
  18. // 创建 S3Client 实例
  19. AmazonS3 s3client = new AmazonS3Client(awsCredentials, clientConfig);
  20. s3client.setEndpoint(endpointUrl.getHost() + ":" + port);
  21. s3client.setS3ClientOptions(S3ClientOptions.builder().setPathStyleAccess(true).build());
  22. this.s3client = s3client;
  23. }

建桶

  1. public boolean createBucket(String bucket) {
  2. String bucketName = parseBucketName(bucket);
  3. try {
  4. if (s3client.doesBucketExist(bucketName)) {
  5. logger.warn("bucket[{}]已存在", bucketName);
  6. return false;
  7. }
  8. s3client.createBucket(bucketName);
  9. } catch (Exception e) {
  10. e.printStackTrace();
  11. }
  12. return true;
  13. }

删桶

  1. public boolean deleteBucket(String bucket) {
  2. try {
  3. s3client.deleteBucket(bucket);
  4. logger.info("删除bucket[{}]成功", bucket);
  5. } catch (Exception e) {
  6. e.printStackTrace();
  7. return false;
  8. }
  9. return true;
  10. }

判断桶是否存在

  1. public boolean bucketExists(String bucket) {
  2. try {
  3. return s3client.doesBucketExist(bucket);
  4. } catch (Exception e) {
  5. e.printStackTrace();
  6. }
  7. return false;
  8. }

上传对象

  1. public void upload(String bucket, String objectId, InputStream input) {
  2. try {
  3. // 创建文件上传的元数据
  4. ObjectMetadata meta = new ObjectMetadata();
  5. // 设置文件上传长度
  6. meta.setContentLength(input.available());
  7. // 上传
  8. s3client.putObject(bucket, objectId, input, meta);
  9. } catch (Exception e) {
  10. e.printStackTrace();
  11. }
  12. }

下载对象

  1. public InputStream download(String bucket, String objectId) {
  2. try {
  3. S3Object o = s3client.getObject(bucket, objectId);
  4. return o.getObjectContent();
  5. } catch (Exception e) {
  6. e.printStackTrace();
  7. }
  8. return null;
  9. }
  10. public void download(String bucket, String objectId, OutputStream out) {
  11. S3Object o = s3client.getObject(bucket, objectId);
  12. try (InputStream in = o.getObjectContent()) {
  13. IOUtils.copyLarge(in, out);
  14. } catch (Exception e) {
  15. e.printStackTrace();
  16. }
  17. }

删除对象

  1. public boolean deleteObject(String bucket, String objectId) {
  2. try {
  3. s3client.deleteObject(bucket, objectId);
  4. } catch (Exception e) {
  5. e.printStackTrace();
  6. return false;
  7. }
  8. return true;
  9. }

判断对象是否存在

  1. public boolean existObject(String bucket, String objectId) {
  2. try {
  3. return s3client.doesObjectExist(bucket, objectId);
  4. } catch (Exception e) {
  5. e.printStackTrace();
  6. return false;
  7. }
  8. }

完整代码

  1. import java.io.InputStream;
  2. import java.io.OutputStream;
  3. import java.net.MalformedURLException;
  4. import java.net.URL;
  5. import org.apache.tomcat.util.http.fileupload.IOUtils;
  6. import org.slf4j.Logger;
  7. import org.slf4j.LoggerFactory;
  8. import com.amazonaws.ClientConfiguration;
  9. import com.amazonaws.Protocol;
  10. import com.amazonaws.auth.AWSCredentials;
  11. import com.amazonaws.auth.BasicAWSCredentials;
  12. import com.amazonaws.services.s3.AmazonS3;
  13. import com.amazonaws.services.s3.AmazonS3Client;
  14. import com.amazonaws.services.s3.S3ClientOptions;
  15. import com.amazonaws.services.s3.model.ObjectMetadata;
  16. import com.amazonaws.services.s3.model.S3Object;
  17. /**
  18. * S3对象存储官方SDK实现
  19. *
  20. * @author ZhangChenguang
  21. * @date 2023年2月2日
  22. */
  23. @SuppressWarnings("deprecation")
  24. public class S3SdkDemo {
  25. private static final Logger logger = LoggerFactory.getLogger(S3SdkDemo.class);
  26. private AmazonS3 s3client;
  27. private String endpoint = "http://192.168.0.8:9200";
  28. private String accessKey = "testKey";
  29. private String secretKey = "testSecretKey";
  30. public void init() throws MalformedURLException {
  31. URL endpointUrl = new URL(endpoint);
  32. String protocol = endpointUrl.getProtocol();
  33. int port = endpointUrl.getPort() == -1 ? endpointUrl.getDefaultPort() : endpointUrl.getPort();
  34. ClientConfiguration clientConfig = new ClientConfiguration();
  35. clientConfig.setSignerOverride("S3SignerType");
  36. clientConfig.setProtocol(Protocol.valueOf(protocol.toUpperCase()));
  37. // 禁用证书检查,避免https自签证书校验失败
  38. System.setProperty("com.amazonaws.sdk.disableCertChecking", "true");
  39. // 屏蔽 AWS 的 MD5 校验,避免校验导致的下载抛出异常问题
  40. System.setProperty("com.amazonaws.services.s3.disableGetObjectMD5Validation", "true");
  41. AWSCredentials awsCredentials = new BasicAWSCredentials(accessKey, secretKey);
  42. // 创建 S3Client 实例
  43. AmazonS3 s3client = new AmazonS3Client(awsCredentials, clientConfig);
  44. s3client.setEndpoint(endpointUrl.getHost() + ":" + port);
  45. s3client.setS3ClientOptions(S3ClientOptions.builder().setPathStyleAccess(true).build());
  46. this.s3client = s3client;
  47. }
  48. public boolean createBucket(String bucket) {
  49. try {
  50. s3client.createBucket(bucket);
  51. } catch (Exception e) {
  52. e.printStackTrace();
  53. }
  54. return true;
  55. }
  56. public boolean deleteBucket(String bucket) {
  57. try {
  58. s3client.deleteBucket(bucket);
  59. logger.info("删除bucket[{}]成功", bucket);
  60. } catch (Exception e) {
  61. e.printStackTrace();
  62. return false;
  63. }
  64. return true;
  65. }
  66. public boolean bucketExists(String bucket) {
  67. try {
  68. return s3client.doesBucketExist(bucket);
  69. } catch (Exception e) {
  70. e.printStackTrace();
  71. }
  72. return false;
  73. }
  74. public void upload(String bucket, String objectId, InputStream input) {
  75. try {
  76. // 创建文件上传的元数据
  77. ObjectMetadata meta = new ObjectMetadata();
  78. // 设置文件上传长度
  79. meta.setContentLength(input.available());
  80. // 上传
  81. s3client.putObject(bucket, objectId, input, meta);
  82. } catch (Exception e) {
  83. e.printStackTrace();
  84. }
  85. }
  86. public InputStream download(String bucket, String objectId) {
  87. try {
  88. S3Object o = s3client.getObject(bucket, objectId);
  89. return o.getObjectContent();
  90. } catch (Exception e) {
  91. e.printStackTrace();
  92. }
  93. return null;
  94. }
  95. public void download(String bucket, String objectId, OutputStream out) {
  96. S3Object o = s3client.getObject(bucket, objectId);
  97. try (InputStream in = o.getObjectContent()) {
  98. IOUtils.copyLarge(in, out);
  99. } catch (Exception e) {
  100. e.printStackTrace();
  101. }
  102. }
  103. public boolean existObject(String bucket, String objectId) {
  104. try {
  105. return s3client.doesObjectExist(bucket, objectId);
  106. } catch (Exception e) {
  107. e.printStackTrace();
  108. return false;
  109. }
  110. }
  111. public boolean deleteObject(String bucket, String objectId) {
  112. try {
  113. s3client.deleteObject(bucket, objectId);
  114. } catch (Exception e) {
  115. e.printStackTrace();
  116. return false;
  117. }
  118. return true;
  119. }
  120. public void close() {
  121. s3client = null;
  122. }
  123. }

遇到的问题

1、bucket名称必须是小写,不支持下划线

  • 处理方式:写方法转换下bucket名称,将大写转小写,将下划线替换为中划线。

2、minio客户端下载非官方S3存储的文件时,如果响应头的Content-Length与实际文件大小不符,会导致minio客户端包装的okhttp3报错

报错信息:

  1. Caused by: java.net.ProtocolException: unexpected end of stream
  2. at okhttp3.internal.http1.Http1ExchangeCodec$FixedLengthSource.read(Http1ExchangeCodec.java:430) ~[okhttp-3.14.9.jar:?]
  3. at okhttp3.internal.connection.Exchange$ResponseBodySource.read(Exchange.java:286) ~[okhttp-3.14.9.jar:?]
  4. at okio.RealBufferedSource$1.read(RealBufferedSource.java:447) ~[okio-1.17.2.jar:?]
  5. at com.jiuqi.nr.file.utils.FileUtils.writeInput2Output(FileUtils.java:83) ~[nr.file-2.5.7.jar:?]
  6. at com.jiuqi.nr.file.impl.FileAreaServiceImpl.download(FileAreaServiceImpl.java:395) ~[nr.file-2.5.7.jar:?]
  7. ... 122 more

抓包发现问题的图:

最终换成了S3官方SDK可用了。

PS:客户现场部署的S3是浪潮公司提供的,如果现场遇到这个情况,就不要固执去找对方对线了,完全没用。。

总结

S3存储的基本操作就记录到这里了,由于没有S3存储就没尝试官方SDK的V2版本,由于这些代码是总结时从业务代码里抽取出来的,可能会有点问题,但大体思路已经有了。

希望对读者有所用处,觉得写得不错和有帮到你,欢迎点个赞,您的支持就是我的鼓励!

原文链接:https://www.cnblogs.com/hellxz/p/17359828.html

 友情链接:直通硅谷  点职佳  北美留学生论坛

本站QQ群:前端 618073944 | Java 606181507 | Python 626812652 | C/C++ 612253063 | 微信 634508462 | 苹果 692586424 | C#/.net 182808419 | PHP 305140648 | 运维 608723728

W3xue 的所有内容仅供测试,对任何法律问题及风险不承担任何责任。通过使用本站内容随之而来的风险与本站无关。
关于我们  |  意见建议  |  捐助我们  |  报错有奖  |  广告合作、友情链接(目前9元/月)请联系QQ:27243702 沸活量
皖ICP备17017327号-2 皖公网安备34020702000426号