经验首页 前端设计 程序设计 Java相关 移动开发 数据库/运维 软件/图像 大数据/云计算 其他经验
当前位置:技术经验 » 程序设计 » Go语言 » 查看文章
【对象存储】Minio本地运行和 golang客户端基本操作
来源:cnblogs  作者:Mr.YF  时间:2023/10/20 8:45:34  对本文有异议

运行环境

OS和Golang版本: 

  1. go version go1.21.0 darwin/arm64

 

安装

源码安装

  • 下载最新版本的源码,地址https://github.com/minio/minio后编译
    1. cd minio
    2. go build main.go
    3. # 得到 116M Oct 19 15:49 main

    把 main 改名为 minio

二进制安装

参考https://www.minio.org.cn/docs/minio/macos/index.html的安装步骤。

 

启动

  1. minio server --address=0.0.0.0:8877 ./data

控制台输出如下信息

  1. ? minio minio server --address=0.0.0.0:8877 ./data
  2. ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  3. You are running an older version of MinIO released 2 years ago
  4. Update: Run `mc admin update`
  5. ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
  6. API: http://0.0.0.0:8877
  7. RootUser: minioadmin
  8. RootPass: minioadmin
  9. Console: http://10.78.14.68:56056 http://192.168.255.10:56056 http://127.0.0.1:56056
  10. RootUser: minioadmin
  11. RootPass: minioadmin
  12. Command-line: https://docs.min.io/docs/minio-client-quickstart-guide
  13. $ mc alias set myminio http://0.0.0.0:8877 minioadmin minioadmin
  14. Documentation: https://docs.min.io

 

启动控制台页面

使用浏览器打开 http://127.0.0.1:56056,使用账号密码登录(minioadmin/minioadmin) 

 

创建aksk

在左侧菜单中选择 Account,点击Create Service Account,创建得到一个 aksk

 

创建Bucket

 

mc命令

mc命令安装

参考https://min.io/docs/minio/linux/reference/minio-mc.html

mc命令运行

mc给本地的minio链接做一个别名

  1. ? ~ mc alias set myminio http://0.0.0.0:8877
  2. Enter Access Key: KLN00KFT1K5EP9I39I9N
  3. Enter Secret Key:
  4. Added `myminio` successfully.

mc查看 minio 节点信息

  1. ? ~ mc admin info myminio
  2. 0.0.0.0:8877
  3. Uptime: 3 hours
  4. Version: 2021-08-05T22:01:19Z

mc 对 minio 做 ping 检查链接是否 ok

  1. ? minio mc ping myminio
  2. 1: http://0.0.0.0:8877:8877 min=9.36ms max=9.36ms average=9.36ms errors=0 roundtrip=9.36ms
  3. 2: http://0.0.0.0:8877:8877 min=0.64ms max=9.36ms average=5.00ms errors=0 roundtrip=0.64ms

 

mc上传文件

  1. ? minio mc cp ./minio-dev.yaml myminio/mybucket/3.yaml
  2. ...o-dev.yaml: 1.46 KiB / 1.46 KiB ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 49.36 KiB/s 0s

 

mc下载文件

  1. ? minio mc cp myminio/mybucket/3.yaml ./3-get.yaml
  2. ...ket/3.yaml: 1.46 KiB / 1.46 KiB ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 67.56 KiB/s 0s

 

mc监听桶变化

  1. ? minio mc watch myminio/mybucket/
  2. [2023-10-19T07:09:05.751Z] 1.5 KiB s3:ObjectCreated:Put http://0.0.0.0:8877/mybucket/3.yaml
  3. [2023-10-19T07:27:35.675Z] 36 B s3:ObjectCreated:Put http://0.0.0.0:8877/mybucket/test/1.txt
  4. [2023-10-19T07:28:46.813Z] s3:ObjectAccessed:Get http://0.0.0.0:8877/mybucket/test/1.txt
  5. [2023-10-19T07:28:58.157Z] s3:ObjectAccessed:Head http://0.0.0.0:8877/mybucket/test/1.txt
  6. [2023-10-19T08:11:38.631Z] s3:ObjectAccessed:Head http://0.0.0.0:8877/mybucket/3.yaml
  7. [2023-10-19T08:11:38.633Z] s3:ObjectAccessed:Head http://0.0.0.0:8877/mybucket/3.yaml
  8. [2023-10-19T08:11:38.634Z] s3:ObjectAccessed:Head http://0.0.0.0:8877/mybucket/3.yaml
  9. [2023-10-19T08:11:38.638Z] s3:ObjectAccessed:Head http://0.0.0.0:8877/mybucket/3.yaml
  10. [2023-10-19T08:11:38.644Z] s3:ObjectAccessed:Head http://0.0.0.0:8877/mybucket/3.yaml
  11. [2023-10-19T08:11:38.653Z] s3:ObjectAccessed:Head http://0.0.0.0:8877/mybucket/3.yaml
  12. [2023-10-19T08:11:38.655Z] s3:ObjectAccessed:Get http://0.0.0.0:8877/mybucket/3.yaml

 

mc查看桶统计信息

  1. ? minio mc stat myminio/mybucket/
  2. Name : 2.yaml
  3. Date : 2023-10-19 14:59:57 CST
  4. Size : 1.5 KiB
  5. ETag : 34095c50340c4381e0fdc5fd61eecc76
  6. Type : file
  7. Metadata :
  8. Content-Type: text/yaml
  9. Name : 3.yaml
  10. Date : 2023-10-19 15:09:05 CST
  11. Size : 1.5 KiB
  12. ETag : 34095c50340c4381e0fdc5fd61eecc76
  13. Type : file
  14. Metadata :
  15. Content-Type: text/yaml
  16. Name : minio-dev.yaml
  17. Date : 2023-10-19 14:54:23 CST
  18. Size : 1.5 KiB
  19. ETag : 34095c50340c4381e0fdc5fd61eecc76
  20. Type : file
  21. Metadata :
  22. Content-Type: text/yaml
  23. Name : test/
  24. Date : 2023-10-19 16:13:06 CST
  25. Type : folder

这个etag 是怎么计算的呢?其实就是文件的 md5值

  1. ? minio md5 3-get.yaml
  2. MD5 (3-get.yaml) = 34095c50340c4381e0fdc5fd61eecc76

 

mc列出桶中的文件

  1. ? minio mc ls myminio/mybucket/
  2. [2023-10-19 14:59:57 CST] 1.5KiB STANDARD 2.yaml
  3. [2023-10-19 15:09:05 CST] 1.5KiB STANDARD 3.yaml
  4. [2023-10-19 14:54:23 CST] 1.5KiB STANDARD minio-dev.yaml
  5. [2023-10-19 16:18:25 CST] 0B test/

 

服务端文件存储

 启动时,将minio 的工作目录设置到 data 下,在 data 目录下主要有两个目录

  • .minio.sys 是minio系统信息
    • 包括桶定义和桶中的文件索引目录 ./.minio.sys/buckets/mybucket
    • 账号信息和iam信息  ./.minio.sys/config/iam/service-accounts/KLN00KFT1K5EP9I39I9N
  • mybucket 是创建的一个桶名称
  1. ? data find .
  2. .
  3. ./.minio.sys
  4. ./.minio.sys/buckets
  5. ./.minio.sys/buckets/.usage-cache.bin
  6. ./.minio.sys/buckets/.minio.sys
  7. ./.minio.sys/buckets/.minio.sys/buckets
  8. ./.minio.sys/buckets/.minio.sys/buckets/.usage-cache.bin
  9. ./.minio.sys/buckets/.minio.sys/buckets/.usage-cache.bin/fs.json
  10. ./.minio.sys/buckets/.minio.sys/buckets/.bloomcycle.bin
  11. ./.minio.sys/buckets/.minio.sys/buckets/.bloomcycle.bin/fs.json
  12. ./.minio.sys/buckets/.minio.sys/buckets/mybucket
  13. ./.minio.sys/buckets/.minio.sys/buckets/mybucket/.usage-cache.bin
  14. ./.minio.sys/buckets/.minio.sys/buckets/mybucket/.usage-cache.bin/fs.json
  15. ./.minio.sys/buckets/.minio.sys/buckets/.usage.json
  16. ./.minio.sys/buckets/.minio.sys/buckets/.usage.json/fs.json
  17. ./.minio.sys/buckets/.bloomcycle.bin
  18. ./.minio.sys/buckets/mybucket
  19. ./.minio.sys/buckets/mybucket/.usage-cache.bin
  20. ./.minio.sys/buckets/mybucket/test
  21. ./.minio.sys/buckets/mybucket/test/1.txt
  22. ./.minio.sys/buckets/mybucket/test/1.txt/fs.json
  23. ./.minio.sys/buckets/mybucket/minio-dev.yaml
  24. ./.minio.sys/buckets/mybucket/minio-dev.yaml/fs.json
  25. ./.minio.sys/buckets/mybucket/3.yaml
  26. ./.minio.sys/buckets/mybucket/3.yaml/fs.json
  27. ./.minio.sys/buckets/mybucket/2.yaml
  28. ./.minio.sys/buckets/mybucket/2.yaml/fs.json
  29. ./.minio.sys/buckets/mybucket/.metadata.bin
  30. ./.minio.sys/buckets/.tracker.bin
  31. ./.minio.sys/buckets/.usage.json
  32. ./.minio.sys/config
  33. ./.minio.sys/config/config.json
  34. ./.minio.sys/config/iam
  35. ./.minio.sys/config/iam/service-accounts
  36. ./.minio.sys/config/iam/service-accounts/KLN00KFT1K5EP9I39I9N
  37. ./.minio.sys/config/iam/service-accounts/KLN00KFT1K5EP9I39I9N/identity.json
  38. ./.minio.sys/config/iam/policydb
  39. ./.minio.sys/config/iam/policydb/sts-users
  40. ./.minio.sys/config/iam/policydb/sts-users/P1Y2O1AO30UYBE2UODBY.json
  41. ./.minio.sys/config/iam/policydb/users
  42. ./.minio.sys/config/iam/policydb/users/ak00123456789.json
  43. ./.minio.sys/config/iam/users
  44. ./.minio.sys/config/iam/users/ak00123456789
  45. ./.minio.sys/config/iam/users/ak00123456789/identity.json
  46. ./.minio.sys/config/iam/format.json
  47. ./.minio.sys/config/iam/sts
  48. ./.minio.sys/config/iam/sts/P1Y2O1AO30UYBE2UODBY
  49. ./.minio.sys/config/iam/sts/P1Y2O1AO30UYBE2UODBY/identity.json
  50. ./.minio.sys/multipart
  51. ./.minio.sys/format.json
  52. ./.minio.sys/tmp
  53. ./.minio.sys/tmp/07c1ffc6-ae6f-4a99-a57e-cb5e55530603
  54. ./mybucket
  55. ./mybucket/test
  56. ./mybucket/test/1.txt
  57. ./mybucket/minio-dev.yaml
  58. ./mybucket/3.yaml
  59. ./mybucket/2.yaml

 

aws-s3操作文件

aws-s3的 sdk 代码简单包装

go.mod

  1. module minio-demo
  2. go 1.18
  3. require (
  4. github.com/aws/aws-sdk-go v1.43.21
  5. )

aws_s3.go

  1. package minio
  2. import (
  3. "bytes"
  4. "io"
  5. "time"
  6.  
  7. "github.com/aws/aws-sdk-go/aws"
  8. "github.com/aws/aws-sdk-go/aws/credentials"
  9. "github.com/aws/aws-sdk-go/aws/session"
  10. "github.com/aws/aws-sdk-go/service/s3"
  11. )
  12. const (
  13. //token暂时为空
  14. DefaultToken = ""
  15. //测试用的regin,调用方需自行配置
  16. DefaultRegion = "us-east-1"
  17. )
  18. // AwsS3 aws s3服务应用层客户端
  19. type AwsS3 struct {
  20. SecretId string
  21. SecretKey string
  22. Region string
  23. Bucket string
  24. Endpoint string
  25. Token string
  26. Client *s3.S3
  27. }
  28. // NewAwsS3 创建aws s3实例
  29. func NewAwsS3(secretId, secretKey, region, bucket, endpoint, token string) *AwsS3 {
  30. var awsS3Instance AwsS3
  31. awsS3Instance.SecretId = secretId
  32. awsS3Instance.SecretKey = secretKey
  33. awsS3Instance.Region = region
  34. awsS3Instance.Bucket = bucket
  35. awsS3Instance.Endpoint = endpoint
  36. awsS3Instance.Token = token
  37. config := &aws.Config{
  38. Credentials: credentials.NewStaticCredentials(secretId, secretKey, token),
  39. Region: aws.String(region),
  40. Endpoint: aws.String(endpoint),
  41. //DisableSSL: aws.Bool(true),
  42. S3ForcePathStyle: aws.Bool(true),
  43. }
  44. sess := session.Must(session.NewSession(config))
  45. awsS3Instance.Client = s3.New(sess)
  46. return &awsS3Instance
  47. }
  48. // PutObject 根据内容上传文件对象
  49. func (a *AwsS3) PutObject(awsPath string, content []byte) (string, error) {
  50. putObjectInput := &s3.PutObjectInput{
  51. Bucket: aws.String(a.Bucket),
  52. Key: aws.String(awsPath),
  53. Body: aws.ReadSeekCloser(bytes.NewReader(content)),
  54. }
  55. resp, err := a.Client.PutObject(putObjectInput)
  56. if err != nil {
  57. return "", err
  58. }
  59. return *(resp.ETag), nil
  60. }
  61. // GetObject 下载文件对象内容
  62. func (a *AwsS3) GetObject(awsPath string) ([]byte, string, error) {
  63. getObjectInput := &s3.GetObjectInput{
  64. Bucket: aws.String(a.Bucket),
  65. Key: aws.String(awsPath),
  66. }
  67. resp, err := a.Client.GetObject(getObjectInput)
  68. if err != nil {
  69. return nil, "", err
  70. }
  71. content, err := io.ReadAll(resp.Body)
  72. if err != nil {
  73. return nil, "", err
  74. }
  75. return content, *(resp.ETag), nil
  76. }
  77. // DeleteObject 删除文件对象
  78. func (a *AwsS3) DeleteObject(awsPath string) error {
  79. deleteObject := &s3.DeleteObjectInput{
  80. Bucket: aws.String(a.Bucket),
  81. Key: aws.String(awsPath),
  82. }
  83. _, err := a.Client.DeleteObject(deleteObject)
  84. if err != nil {
  85. return err
  86. }
  87. return nil
  88. }
  89. // HeadObject 获取对象元数据信息,包括md5和上次修改时间
  90. func (a *AwsS3) HeadObject(awsPath string) (string, *time.Time, error) {
  91. headObject := &s3.HeadObjectInput{
  92. Bucket: aws.String(a.Bucket),
  93. Key: aws.String(awsPath),
  94. }
  95. resp, err := a.Client.HeadObject(headObject)
  96. if err != nil {
  97. return "", nil, err
  98. }
  99. return *(resp.ETag), resp.LastModified, nil
  100. }

 

minio_test.go

  1. package minio
  2. import (
  3. "os"
  4. "testing"
  5. )
  6. // minio测试配置
  7. var (
  8. SecretId = "KLN00KFT1K5EP9I39I9N"
  9. SecretKey = "k******j"
  10. Region = DefaultRegion
  11. Bucket = "mybucket"
  12. Token = DefaultToken
  13. Endpoint = "http://127.0.0.1:8877"
  14. )
  15. var awsS3Instance = NewAwsS3(SecretId, SecretKey, Region, Bucket, Endpoint, Token)

 

上传文件 Put http://0.0.0.0:8877/mybucket/test/1.txt

  1. func TestPutObject(t *testing.T) {
  2. // 测试时修改本地路径
  3. localFilePath := "./testdata/1.txt"
  4. t.Logf("local file path %s", localFilePath)
  5. fileContent, err := os.ReadFile(localFilePath)
  6. if err != nil {
  7. t.Fatalf("read file error: %s!", err.Error())
  8. return
  9. }
  10. // 测试时修改aws路径
  11. awsPath := "/test/1.txt"
  12. _, err = awsS3Instance.PutObject(awsPath, fileContent)
  13. if err != nil {
  14. t.Fatalf("put object error: %s", err.Error())
  15. }
  16. t.Logf("put object success")
  17. }

 

 

下载文件 Get http://0.0.0.0:8877/mybucket/test/1.txt

  1. func TestGetObject(t *testing.T) {
  2. // 测试时修改aws路径
  3. awsPath := "/test/1.txt"
  4. contentBytes, _, err := awsS3Instance.GetObject(awsPath)
  5. if err != nil {
  6. t.Fatalf("get object error: %s", err.Error())
  7. }
  8. //获取当前系统根目录
  9. if err != nil {
  10. t.Fatalf("get home dir error: %s!", err.Error())
  11. return
  12. }
  13. // 测试时修改本地路径
  14. localFilePath := "./testdata/1-get.txt"
  15. err = os.WriteFile(localFilePath, contentBytes, 0644)
  16. if err != nil {
  17. t.Fatal("write error")
  18. return
  19. }
  20. t.Logf("get object success")
  21. }

 

 

Head文件 Head http://0.0.0.0:8877/mybucket/test/1.txt

  1. func TestHeadObject(t *testing.T) {
  2. // 测试时修改aws路径
  3. awsPath := "/test/1.txt"
  4. eTag, lastModifyTime, err := awsS3Instance.HeadObject(awsPath)
  5. if err != nil {
  6. t.Fatalf("head object error: %s", err.Error())
  7. }
  8. t.Logf("head object success,eTag : %s, lastModifyTime : %v", eTag, lastModifyTime)
  9. }

 

删除文件 Delete http://0.0.0.0:8877/mybucket/test/1.txt

  1. func TestDeleteObject(t *testing.T) {
  2. // 测试时修改aws路径
  3. awsPath := "/test/1.txt"
  4. err := awsS3Instance.DeleteObject(awsPath)
  5. if err != nil {
  6. t.Fatalf("delete object error: %s", err.Error())
  7. }
  8. t.Logf("delete object success")
  9. }

 

mc 监听桶 mybucket 的变化,可以看出

  1. ? minio mc watch myminio/mybucket/
  2. [2023-10-19T07:27:35.675Z] 36 B s3:ObjectCreated:Put http://0.0.0.0:8877/mybucket/test/1.txt
  3. [2023-10-19T07:28:46.813Z] s3:ObjectAccessed:Get http://0.0.0.0:8877/mybucket/test/1.txt
  4. [2023-10-19T07:28:58.157Z] s3:ObjectAccessed:Head http://0.0.0.0:8877/mybucket/test/1.txt
  5. [2023-10-19T08:42:23.065Z] s3:ObjectRemoved:Delete http://0.0.0.0:8877/mybucket/test/1.txt

 

minio控制台页面统计信息

 

详细文档参考

https://min.io/docs/minio/kubernetes/upstream/index.html?ref=docs-redirect&ref=con

done.

祝玩的开心~

原文链接:https://www.cnblogs.com/voipman/p/17774860.html

 友情链接:直通硅谷  点职佳  北美留学生论坛

本站QQ群:前端 618073944 | Java 606181507 | Python 626812652 | C/C++ 612253063 | 微信 634508462 | 苹果 692586424 | C#/.net 182808419 | PHP 305140648 | 运维 608723728

W3xue 的所有内容仅供测试,对任何法律问题及风险不承担任何责任。通过使用本站内容随之而来的风险与本站无关。
关于我们  |  意见建议  |  捐助我们  |  报错有奖  |  广告合作、友情链接(目前9元/月)请联系QQ:27243702 沸活量
皖ICP备17017327号-2 皖公网安备34020702000426号