preface
MinIO Gateway is a software that can agent S3, Azure, Nas, HDFS and other services. Users can access the proxy service in an S3 compatible manner.
For details, see: https://docs.min.io/docs/minio-gateway-for-s3.html
Usage scenario
1, A set of code supports different object storage products.
Common object storage products on the market include:
- Alibaba cloud OSS
- Tencent cloud COS
- Huawei cloud OBS
- Amazon S3
- Open source MinIO
If your service needs to use object storage, but the object storage services used in different scenarios are inconsistent, in order to avoid increasing the complexity of adapting multiple products in code development,
MinIO Gateway can be used as a Layer-1 agent. The code only needs to support the access mode of MinIO.
2, Avoid opening a large number of cloud service sub accounts.
Within the company, when we use the object storage products of cloud service providers, we must open a sub account of cloud products to access. The opening and management of sub accounts are not
Very convenient, and there is no way to access account management systems such as LDAP. At this time, you can also use MinIO Gateway as a Layer-1 agent for the object storage service, and then use the Gateway
To manage the account of object storage service, and support various account management methods. For example, keyloak, LDAP, internal users, etc.
For details, see: https://docs.min.io/docs/minio-sts-quickstart-guide
Construction of MinIO Gateway
Refer to the official documents for the basic construction steps: https://docs.min.io/docs/minio-gateway-for-s3.html
You can start the Gateway service by using docker or binary files.
Let's say something different
MinIO compilation
The compilation is very simple. MinIO is developed in Go language, so Go needs to be installed. See: https://go.dev/doc/install
Then there is already a written Makefile in the source code. Just execute make directly.
Adapt to Tencent cloud COS
MinIO Gateway will randomly generate a Bucket name when starting, and then use the API to check whether the Bucket exists to confirm whether the S3 service is available. The expected return status code is 404,
However, Tencent cloud COS forces the name suffix of each Bucket to be a string of Digital ID s. If it does not conform to the format, it will respond to 400, resulting in the failure of MinIO Gateway detection.
The log is as follows:
""" ---------START-HTTP--------- GET /probe-bucket-sign-99lrqve1qm4x/?location= HTTP/1.1 Host: cos.ap-beijing.myqcloud.com User-Agent: MinIO (darwin; amd64) minio-go/v7.0.20 Authorization: AWS4-HMAC-SHA256 Credential=AKID2uWrwlJabKnzwd3CCwPbWBZhZBWZLr64/20220118/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED** X-Amz-Content-Sha256: UNSIGNED-PAYLOAD X-Amz-Date: 20220118T155622Z Accept-Encoding: gzip HTTP/1.1 400 Bad Request Content-Length: 437 Connection: keep-alive Content-Type: application/xml Date: Tue, 18 Jan 2022 15:56:25 GMT Server: tencent-cos <?xml version='1.0' encoding='utf-8' ?> <Error> <Code>InvalidURI</Code> <Message>Could not parse the specified URI.</Message> <Resource>cos.ap-beijing.myqcloud.com/probe-bucket-sign-99lrqve1qm4x</Resource> </Error> ---------END-HTTP--------- """
In the case that neither the official of MinIO nor the official of Tencent cloud make any modification, the problem can only be solved by modifying the source code and recompiling.
The location of code modification is as follows:
https://github.com/minio/minio/cmd/gateway/s3/gateway-s3.go
Modify the return value of the randString function to a numeric suffix, as shown in the following figure:
Then compile it.
Use independent account system
By default, MinIO comes with a set of account management system without any configuration, but the disadvantage is that once the service is restarted, the account information will be lost.
In order to store account data persistently, it is necessary to cooperate with Etcd service.
For Etcd deployment, see:
https://github.com/etcd-io/etcd/releases
The startup mode of MinIO Gateway is as follows:
#!/bin/sh export MINIO_ROOT_USER="Access Key" export MINIO_ROOT_PASSWORD="Access Secret" export _MINIO_SERVER_DEBUG=off # Whether DEBUG is enabled only to view the log on|off export MINIO_ETCD_ENDPOINTS=http://localhost:2379 # use ETCD to persist internal users export MINIO_ETCD_PATH_PREFIX=minio/ # Prefix of data stored in ETCD ./minio gateway s3 https://cos.ap-beijing.myqcloud.com --console-address 0.0.0.0:9100
However, this method has a disadvantage. The newly created MinIO Gateway account secret is stored in Etcd in clear text...
The solution is as follows:
-
Control the access rights of Etcd to avoid being accessed by others.
-
Use MinIO's own encryption method, refer to: https://docs.min.io/docs/minio-kms-quickstart-guide.html . However, there are a lot of places to encrypt this function, and there are some small bugs. For example, the list of S3 files displayed is incomplete... So I gave up.
Integrated LDAP
Official documents: https://github.com/minio/minio/blob/master/docs/sts/ldap.md
The startup method is as follows:
#!/bin/sh
export MINIO_ROOT_USER="Access Key"
export MINIO_ROOT_PASSWORD="Access Secret"
export _MINIO_SERVER_DEBUG=off # Whether DEBUG is enabled only to view the log on|off
export MINIO_IDENTITY_LDAP_SERVER_ADDR="LDAP The server:LDAP Service port"
export MINIO_IDENTITY_LDAP_LOOKUP_BIND_DN="{LDAP accounts}" # For example, cn=readonly,dc=test,dc=com, just use the read-only account
export MINIO_IDENTITY_LDAP_LOOKUP_BIND_PASSWORD="{LDAP password}"
export MINIO_IDENTITY_LDAP_USER_DN_SEARCH_BASE_DN='ou=People,dc=test,dc=com' # Search domain
export MINIO_IDENTITY_LDAP_USER_DN_SEARCH_FILTER='(uid=%s)' # The account% s used to filter login will be filled with user name
export MINIO_IDENTITY_LDAP_TLS_SKIP_VERIFY=on
export MINIO_IDENTITY_LDAP_SERVER_INSECURE=on
export MINIO_IDENTITY_LDAP_SERVER_STARTTLS=off
./minio gateway s3 https://cos.ap-beijing.myqcloud.com --console-address 0.0.0.0:9100
The advantage of integrating LDAP is that it can be used without opening an account separately, but the trouble is that users can only be authorized through the command line mc. Moreover, when it is necessary to open an account for external personnel of the company, they must also open an LDAP account, which may face certain risks.
Integrated Audit Log function (Audit Log)
By default, when only the Gateway function of MinIO is used, the Admin API and many features such as Audit log are not open.
At this time, it's time to change the code and recompile.
First, enable the Admin API, and modify the code location as follows:
https://github.com/minio/minio/cmd/gateway-main.go
Change the place shown in the figure below to true:
Then recompile.
The following is to open the Audit log. See: https://github.com/minio/operator/tree/master/logsearchapi
Follow the documentation steps to start the postgres database and then start the logsearch API service. These are all OK.
postgres startup:
docker run -d \
--name postgres \
-p 5432:5432 \
-e POSTGRES_PASSWORD="xxx" \
-e PGDATA=/var/lib/postgresql/data/pgdata \
-v /data/postgres:/var/lib/postgresql/data \
postgres:14.1 -c "log_statement=all"
Start logsearch API (note that the logsearch API command also needs to download the source code and compile it)
export LOGSEARCH_PG_CONN_STR="postgres://postgres:xxx@localhost/postgres?sslmode=disable" export LOGSEARCH_AUDIT_AUTH_TOKEN=logsearch_audit export MINIO_LOG_QUERY_AUTH_TOKEN=logsearch_query export LOGSEARCH_DISK_CAPACITY_GB=5 ./logsearchapi
But when I try to set aduit using the mc admin command_ When the webhook parameter is, inexplicable errors will appear. Give up after a try and use other methods.
Add the following code into the startup script:
# audit log function export MINIO_AUDIT_WEBHOOK_ENABLE_1="on" # audit log function export MINIO_AUDIT_WEBHOOK_ENDPOINT_1="http://localhost:8080/api/ingest? token=logsearch_ Audit "# audit log function export MINIO_LOG_QUERY_URL="http://localhost:8080" export MINIO_LOG_QUERY_AUTH_TOKEN="logsearch_query" export LOGSEARCH_QUERY_AUTH_TOKEN="logsearch_query"