Use of Tencent cloud cosfs tool and automatic mount after startup

In order to provide file access and CDN acceleration for multiple servers, I chose to use COS to store files. However, the official boot auto mount scheme is not stable. I study the use of cosfs tool and cooperate with systemctl to realize boot auto mount.

Compared with the official / etc/fstab mount scheme, using systemctl to manage cosfs process is more stable, because systemctl provides automatic restart function, that is, cosfs process will start automatically after hanging.

Using the / etc/fstab mount scheme, if the file is modified too frequently, the mount will fail. The performance of the two has not been specifically tested, but there should be no big difference.

Install cosfs

I use Ubuntu 20.04 here. If you use other systems, you can refer to it Official documents Install.

Download package

wget https://github.com/tencentyun/cosfs/releases/download/v1.0.19/cosfs_1.0.19-ubuntu20.04_amd64.deb

Installation package

sudo dpkg -i cosfs_1.0.19-ubuntu20.04_amd64.deb

Edit key file

sudo nano /etc/passwd-cosfs

Key file content

<BucketName-APPID>:<SecretId>:<SecretKey>

< bucketname appid > is the format of bucket name. < secret ID > and < secret key > are key information. You can go to the Cloud API key management View and create in.

Modify key file permissions

sudo chmod 640 /etc/passwd-cosfs
sudo chown www:www /etc/passwd-cosfs

Because I mount cos for the website, I use www users to mount it. The later chown is used for this purpose

Create mount directory

When mounting cos, the directory to be mounted exists, so we create it in advance

sudo mkdir -p /mnt/cosfs
sudo chown www:www /mnt/cosfs

Create cache directory

sudo mkdir -p /var/cache/cosfs
sudo chown www:www /var/cache/cosfs

Edit fuse conf

Edit the fuse configuration to allow other users to access the mounted files

sudo nano /etc/fuse.conf

Find user_allow_other, if it is commented, the comment will be cancelled. If it is not configured, a new line of configuration will be added.

user_allow_other

If it is not set, other users will not be able to access the mounted directory fusermount: option allow_other only allowed if 'user_allow_other' is set in /etc/fuse.conf

Set systemctl service

Edit service file

sudo nano /etc/systemd/system/cosfs.service
[Unit]
Description=cosfs Service
After=network.target

[Service]
Type=simple
User=www
Group=www
Restart=on-failure
RestartSec=5s
ExecStart=cosfs -f cos-1251274180 /mnt/cosfs -ourl=http://cos.ap-guangzhou.myqcloud.com -odbglevel=info -oallow_other -ouse_cache=/var/cache/cosfs

[Install]
WantedBy=multi-user.target

cos-1251274180 is the name of the storage bucket /mnt/cosfs is the local mount directory -ourl is the access domain name. See Region and access domain name -ogid user group ID -ouid user ID -ouse_cache directory

Common mount options

-omultipart_size=[size]
Used to specify the size of a single block during block upload (unit: MB),The default is 10 MB.  Since block upload has the maximum limit on the number of single file blocks (10000 blocks), for more than 100 blocks GB(10MB * 10000)This parameter needs to be adjusted according to the specific situation.

-oallow_other
 If you want to allow other users to access the mount folder, you can run COSFS Specify this parameter when.

-odel_cache
 By default, COSFS Tools to optimize performance umount After, the local cache data will not be cleared. If necessary COSFS When exiting, the cache will be cleared automatically. You can add this option when mounting.

-onoxattr
 Disable getattr/setxattr Function, in 1.0.9 Previous versions COSFS Setting and obtaining extended properties are not supported. If it is used during mounting use_xattr Options that may cause mv File to Bucket Failed.

-opasswd_file=[path]
This option can be specified COSFS The path of the key file. The key file set by this option needs to be set to 600.

-odbglevel=[dbg|info|warn|err|crit]
set up COSFS Logging level, optional info,dbg,warn,err and crit. Recommended setting in production environment info,It can be set to dbg. If your system logs are not cleaned up regularly and a large number of logs are generated due to heavy traffic, you can set it to err perhaps crit. 

-oumask=[perm]
This option can remove the operation permission of a given type of user on the files in the mounted directory. For example,-oumask=755,The permission of the corresponding mount directory changes to 022.

-ouid=[uid]
This option allows the user to id by [uid] The user of is not limited by the file permission bit in the mount directory and can access all files in the mount directory.
Get user uid have access to id Commands, formatting id -u username. For example, execution id -u user_00,Get user user_00 of uid. 

-oensure_diskfree=[size]
COSFS In order to improve the performance of the tool, the system disk is used by default to store the temporary cache of upload and download, and the space will be released after the file is closed. When a large number of files are opened concurrently or large files are read and written, COSFS The tool will use the hard disk as much as possible to improve the performance. By default, only 100 will be reserved MB The free space of the hard disk can be used by other programs through options oensure_diskfree=[size] set up COSFS The amount of free hard disk space reserved by the tool, in MB. for example-oensure_diskfree=1024,COSFS The tool retains 1024 MB Remaining space.

Startup and installation services

At this point, we try to start the service

sudo systemctl start cosfs

Then view the service status

sudo systemctl status cosfs

If the status of Active is active (running), it means that the service is running normally.

Reference operation status:

cosfs.service - cosfs Service
     Loaded: loaded (/etc/systemd/system/cosfs.service; disabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-02-17 17:36:59 CST; 20s ago
   Main PID: 9689 (cosfs)
      Tasks: 3 (limit: 2328)
     Memory: 2.8M
     CGroup: /system.slice/cosfs.service
             └─9689 /usr/local/bin/cosfs -f cos-1251274180 /mnt/cosfs -ourl=http://cos.ap-guangzhou.myqcloud.com -odbglevel=info -ogid=1002 -ouid=1002 -ouse_cache=/tmp/cosfs

Feb 17 17:36:59 VM-8-16-ubuntu systemd[1]: cosfs.service: Scheduled restart job, restart counter is at 37.
Feb 17 17:36:59 VM-8-16-ubuntu systemd[1]: Stopped cosfs Service.
Feb 17 17:36:59 VM-8-16-ubuntu systemd[1]: Started cosfs Service.
Feb 17 17:36:59 VM-8-16-ubuntu s3fs[9689]: [tid:9689]s3fs.cpp:main(5001): init v1.0.19(commit:unknown) with OpenSSL

Then we can start the service directly, and the service will start automatically when it is started.

sudo systemctl enable cosfs

If the status is abnormal, please see the following debugging service

Link file

If we need to mount a file or directory in cos to the website, we can use soft links.

For example: mount the files directory in cos to / www/wwwroot/default in the website directory, you can use the following command

sudo ln -s /mnt/cosfs/files /www/wwwroot/default/files

Commissioning services

If the service cannot run normally, you can use the command to view the service operation log, and then adjust the configuration and service according to the log information.

journalctl -f -u cosfs.service

If you modify the service, you need to reload the service and restart the service before it takes effect

sudo systemctl daemon-reload
sudo systemctl restart cosfs

Reference link

This article is synchronously published to Zero blog.

Finally, there is a welfare. Developers, welcome to join the Tengyun pioneer (TDP) feedback exchange group. There are rich activities in the group, which can harvest points and growth value and exchange for surprise benefits. Joining method: https://cloud.tencent.com/developer/article/1855195

We are the team of Tencent cloud pioneer (TDP) and the technical developer Group officially established and operated by Tencent cloud GTS. There are the most professional developers & customers, who can have close contact with product personnel, proprietary Problem & demand feedback channels, and a group of like-minded brothers and sisters. We look forward to your joining us!

Added by duny on Fri, 18 Feb 2022 11:22:00 +0200