FastDFS

fastDFS is an open source lightweight distributed file system, which can manage files. Its functions include:

• file store
• Automatic file synchronization
• File access (file upload and download, etc.)

It solves the problem of mass storage and load balancing, and is especially suitable for online services based on files, such as photo album websites, video websites, shopping malls, etc.

FastDFS is specially tailored for the Internet. It fully considers redundant backup, load balancing and capacity expansion mechanisms, and pays attention to high availability, high performance and other indicators. What is the purpose of using FastDFS to build a high-performance service cluster to provide services.

There are two core roles in FastDFS:

• Tracker

  Be responsible for scheduling and load balancing on access

• Storage node

  Document storage and document management operation. It provides interfaces for document storage, synchronization, extraction and so on

FastDFS will manage the metaData of the file. The so-called metaData refers to the relevant attributes of the file and the form of key value. If we upload a picture, the width and high attributes of the picture will be recorded as width = 888. metaData is the attribute list of the file, which contains multiple key value pairs.

Both the tracker and the storage node can be composed of one or more service servers, and the services of the tracker and the storage node should not want the normal operation of the service. All services in the tracker are equal and can be adjusted according to the pressure of the server.

In order to support the storage of high-capacity data, the storage node adopts the volume organization mode, and the storage system is composed of one or more volumes. The systems between volumes are independent of each other. In my opinion, the capacity accumulation of all volumes is all files of the whole storage system. Each volume can be composed of one or more servers. The files in all storage services in the volume are the same. Multiple services in the volume realize redundant backup and highly available load balancing. When a server is added to the volume, all files are synchronized automatically by the system. When the file synchronization is completed, the system will automatically put the server online to provide services.

When the storage space is insufficient, volumes can be added dynamically to expand the capacity. The file identifier in FastDFS consists of two parts

Volume name + file name

Installing FastDFS

1. Install gcc and c++

   [root@Test1 ~]# yum install gcc
   [root@Test1 ~]# yum install gcc -c++
   

2. Unzip v1 0.7 (some basic libraries for fastdfs)

   [root@Test1 local]# tar -zxvf V1.0.7.tar.gz
   

3. compile

   [root@Test1 ~]# cd libfastcommon-1.0.7
   [root@Test1 libfastcommon-1.0.7]# ./make.sh
   

4. install

   [root@Test1 libfastcommon-1.0.7]# ./make.sh install
   

5. Resume soft connection

   [root@Test1 libfastcommon-1.0.7]# ln -s /usr/lib64/libfastcommmon.so /usr/local/lib/libfafstcommon.so
   [root@Test1 libfastcommon-1.0.7]# ln -s /usr/lib64/libfastcommmon.so /usr/lib/libfafstcommon.so
   [root@Test1 libfastcommon-1.0.7]# ln -s /usr/lib64/libfdfsclient.so /usr/local/lib/libfdfsclient.so
   [root@Test1 libfastcommon-1.0.7]#  ln -s /usr/lib64/libfdfsclient.so /usr/lib/libfdfsclient.so
   

Install tracker

[root@Test1 local]# tar -zxvf V5.05.tar.gz 
[root@Test1 fastdfs-5.05]# ./make.sh
[root@Test1 fastdfs-5.05]# ./make.sh install
[root@Test1 fastdfs-5.05]# cd  /etc/fdfs/

# Copy the same file
[root@Test1 fdfs]# cp client.conf.sample  client.conf
[root@Test1 fdfs]# cp storage.conf.sample storage.conf
[root@Test1 fdfs]# cp tracker.conf.sample tracker.conf


[root@Test1 local]# cd /etc/fdfs
[root@Test1 fdfs]# ls
client.conf         storage.conf         tracker.conf
client.conf.sample  storage.conf.sample  tracker.conf.sample

# Modify profile
[root@Test1 fdfs]# vim tracker.conf
bind_addr=192.168.218.130
bese_path=/usrdata/fastdfs

[root@Test1 fdfs]# vim storage.conf
bind_addr=192.168.218.130
bese_path=/usrdata/fastdfs
store_path0=usrdata/fastdfs
tracker_server=192.168.218.130:22122

[root@Test1 fdfs]# vim client.conf
bese_path=/usrdata/fastdfs
tracker_server=192.168.218.130:22122

# Create directory
[root@Test1 fdfs]# mkdir /usrdata
[root@Test1 fdfs]# mkdir /usrdata/fastdfs
# Start service
[root@Test1 fdfs]# fdfs_trackerd /etc/fdfs/tracker.conf
[root@Test1 fdfs]# fdfs_storaged /etc/fdfs/storage.conf

## Installing nginx
# Unzip file
[root@Test1 local]# tar -zxvf fastdfs-nginx-module_v1.16.tar.gz

# Switch directory
[root@Test1 local]# cd fastdfs-nginx-module/src/

# Modify profile
[root@Test1 src]# vim config 
CORE_INCS="$CORE_INCS/usr/include/fastdfs/usr/include/fastcommon/"

# Copy file
[root@Test1 src]# cp mod_fastdfs.conf /etc/fdfs
# Edit file
[root@Test1 src]# vim /etc/fdfs/mod_fastdfs.conf 

base_path=/usrdata/fastdfs
tracker_server=192.168.218.130:22122
url_have_group_name = true
store_path0=/usrdata/fastdfs

# Establish soft connection
[root@Test1 src]# ln -s /usrdata/fastdfs/data /usrdata/fastdfs/data/M00

# Copy http related files to etc fdfs
[root@Test1 src]# cp /usr/local/fastdfs-5.05/conf/http.conf  /etc/fdfs/
[root@Test1 src]# cp /usr/local/fastdfs-5.05/conf/mime.types  /etc/fdfs/

## ngin installation
# Install pcre
[root@Test1 local]# tar -zxvf pcre-8.36.tar.gz 
[root@Test1 local]# cd pcre-8.36
[root@Test1 pcre-8.36]# ./configure
[root@Test1 pcre-8.36]# make && make install

inspect/lib64/Is there any in the next libpcre.so.1 If no soft connection is established(If there is, you don't have to execute this command)
[root@Test1 pcre-8.36]# ln -s /usr/local/lib/libpcre.so.1/lib64

# Install zlib
[root@Test1 local]# tar -zxvf zlib-1.2.8.tar.gz
[root@Test1 local]# cd zlib-1.2.8
[root@Test1 zlib-1.2.8]# ./configure
[root@Test1 zlib-1.2.8]# make && make install

# Install openssl
[root@Test1 zlib-1.2.8]# yum install openssl openssl-devel


## Formal installation of nginx
[root@Test1 local]# tar -zxvf nginx-1.8.0.tar.gz 
[root@Test1 local]# cd nginx-1.8.0
[root@Test1 nginx-1.8.0]# ./configure --prefix=/usr/local/ngin --add-module=/usr/local/fastdfs-nginx-module/src
[root@Test1 nginx-1.8.0]# make && make install

[root@Test1 local]# cd nginx/conf/
[root@Test1 conf]# vim nginx.conf

# Line 2
user  root;
# Insert the following on line 47
location /group1{
    root /usrdata/fastdfs/data;
    ngx_fastdfs_module;
}

# Install killall
[root@Test1 conf]# yum install psmisc

# start nginx 
[root@Test1 sbin]# ./nginx -c /usr/local/nginx/conf/nginx.conf

# close command
[root@Test1 sbin]# killall fdfs_trackerd
[root@Test1 sbin]# killall fdfs_storaged

# Start command
[root@Test1 sbin]# fdfs_trackerd /etc/fdfs/tracker.conf
[root@Test1 sbin]# fdfs_trackerd /etc/fdfs/storage.conf

# nginx related commands
# Profile startup
[root@Test1 sbin]# ./nginx -c /usr/local/ngin/conf/nginx.conf
# close command
[root@Test1 sbin]# ./nginx -s quit
[root@Test1 sbin]# ./nginx -s stop

# Normal start
[root@Test1 sbin]# service nginx restart

# Reload profile
[root@Test1 sbin]# ./nginx -s reload

# see
[root@Test1 sbin]# tail -f /usr/local/ngin/logs/error.log

# Test upload
[root@Test1 sbin]# /usr/bin/fdfs_test /etc/fdfs/client.conf upload /1.png