1. fastDFS

fastDFS is an open source lightweight distributed file system developed in C language. It is generally used as a resource server

2. Main functions of fastdfs

• file store
• File synchronization
• File access (file upload | file download)
• It is especially suitable for online services based on documents (picture websites, video websites, shopping websites)

3. Roles in fastdfs

3.1 tracker server

• The tracking server is responsible for receiving requests from clients
• It is mainly used for scheduling and load balancing
• The information that the Tracker needs to manage is also stored in memory, and all trackers in it are peer-to-peer (each node has the same status), which is easy to expand
• When the client accesses the cluster, it will randomly assign a Tracker to interact with the client
• The storage server will send a heartbeat to the tracker server to keep the connection

3.2 storage server

Files and file attributes (metadata) are saved to the Storage server. The Storage server directly uses the OS file system call to manage files.

3.3 Group

• Groups, also known as volumes. The files on the servers in the same group are identical. The storage servers in the same group are peer-to-peer. File upload, deletion and other operations can be performed on any storage server.
• The data of machines in the same group is synchronized by themselves

3.4 meta data

meta data: File related attributes, key value pairs( Key Value Pair) Methods, such as: width=1024,heigth=768 . 

4. Advantages of fastdfs

• Massive storage: master-slave distributed storage, convenient expansion of storage space,
• fastDFS hash es file names to avoid duplicate files
• Then fastDFS is integrated with Nginx to improve the efficiency of the website
• It is especially suitable for online services with small and medium-sized files (recommended range: 4KB to 500MB), such as photo album websites, video websites, etc.

5. fastDFS data reading and writing process

5.1 fastDFS write data

During the write operation, storage will create two-level subdirectories under all the data storage directories it mounts, 256 at each level, with a total of 65536. The newly written files will be hash ed to one of the subdirectories, and then the file data will be stored in the directory as a local file

5.2 fastDFS read data

When the client sends a download request to the Tracker, it will not download directly, but first query the storage server (detect the synchronization status) and return the ip and port of the storage server,
Then the client will take the file information (group name, path, file name) to access the relevant storage, and then download the file.

[the external chain picture transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-A9UBzYnH-1628510437879)(assets/14027542-9b9dcd61bd1a9a57.png)]

6. Installation of the stand-alone version of fastdfs

6.1 preparation of installation documents

FastDFS_v5.08.tar.gz
fastdfs-nginx-module_v1.16.tar.gz
libfastcommon-master.zip
nginx-1.10.0.tar.gz

6.2 installation dependency

6.2. 1 install GCC dependency

GCC is used to compile and run C language code. Use the yum command to install:

 yum -y install gcc

6.2. 2. Install unzip tool

The unzip tool is used to A tool for extracting zip files

 yum install -y unzip zip

6.2. 3 install libevent

yum -y install libevent

6.2. 4 dependencies for installing Nginx

yum -y install pcre pcre-devel zlib zlib-devel openssl openssl-devel

6.2. 5. Install libfastcommon master

• Unzip the libfastcommon master zip

  unzip libfastcommon-master.zip
  

• Enter the extracted Directory:

  cd libfastcommon-master
  

• Compile and install:

  ./make.sh 
  ./make.sh install
  

So far, all dependencies have been installed (/ usr/lib64/libfastcommon.so /usr/lib/libfastcommon.so). Next, let's install FastDFS:

6.3 installing stand-alone fastDFS

6.3. 1 compile and install fastDFS

• decompression

  tar -zxvf FastDFS_v5.08.tar.gz  -C /opt/
  

• Enter directory

  cd FastDFS
  

• Compile and install

  ./make.sh 
  ./make.sh install
  

• After the installation is completed, we should be able to install it in / etc / init D / directory through the command ll / etc / init D / | grep FDFS sees the startup script provided by FastDFS:

  fdfs_trackerd #tracker startup script
  fdfs_storaged #storage startup script
  

• We can view the following configuration file template through the command in the / etc/fdfs directory

  • tarcker.conf.sample is the configuration file template of the tracker
  • storage.conf.sample is the configuration file template for storage
  • client.conf.sample is the configuration file template for the client

6.3. 2. Configure tracker

• Edit tracker configuration

First, we assign and rename the template file:

cp tracker.conf.sample tracker.conf
vim tracker.conf

Open tracker Conf, modify base_path configuration:

#bind_addr = if not configured, it means listening to all addresses
bind_addr=
base_path=/var/fdfs/tracker # Data and log storage directory of tracker

• Create directory

  The directory just configured may not exist. Let's create it

  mkdir -p  /var/fdfs/tracker
  

• Start tracker

  We can use SH / etc / init d/fdfs_ Tracker is started, but FDFS has been set as a system service during installation. We can use the familiar service startup method:

  service fdfs_trackerd start # Start FDFS_ Tracker service, stop using stop
  systemctl start fdfs_trackerd #Startup mode of centeros7
  

6.3. 3. Configure storage

• Edit storage configuration

  First, we assign and rename the template file:

  cp storage.conf.sample storage.conf
  vim storage.conf
  

  Open storage Conf, modify base_path configuration:

  base_path=/var/fdfs/storage # Storage data and log storage directory
  store_path0=/var/fdfs/storage_data # Storage upload file storage path
  tracker_server=10.10.10.11:22122 # tracker's address
  

• Create directory

  The directory just configured may not exist. We create it. If there is no directory, the startup will fail

  mkdir -p /var/fdfs/storage
  mkdir -p /var/fdfs/storage_data
  

• Start storage

  We can use SH / etc / init d/fdfs_ For stored startup, we can also use the service startup method:

  service fdfs_storaged start  # Start FDFS_ Stored service, stop
  

  Finally, view the process through netstat -nltp|grep fdfs:

  tcp        0      0 0.0.0.0:22122           0.0.0.0:*               LISTEN      2192/fdfs_trackerd  
  tcp        0      0 0.0.0.0:23000           0.0.0.0:*               LISTEN      2215/fdfs_storaged
  

6.4 installation of FastDFS module of nginx

6.4. 1. Nginx module of fastdfs

• decompression

  tar -zxvf fastdfs-nginx-module_v1.16.tar.gz
  

• Configuration config file (Library in fdfs to be used when installing fastdfs nginx module)

  # Enter the configuration directory
  cd /soft/fastdfs-nginx-module/src/
  # Modify configuration
  vim config
  # Execute the following command (change / usr/local in the configuration to / USR):
  :%s+/usr/local/+/usr/+g
  

• Configure mod_fastdfs.conf

  # Edit the file vim /etc/fdfs/mod_fastdfs.cof
  connect_timeout=10                  # Timeout length of client access file connection (unit: seconds)
  tracker_server=192.168.56.101:22122  	# tracker service IP and port
  url_have_group_name=true            		# Access link prefix plus group name
  store_path0=/var/fdfs/storage_data       		# File storage path
  

• Set mod in src directory_ fastdfs. Copy conf to the / etc/fdfs directory

  cp mod_fastdfs.conf  /etc/fdfs/
  

• Copy some configuration file templates of FastDFS to the / etc/fdfs directory

  cd /opt/fdfs/conf
  cp http.conf mime.types /etc/fdfs/
  

6.4. 2 install Nginx

• decompression

  tar -zxvf nginx-1.10.0.tar.gz
  

• to configure

  ./configure --prefix=/opt/nginx  --add-module=/soft/fastdfs-nginx-module/src
  

• Compile and install

  make &&  make install
  

• Configure nginx integration fastdfs module

  If you need to modify the nginx configuration file, go to / opt / nginx / config / nginx In the conf file:

  vim  /opt/nginx/conf/nginx.conf
  

  Add an agent for the server:

  server {
      listen       80;
      server_name  image.leyou.com;
  
      # Listen for group s in the domain name and submit them to the FastDFS module for processing
      location ~/group([0-9])/ {
      ngx_fastdfs_module;
      }
  }
  

• start-up

  nginx # start-up
  nginx -s stop # stop it
  nginx -s reload # Reload configuration
  

7. Use the fastDFS command to operate fastDFS

• Configure / etc / FDFS / client conf

  #Temporary data store directory for client
  base_path=/var/fdfs/client
  #Service address of tracker
  tracker_server=tracker of Ip:22122
  

• Use command

  #Upload file
  fdfs_upload_file  /etc/fdfs/client.conf  xxx file
  
  eg: 
  	fdfs_upload_file /etc/fdfs/client.conf  /soft/nginx-1.10.0.tar.gz
  
  return: group1/M00/00/00/CgoKC14KxIOAK7WrAA3emu5kyOM.tar.gz
    group1:Group information
    M00:corresponding store_path0
    /00/00:Disk path 
  group1/M00/00/00/CgoKFWEPit6Af4KgAA3emu5kyOM.tar.gz
  #Download File
  fdfs_download_file /etc/fdfs/client.conf 
  

8. Establishment of fastdfs cluster

• Copy the fastdfs installed in the stand-alone version to other machines
• If there are multiple groups, you only need to modify the goupname of the corresponding storage
• Other machines also need to install nginx and its fastdfs module (the old version of fastdfs provides http services, but the new version no longer provides services. We need to provide http services through nginx and its fastdfs module)
• The http service ports provided by nginx on storage and tracker should be consistent

9. fastDFS api operation

• Take the SpringBoot operation as an example

  <!--because fast Provided by the author fastdfs of api Not very friendly,So we use goyhub Open source implementations by other authors-->
  <dependency>
      <groupId>com.github.tobato</groupId>
      <artifactId>fastdfs-client</artifactId>
      <version>1.26.5</version>
  </dependency>
  

• The startup class uses the Import annotation to Import the configuration class of FastDfs

  @Import(FdfsClientConfig.class)
  public class LyItemApplication {
      public static void main(String[] args) {
          SpringApplication.run(LyItemApplication.class, args);
      }
  }
  

• The configuration file configures the configuration of fastdfs

  fdfs:
    tracker-list:
      - 10.10.10.11:22122
  

• Upload files using fastdfs api

  @SpringBootTest()
  @RunWith(SpringRunner.class)
  public class MyTest {
  
      @Autowired
      private FastFileStorageClient storageClient;
  
      @Test
      public void test01() throws FileNotFoundException {
          File file = new File("/f:/1.png");
          FileInputStream fis = new FileInputStream(file);
          StorePath storePath = storageClient.uploadFile(fis, file.length(), FilenameUtils.getExtension(file.getName()), null);
          System.out.println(storePath.getFullPath());
      }
  }