1. What is FastDFS

1. FastDFS Feature

• Files are not stored in blocks, and the uploaded files correspond to the files in the OS file system
• Save only one file that supports the same content, saving disk space
• Download files support the HTTP protocol and can be used with built-in Web Servers or with other Web Servers
• Support online expansion
• Supports master-slave files
• Storage server can save file properties (meta-data) V2.0 network communication uses libevent, supports large concurrent access, and overall performance is better

1. Concepts related to FastDFS

• Tracker Server: Track the server, mainly for scheduling and load balancing.All storage groups and state information of storage servers in the memory record cluster are the hub for interaction between clients and data servers.Master in GFS is more compact, does not record file index information, and takes up less memory.
• Storage Server: A storage server (also known as a storage node or data server) where files and file attributes (Meta Data) are saved.Storage Server calls management files directly from the OS's file system.
• Client: The client, as the originator of a business request, uses the TCP/IP protocol to interact with the tracker server or storage node through a proprietary interface.FastDFS provides users with basic file access interfaces such as upload, download, append, delete, and so on, which are provided as client libraries for users to use.

1. Upload mechanism

• First, the client requests the Tracker service to get the IP address and port of the storage server.
• Then the client requests to upload the file according to the returned IP address and port number.
• After the storage server receives the request, it produces the file and writes the file contents to disk and returns the file_id, path information, file name and other information to the client. The client saves the relevant information and uploads it.

1. Download mechanism

• The client requests the Tracker service with file name information to get the IP address and port of the storage server;
• The client then requests to download the file based on the IP address and port number returned, and the storage server returns the file to the client upon receiving the request.

2. Spring Boot Integrated FastDFS

The example project in this section builds on the previous section, uploads files from the front end to the back end, passes them directly to the FastDFS cluster, and returns the address of the file store.

POM Package Configuration

Java client with FastDFS:

<dependency>
    <groupId>org.csource</groupId>
    <artifactId>fastdfs-client-java</artifactId>
    <version>1.27-SNAPSHOT</version>
</dependency>

FastDFS Configuration

Add the fdfs_client.conf file to the project resources directory, which sets the connection timeout, encoding format, and tracker_server address.

connect_timeout = 60 # Connection timeout
network_timeout = 60 # Network Timeout
charset = UTF-8 # Encoding Format
http.tracker_http_port = 8080 #tracker port
http.anti_steal_token = no #token anti-theft chain function
http.secret_key = 123456 #secret key
# List of tracer servers, if multiple tracer servers, listed in separate lines
tracker_server = 192.168.53.85:22122
tracker_server = 192.168.53.86:22122

Encapsulate FastDFS Upload Tool Class

Encapsulates FastDFSFile, including file name, content, file type, author, and so on.

public class FastDFSFile {
    private String name;
    private byte[] content;
    private String ext;
    private String md5;
    private String author;
    //Omit getter, setter
}

FastDFSClient encapsulates the FastDFSClient class, which encapsulates the most basic operations, including upload, download, delete, and so on.The FastDFSFile task can be an encapsulation of FastDFS uploaded files, one instance for each file during operation.

The configuration information is read and initialized when the class is loaded.

static {
    try {
        String filePath = new ClassPathResource("fdfs_client.conf").getFile().getAbsolutePath();
	//The ClientGlobal.init method reads the configuration file and initializes the corresponding properties.
        ClientGlobal.init(filePath);
    } catch (Exception e) {
        logger.error("FastDFS Client Init Fail!",e);
    }
}

1. File Upload

• Use the client storageClient provided by FastDFS to upload the file and return the uploaded result.

public static String[] upload(FastDFSFile file) {
    logger.info("File Name: " + file.getName() + "File Length:" + file.getContent().length);

    //File Property Information
    NameValuePair[] meta_list = new NameValuePair[1];
    meta_list[0] = new NameValuePair("author", file.getAuthor());

    long startTime = System.currentTimeMillis();
    String[] uploadResults = null;
    StorageClient storageClient=null;
    try {
        //Obtain 
        storageClient = getStorageClient();
        //upload
        uploadResults = storageClient.upload_file(file.getContent(), file.getExt(), meta_list);
    } catch (IOException e) {
        logger.error("IO Exception when uploadind the file:" + file.getName(), e);
    } catch (Exception e) {
        logger.error("Non IO Exception when uploadind the file:" + file.getName(), e);
    }
    logger.info("upload_file time used:" + (System.currentTimeMillis() - startTime) + " ms");

    //Verify upload results
    if (uploadResults == null && storageClient!=null) {
        logger.error("upload file fail, error code:" + storageClient.getErrorCode());
    }
    //Successfully uploading the file returns groupName.
    logger.info("upload file successfully!!!" + "group_name:" + uploadResults[0] + ", remoteFileName:" + " " + uploadResults[1]);
    return uploadResults;
}

• NameValuePair, which stores some basic properties of the file, such as author information, creation time, etc.
• getStorageClient(), which encapsulates how to get the client.

Get TrackerServer information first, and use TrackerServer to build a client instance StorageClient for each operation.The detailed code is as follows:

private static StorageClient getStorageClient() throws IOException {
    TrackerServer trackerServer = getTrackerServer();
    StorageClient storageClient = new StorageClient(trackerServer, null);
    return  storageClient;
}

Below is an encapsulation of how to get TrackerServer:

private static TrackerServer getTrackerServer() throws IOException {
    TrackerClient trackerClient = new TrackerClient();
    TrackerServer trackerServer = trackerClient.getConnection();
    return  trackerServer;
}

1. get files

Get file information based on groupName and file name.Groups are also called volumes. Files on servers in the same group are identical. Storage servers in the same group are equal. File upload and deletion can be done on any one of the storage server s.

public static FileInfo getFile(String groupName, String remoteFileName) {
    try {
        storageClient = new StorageClient(trackerServer, storageServer);
        return storageClient.get_file_info(groupName, remoteFileName);
    } catch (IOException e) {
        logger.error("IO Exception: Get File from Fast DFS failed", e);
    } catch (Exception e) {
        logger.error("Non IO Exception: Get File from Fast DFS failed", e);
    }
    return null;
}

1. Download Files

Obtain the byte stream of the file based on the API of the storageClient and return:

public static InputStream downFile(String groupName, String remoteFileName) {
    try {
        StorageClient storageClient = getStorageClient();
        byte[] fileByte = storageClient.download_file(groupName, remoteFileName);
        InputStream ins = new ByteArrayInputStream(fileByte);
        return ins;
    } catch (IOException e) {
        logger.error("IO Exception: Get File from Fast DFS failed", e);
    } catch (Exception e) {
        logger.error("Non IO Exception: Get File from Fast DFS failed", e);
    }
    return null;
}

1. Delete Files

Delete the corresponding file based on the file name and group; when using FastDFS, call the FastDFSClient method directly.

public static void deleteFile(String groupName, String remoteFileName)
        throws Exception {
    StorageClient storageClient = getStorageClient();
    int i = storageClient.delete_file(groupName, remoteFileName);
    logger.info("delete file successfully!!!" + i);
}

Write Upload Control Class

Read the file information from MultipartFile, then use FastDFSClient to upload the file to the FastDFS cluster, encapsulate a saveFile() method to call the FastDFS tool class encapsulated above, upload the MultipartFile file to FastDFS, and return the address information of the uploaded file.

public String saveFile(MultipartFile multipartFile) throws IOException {
    String[] fileAbsolutePath={};
    String fileName=multipartFile.getOriginalFilename();
    String ext = fileName.substring(fileName.lastIndexOf(".") + 1);
    byte[] file_buff = null;
    InputStream inputStream=multipartFile.getInputStream();
    if(inputStream!=null){
        int len1 = inputStream.available();
        file_buff = new byte[len1];
        inputStream.read(file_buff);
    }
    inputStream.close();
    FastDFSFile file = new FastDFSFile(fileName, file_buff, ext);
    try {
        fileAbsolutePath = FastDFSClient.upload(file);  //upload to fastdfs
    } catch (Exception e) {
        logger.error("upload file Exception!",e);
    }
    if (fileAbsolutePath==null) {
        logger.error("upload file failed,please upload again!");
    }
    String path=FastDFSClient.getTrackerUrl()+fileAbsolutePath[0]+ "/"+fileAbsolutePath[1];
    return path;
}

When the upload request is passed to the back end, the above method saveFile() is called.

@PostMapping("/upload") 
public String singleFileUpload(@RequestParam("file") MultipartFile file,
                               RedirectAttributes redirectAttributes) {
    if (file.isEmpty()) {
        redirectAttributes.addFlashAttribute("message", "Please select a file to upload");
        return "redirect:uploadStatus";
    }
    try {
        String path=saveFile(file);
        redirectAttributes.addFlashAttribute("message",
                "You successfully uploaded '" + file.getOriginalFilename() + "'");
        redirectAttributes.addFlashAttribute("path","file path url '" + path + "'");
    } catch (Exception e) {
        logger.error("upload file failed",e);
    }
    return "redirect:/uploadStatus";
}

After successful upload, the path to the file is displayed on the page as follows:

3. Summary