Breakpoint continuation of large files based on elementui

1, Pain points of uploading large files at the front end

1. Too many files lead to tight bandwidth resources and reduced request speed;
2. If the service is interrupted, the network is interrupted and the page crashes during the upload process, the file may be uploaded again.

2, Analysis of pain points

The front end selects the file and uploads it. In the process of processing the file, the back end will first load the file into the running memory, and then call the corresponding API to write it into the hard disk memory to complete the upload of the whole file.

However, uploading files directly in this way may lead to an avalanche of the whole process due to a problem in a certain link, so it is not advisable to upload large files directly.

For example, the best way to solve the problem is to transfer the large file to 100M.

3, Principle of breakpoint continuation

As the name suggests, it is breakpoint and sequel

What is a breakpoint?

At this time, the upload process will be interrupted by N threads, and then the upload process will be interrupted by one or more concurrent breakpoints.

Every time the front-end uploads a piece, it will be loaded into the running memory, and then written to the hard disk. At this time, the temporary variable of the running memory will be released, and then this temporary variable will be occupied by the next piece, and then written and released

What is a sequel?

It means to continue to upload the remaining files from the interrupted location, rather than from the beginning.

After uploading, merge at the server (the merging operation is carried out at the back end, and the front end only calls the interface. The merging method is determined by the back end. Whether to merge one piece after uploading, or to merge all the pieces after uploading).

Implementation of breakpoint continuation

1) Implementation of slicing

The way before html5z was flash and activeX
HTML 5 provides slice method to split binary stream of files.

const chunks = Math.ceil(file.size / eachSize)

The fragmentation of documents is generally between 2-5M. This step obtains the content of each piece of file, the serial number of each block, the size of each block, the total number of blocks and other data.

2) Implementation of continuous transmission

  1. To continue transmission, we must first determine which file needs to be uploaded. The way to determine a file is to encrypt the file. As long as the content of a file changes, it needs to be uploaded again. MD5 encryption can be applied to the file as the unique identifier of the file. MD5 encryption is irreversible.
    Note: encrypting the entire large file may cause the page to crash, and the file needs to be encrypted in pieces.
    spark-md5 plug-in supports file fragment encryption.
    Use of spark-md5 based on elementUI

  2. Before uploading the first piece of file, you need to use the file name + the unique identifier of this file + the current number of pieces to query whether the file has been uploaded. Through the uploaded results returned by the server, we can obtain the rest through the segmentation results for uploading. If the page is reloaded or the upload is interrupted, you can continue the upload only by interrupting in which piece before re uploading.

  3. After uploading, request the merge interface (the merge interface can also not be requested, and the back-end will merge all the files by itself after getting all the files), merge the files at the server, and then the whole file upload is completed.

  4. In the process of uploading, the front-end progress bar can be displayed according to the number of successfully uploaded films and the total number of films returned by the server.

Use of Upload spark-md5 in element UI

//js part
import chunkedUpload from './chunkedUpload'
export default {
  data() {
    return {
      uploadData: {
        //There are additional parameters in it
      //File upload path
      uploadUrl: process.env.BASE_API + '/oss/oss/uploadChunkFile', //File upload path
      chunkedUpload: chunkedUpload // The customized method of fragment upload is introduced in the header
  methods: {
    onError(err, file, fileList) {
      this.$store.getters.chunkedUploadXhr.forEach(item => {
      this.$alert('File upload failed, please try again', 'error', {
        confirmButtonText: 'determine'
    beforeRemove(file) {
      // If you are uploading in pieces, cancel the uploading in pieces
      if (file.percentage !== 100) {
        this.$store.getters.chunkedUploadXhr.forEach(item => {
import SparkMD5 from 'spark-md5'
import axios from 'axios'
import store from '@/store'
// If the upload is wrong, get the error information
function getError(action, option, xhr) {
  let msg
  if (xhr.response) {
    msg = `${xhr.response.error || xhr.response}`
  } else if (xhr.responseText) {
    msg = `${xhr.responseText}`
  } else {
    msg = `fail to post ${action} ${xhr.status}`
  const err = new Error(msg)
  err.status = xhr.status
  err.method = 'post'
  err.url = action
  return err
// After the upload completes the merge successfully, get the information returned by the server
function getBody(xhr) {
  const text = xhr.responseText || xhr.response
  if (!text) {
    return text
  try {
    return JSON.parse(text)
  } catch (e) {
    return text

// For the custom request of fragment upload, the following requests will override the default upload behavior of element
export default function upload(option) {
  if (typeof XMLHttpRequest === 'undefined') {
  const spark = new SparkMD5.ArrayBuffer()// ArrayBuffer encryption class of md5
  const fileReader = new FileReader()// File reading class
  const action = option.action // File upload path
  const chunkSize = 1024 * 1024 * 30 // Single slice size
  let md5 = ''// Unique identification of the file
  const optionFile = option.file // Files requiring fragmentation
  let fileChunkedList = [] // Array after file fragmentation
  const percentage = [] // An array of file upload progress. A single item is a piece of progress

  // Start slicing the file, push it into the fileChunkedList array, and use the first slicing to calculate the md5 of the file
  for (let i = 0; i < optionFile.size; i = i + chunkSize) {
    const tmp = optionFile.slice(i, Math.min((i + chunkSize), optionFile.size))
    if (i === 0) {

  // After reading the file, start to calculate the file md5 as the unique ID of the file
  fileReader.onload = async(e) => {
    md5 = spark.end() + new Date().getTime()
    console.log('file md5 by--------', md5)
    // Convert fileChunkedList into FormData object and add the data required for uploading
    fileChunkedList =, index) => {
      const formData = new FormData()
      if ( {
        // Add additional incoming data from outside
        Object.keys( => {
        // These fields can be transferred according to what the backend needs. You can also add additional parameters yourself
        formData.append(option.filename, item, file
        formData.append('chunkNumber', index + 1)// Current file block
        formData.append('chunkSize', chunkSize)// Single block size
        formData.append('currentChunkSize', item.size)// Current block size
        formData.append('totalSize', optionFile.size)// Total file size
        formData.append('identifier', md5)// Document identification
        formData.append('filename', file name
        formData.append('totalChunks', fileChunkedList.length)// Total number of blocks
      return { formData: formData, index: index }

    // How to update the percentage of upload progress bar
    const updataPercentage = (e) => {
      let loaded = 0// Total size of uploaded files
      percentage.forEach(item => {
        loaded += item
      e.percent = loaded / optionFile.size * 100

    // Create queue upload tasks. limit is the number of concurrent uploads
    function sendRequest(chunks, limit = 3) {
      return new Promise((resolve, reject) => {
        const len = chunks.length
        let counter = 0
        let isStop = false
        const start = async() => {
          if (isStop) {
          const item = chunks.shift()
          if (item) {
            const xhr = new XMLHttpRequest()
            const index = item.index
            // Fragment upload failure callback
            xhr.onerror = function error(e) {
              isStop = true
            // Fragment upload successful callback
            xhr.onload = function onload() {
              if (xhr.status < 200 || xhr.status >= 300) {
                isStop = true
                reject(getError(action, option, xhr))
              if (counter === len - 1) {
                // The last upload is complete
              } else {
            // Callback during fragment upload
            if (xhr.upload) {
              xhr.upload.onprogress = function progress(e) {
                if ( > 0) {
                  e.percent = e.loaded / * 100
                percentage[index] = e.loaded
  'post', action, true)
            if (option.withCredentials && 'withCredentials' in xhr) {
              xhr.withCredentials = true
            const headers = option.headers || {}
            for (const item in headers) {
              if (headers.hasOwnProperty(item) && headers[item] !== null) {
                xhr.setRequestHeader(item, headers[item])
            // File start upload
            //Here is to save all the xhr uploaded in pieces to the global. If the user cancels the upload manually or there is an error in the upload, you need to call xhr Abort() stops all xhr in the store, otherwise the file will continue to upload
            store.commit('SET_CHUNKEDUPLOADXHR', xhr)
        while (limit > 0) {
          setTimeout(() => {
          }, Math.random() * 1000)
          limit -= 1

    try {
      // Call the upload queue method and wait for all files to be uploaded
      await sendRequest(fileChunkedList, 3)
      // The parameters here are written according to your actual situation
      const data = {
        identifier: md5,
        totalSize: optionFile.size
      // Send file merge request to backend
      const fileInfo = await axios({
        method: 'post',
        url: '/api/oss/oss/mergeChunkFile',
        data: data
      // This 8200 is the code that we successfully stored in oss. It can be changed according to our actual situation
      if ( === 8200) {
        const success = getBody(fileInfo.request)
    } catch (error) {

Front end through spark-md5 JS calculates the local file MD5

Two methods are provided here; One is to use sparkmd5 Hashbinary () directly transfers the binary code of the whole file into md5 and directly returns the file. This method has advantages for small files - simple and fast.
Another method is to use the slice() method of the File object in js (File.prototype.slice()) to slice the File and then transfer it to spark. JSP one by one Appendbinary() method, and finally through spark The end () method outputs the results. Obviously, this method will be very beneficial for large files - it is not error prone and can provide the progress information of the calculation

The first way:

 var running = false;    //running is used to determine whether md5 is being calculated
            function doNormalTest( input ) {    //It is assumed that the dom reference of the file selection box is passed in directly
                if (running) {    // If the calculation is in progress, the next calculation is not allowed
                var fileReader = new FileReader(),    //Create FileReader instance
                fileReader.onload = function (e) {    //The load event of FileReader is triggered when the file is read
                    running = false;
                    // points to the fileReader instance above
                    if (file.size != {    //If the two are inconsistent, it indicates that there is an error in reading
                       alert("ERROR:Browser reported success but could not read the file until the end.");
                    } else {
                        console.log(Finished loading!success!!);
                         return SparkMD5.hashBinary(;    //Calculate md5 and return the result
                fileReader.onerror = function () {    //If there is an error reading the file, cancel the reading status and pop-up an error
                    running = false;
                    alert("ERROR:FileReader onerror was triggered, maybe the browser aborted due to high memory usage.");
                running = true;
                fileReader.readAsBinaryString( input.files[0] );    //Read file binary code through fileReader

The second way

function doIncrementalTest( input ) {    //It is assumed that the dom reference of the file selection box is passed in directly
                if (running) {
                //The slice() method of File needs to be used here. The following is the compatible writing method
                var blobSlice = File.prototype.slice || File.prototype.mozSlice || File.prototype.webkitSlice,
                    file = input.files[0],
                    chunkSize = 2097152,                           // Read one by one with the size of 2MB per slice
                    chunks = Math.ceil(file.size / chunkSize),
                    currentChunk = 0,
                    spark = new SparkMD5(),    //Create an instance of SparkMD5
                    fileReader = new FileReader();
                fileReader.onload = function (e) {
                    console("Read chunk number (currentChunk + 1) of  chunks ");
                    spark.appendBinary(;                 // append array buffer
                    currentChunk += 1;
                    if (currentChunk < chunks) {
                    } else {
                        running = false;
                        console.log("Finished loading!");
                        return spark.end();     // Complete the calculation and return the result
                fileReader.onerror = function () {
                    running = false;
                    console.log("something went wrong");
                function loadNext() {
                    var start = currentChunk * chunkSize,
                        end = start + chunkSize >= file.size ? file.size : start + chunkSize;
                    fileReader.readAsBinaryString(, start, end));
                running = true;

Keywords: Front-end Vue.js elementUI

Added by halex on Wed, 09 Mar 2022 08:34:09 +0200