1, Pain points of uploading large files at the front end
1. Too many files lead to tight bandwidth resources and reduced request speed;
2. If the service is interrupted, the network is interrupted and the page crashes during the upload process, the file may be uploaded again.
2, Analysis of pain points
The front end selects the file and uploads it. In the process of processing the file, the back end will first load the file into the running memory, and then call the corresponding API to write it into the hard disk memory to complete the upload of the whole file.
However, uploading files directly in this way may lead to an avalanche of the whole process due to a problem in a certain link, so it is not advisable to upload large files directly.
For example, the best way to solve the problem is to transfer the large file to 100M.
3, Principle of breakpoint continuation
As the name suggests, it is breakpoint and sequel
What is a breakpoint?
At this time, the upload process will be interrupted by N threads, and then the upload process will be interrupted by one or more concurrent breakpoints.
Every time the front-end uploads a piece, it will be loaded into the running memory, and then written to the hard disk. At this time, the temporary variable of the running memory will be released, and then this temporary variable will be occupied by the next piece, and then written and released
What is a sequel?
It means to continue to upload the remaining files from the interrupted location, rather than from the beginning.
After uploading, merge at the server (the merging operation is carried out at the back end, and the front end only calls the interface. The merging method is determined by the back end. Whether to merge one piece after uploading, or to merge all the pieces after uploading).
Implementation of breakpoint continuation
1) Implementation of slicing
Method:
The way before html5z was flash and activeX
HTML 5 provides slice method to split binary stream of files.
const chunks = Math.ceil(file.size / eachSize)
The fragmentation of documents is generally between 2-5M. This step obtains the content of each piece of file, the serial number of each block, the size of each block, the total number of blocks and other data.
2) Implementation of continuous transmission
-
To continue transmission, we must first determine which file needs to be uploaded. The way to determine a file is to encrypt the file. As long as the content of a file changes, it needs to be uploaded again. MD5 encryption can be applied to the file as the unique identifier of the file. MD5 encryption is irreversible.
Note: encrypting the entire large file may cause the page to crash, and the file needs to be encrypted in pieces.
spark-md5 plug-in supports file fragment encryption.
Use of spark-md5 based on elementUI -
Before uploading the first piece of file, you need to use the file name + the unique identifier of this file + the current number of pieces to query whether the file has been uploaded. Through the uploaded results returned by the server, we can obtain the rest through the segmentation results for uploading. If the page is reloaded or the upload is interrupted, you can continue the upload only by interrupting in which piece before re uploading.
-
After uploading, request the merge interface (the merge interface can also not be requested, and the back-end will merge all the files by itself after getting all the files), merge the files at the server, and then the whole file upload is completed.
-
In the process of uploading, the front-end progress bar can be displayed according to the number of successfully uploaded films and the total number of films returned by the server.
Use of Upload spark-md5 in element UI
//js part
import chunkedUpload from './chunkedUpload'
export default {
data() {
return {
uploadData: {
//There are additional parameters in it
},
//File upload path
uploadUrl: process.env.BASE_API + '/oss/oss/uploadChunkFile', //File upload path
chunkedUpload: chunkedUpload // The customized method of fragment upload is introduced in the header
}
},
methods: {
onError(err, file, fileList) {
this.$store.getters.chunkedUploadXhr.forEach(item => {
item.abort()
})
this.$alert('File upload failed, please try again', 'error', {
confirmButtonText: 'determine'
})
},
beforeRemove(file) {
// If you are uploading in pieces, cancel the uploading in pieces
if (file.percentage !== 100) {
this.$store.getters.chunkedUploadXhr.forEach(item => {
item.abort()
})
}
}
}
}
//chunkedUpload.js
import SparkMD5 from 'spark-md5'
import axios from 'axios'
import store from '@/store'
// If the upload is wrong, get the error information
function getError(action, option, xhr) {
let msg
if (xhr.response) {
msg = `${xhr.response.error || xhr.response}`
} else if (xhr.responseText) {
msg = `${xhr.responseText}`
} else {
msg = `fail to post ${action} ${xhr.status}`
}
const err = new Error(msg)
err.status = xhr.status
err.method = 'post'
err.url = action
return err
}
// After the upload completes the merge successfully, get the information returned by the server
function getBody(xhr) {
const text = xhr.responseText || xhr.response
if (!text) {
return text
}
try {
return JSON.parse(text)
} catch (e) {
return text
}
}
// For the custom request of fragment upload, the following requests will override the default upload behavior of element
export default function upload(option) {
if (typeof XMLHttpRequest === 'undefined') {
return
}
const spark = new SparkMD5.ArrayBuffer()// ArrayBuffer encryption class of md5
const fileReader = new FileReader()// File reading class
const action = option.action // File upload path
const chunkSize = 1024 * 1024 * 30 // Single slice size
let md5 = ''// Unique identification of the file
const optionFile = option.file // Files requiring fragmentation
let fileChunkedList = [] // Array after file fragmentation
const percentage = [] // An array of file upload progress. A single item is a piece of progress
// Start slicing the file, push it into the fileChunkedList array, and use the first slicing to calculate the md5 of the file
for (let i = 0; i < optionFile.size; i = i + chunkSize) {
const tmp = optionFile.slice(i, Math.min((i + chunkSize), optionFile.size))
if (i === 0) {
fileReader.readAsArrayBuffer(tmp)
}
fileChunkedList.push(tmp)
}
// After reading the file, start to calculate the file md5 as the unique ID of the file
fileReader.onload = async(e) => {
spark.append(e.target.result)
md5 = spark.end() + new Date().getTime()
console.log('file md5 by--------', md5)
// Convert fileChunkedList into FormData object and add the data required for uploading
fileChunkedList = fileChunkedList.map((item, index) => {
const formData = new FormData()
if (option.data) {
// Add additional incoming data from outside
Object.keys(option.data).forEach(key => {
formData.append(key, option.data[key])
})
// These fields can be transferred according to what the backend needs. You can also add additional parameters yourself
formData.append(option.filename, item, option.file.name)// file
formData.append('chunkNumber', index + 1)// Current file block
formData.append('chunkSize', chunkSize)// Single block size
formData.append('currentChunkSize', item.size)// Current block size
formData.append('totalSize', optionFile.size)// Total file size
formData.append('identifier', md5)// Document identification
formData.append('filename', option.file.name)// file name
formData.append('totalChunks', fileChunkedList.length)// Total number of blocks
}
return { formData: formData, index: index }
})
// How to update the percentage of upload progress bar
const updataPercentage = (e) => {
let loaded = 0// Total size of uploaded files
percentage.forEach(item => {
loaded += item
})
e.percent = loaded / optionFile.size * 100
option.onProgress(e)
}
// Create queue upload tasks. limit is the number of concurrent uploads
function sendRequest(chunks, limit = 3) {
return new Promise((resolve, reject) => {
const len = chunks.length
let counter = 0
let isStop = false
const start = async() => {
if (isStop) {
return
}
const item = chunks.shift()
console.log()
if (item) {
const xhr = new XMLHttpRequest()
const index = item.index
// Fragment upload failure callback
xhr.onerror = function error(e) {
isStop = true
reject(e)
}
// Fragment upload successful callback
xhr.onload = function onload() {
if (xhr.status < 200 || xhr.status >= 300) {
isStop = true
reject(getError(action, option, xhr))
}
if (counter === len - 1) {
// The last upload is complete
resolve()
} else {
counter++
start()
}
}
// Callback during fragment upload
if (xhr.upload) {
xhr.upload.onprogress = function progress(e) {
if (e.total > 0) {
e.percent = e.loaded / e.total * 100
}
percentage[index] = e.loaded
console.log(index)
updataPercentage(e)
}
}
xhr.open('post', action, true)
if (option.withCredentials && 'withCredentials' in xhr) {
xhr.withCredentials = true
}
const headers = option.headers || {}
for (const item in headers) {
if (headers.hasOwnProperty(item) && headers[item] !== null) {
xhr.setRequestHeader(item, headers[item])
}
}
// File start upload
xhr.send(item.formData)
//Here is to save all the xhr uploaded in pieces to the global. If the user cancels the upload manually or there is an error in the upload, you need to call xhr Abort() stops all xhr in the store, otherwise the file will continue to upload
store.commit('SET_CHUNKEDUPLOADXHR', xhr)
}
}
while (limit > 0) {
setTimeout(() => {
start()
}, Math.random() * 1000)
limit -= 1
}
})
}
try {
// Call the upload queue method and wait for all files to be uploaded
await sendRequest(fileChunkedList, 3)
// The parameters here are written according to your actual situation
const data = {
identifier: md5,
filename: option.file.name,
totalSize: optionFile.size
}
// Send file merge request to backend
const fileInfo = await axios({
method: 'post',
url: '/api/oss/oss/mergeChunkFile',
data: data
})
// This 8200 is the code that we successfully stored in oss. It can be changed according to our actual situation
if (fileInfo.data.code === 8200) {
const success = getBody(fileInfo.request)
option.onSuccess(success)
return
}
} catch (error) {
option.onError(error)
}
}
}
Front end through spark-md5 JS calculates the local file MD5
Two methods are provided here; One is to use sparkmd5 Hashbinary () directly transfers the binary code of the whole file into md5 and directly returns the file. This method has advantages for small files - simple and fast.
Another method is to use the slice() method of the File object in js (File.prototype.slice()) to slice the File and then transfer it to spark. JSP one by one Appendbinary() method, and finally through spark The end () method outputs the results. Obviously, this method will be very beneficial for large files - it is not error prone and can provide the progress information of the calculation
The first way:
var running = false; //running is used to determine whether md5 is being calculated
function doNormalTest( input ) { //It is assumed that the dom reference of the file selection box is passed in directly
if (running) { // If the calculation is in progress, the next calculation is not allowed
return;
}
var fileReader = new FileReader(), //Create FileReader instance
time;
fileReader.onload = function (e) { //The load event of FileReader is triggered when the file is read
running = false;
// e.target points to the fileReader instance above
if (file.size != e.target.result.length) { //If the two are inconsistent, it indicates that there is an error in reading
alert("ERROR:Browser reported success but could not read the file until the end.");
} else {
console.log(Finished loading!success!!);
return SparkMD5.hashBinary(e.target.result); //Calculate md5 and return the result
}
};
fileReader.onerror = function () { //If there is an error reading the file, cancel the reading status and pop-up an error
running = false;
alert("ERROR:FileReader onerror was triggered, maybe the browser aborted due to high memory usage.");
};
running = true;
fileReader.readAsBinaryString( input.files[0] ); //Read file binary code through fileReader
};The second way
function doIncrementalTest( input ) { //It is assumed that the dom reference of the file selection box is passed in directly
if (running) {
return;
}
//The slice() method of File needs to be used here. The following is the compatible writing method
var blobSlice = File.prototype.slice || File.prototype.mozSlice || File.prototype.webkitSlice,
file = input.files[0],
chunkSize = 2097152, // Read one by one with the size of 2MB per slice
chunks = Math.ceil(file.size / chunkSize),
currentChunk = 0,
spark = new SparkMD5(), //Create an instance of SparkMD5
time,
fileReader = new FileReader();
fileReader.onload = function (e) {
console("Read chunk number (currentChunk + 1) of chunks ");
spark.appendBinary(e.target.result); // append array buffer
currentChunk += 1;
if (currentChunk < chunks) {
loadNext();
} else {
running = false;
console.log("Finished loading!");
return spark.end(); // Complete the calculation and return the result
}
};
fileReader.onerror = function () {
running = false;
console.log("something went wrong");
};
function loadNext() {
var start = currentChunk * chunkSize,
end = start + chunkSize >= file.size ? file.size : start + chunkSize;
fileReader.readAsBinaryString(blobSlice.call(file, start, end));
}
running = true;
loadNext();
}