background

Recently, when uploading files, the server directly obtains file data through the requested body. It is found that the io performance during uploading is not fully utilized. When uploading large files, 32k is read in a single time. It is hoped that the block size of a single read can be manually adjusted to improve the speed of uploading files.

code implementation

func Post(url, contentType string, body io.Reader) (resp *Response, err error) {
	return DefaultClient.Post(url, contentType, body)
}

Simply look at http It can be seen from the three parameters of post that if you want to change the size of the read block, you can only change it in body io The reader parameter has been used.
I have defined a structure CustomReader and implemented io The reader interface can calculate the file hash while uploading, and make a callback of the upload progress. Without much nonsense, you can directly upload the code.

//#region custom reader
type CustomReader struct {
	curSize int64
	totalSize int64
	reader *os.File
	sha256W hash.Hash
	callback func(curSize, totalSize int64)
}
func NewCustomReader(filePath string, callback func(curSize, totalSize int64)) (*CustomReader, error) {
	reader, err := os.Open(filePath)
	if err != nil {
		return nil, err
	}
	fInfo, err := reader.Stat()
	if err != nil {
		return nil, err
	}
	totalSize := fInfo.Size()
	sha256W := sha256.New()

	return &CustomReader{
		totalSize:totalSize,
		reader:reader,
		sha256W:sha256W,
		callback:callback,
	}, nil
}
func (this *CustomReader) Read(p []byte) (n int, err error) {
	n, err = this.reader.Read(p)
	if n > 0 {
		if n, err := this.sha256W.Write(p[:n]); err != nil {
			return n, err
		}
		this.curSize += int64(n)
		if this.callback != nil {
			this.callback(this.curSize, this.totalSize)
		}
	}
	return
}
func (this *CustomReader) Close() error {
	return this.reader.Close()
}
func (this *CustomReader) WriteTo(w io.Writer) (written int64, err error) {
	buf := make([]byte, 100 * 1024 * 1024) // Specify 100M
	for {
		nr, er := this.reader.Read(buf)
		if nr > 0 {
			nt, et := this.sha256W.Write(buf[0:nr])
			if et != nil {
				err = et
				return
			}
			if nt != nr {
				err = errors.New("invalid write hash")
				return
			}

			nw, ew := w.Write(buf[0:nr])
			if ew != nil {
				err = ew
				return
			}
			if nw != nr {
				err = errors.New("invalid write result")
				return
			}

			written += int64(nw)
			this.curSize = written
		}
		if er != nil {
			if er != io.EOF {
				err = er
			}
			break
		}
		if this.callback != nil {
			this.callback(written, this.totalSize)
		}
	}
	return written, err
}
func (this *CustomReader) Sha256Sum(b []byte) []byte{
	return this.sha256W.Sum(b)
}
//#endregion

// The custom reader uses WriteTo as the io buffer
func UploadByCustomReader(urlStr, filePath string, callback func(curSize, totalSize int64)) (err error) {
	reader, err := NewCustomReader(filePath, callback)
	if err != nil {
		return
	}
	defer reader.Close()

	resp, err := http.Post(urlStr, "multipart/form-data", reader)
	if err != nil {
		return
	}

	if resp.StatusCode != http.StatusOK {
		err = errors.New("no 200 status code")
		return
	}

	fmt.Printf("UploadByCustomReader Upload succeeded, sha256: %x\n", reader.Sha256Sum(nil))
	return
}

Source code analysis

Go version: go1 16.5 windows/amd64
Why http The default single read size of the post body is 32k?

Through the source code, you can see that net / http / Transport. Com is called to send the http request of go RoundTrip method of Transport in go.

Simply give a chain relationship:

`net/http/client.go`
It will be used here `DefaultClient`
http.Post -> Client.Post-> Client.Do -> Client.do -> Client.send -> send ->

`net/http/roundtrip.go`
If not customized `transport`，Can use `DefaultTransport`
Transport.roundTrip ->

`net/http/transport.go`
Transport.roundTrip -> Transport.getConn -> Transport.queueForDial -> Transport.dialConnFor -> Transport.dialConn -> persistConn.writeLoop -> persistConn.writech -> 

`net/http/request.go`
Request.write -> 

`net/http/transfer.go`
transferWriter.writeBody -> transferWriter.doBodyCopy -> 

`io/io.go`
Copy -> copyBuffer

Finally, it is found that uploading and writing data to the body is through io Copy method. Simply look at this method and you can find that it is the 32k block transmission defined here.

func copyBuffer(dst Writer, src Reader, buf []byte) (written int64, err error) {
	// If the reader has a WriteTo method, use it to do the copy.
	// Avoids an allocation and a copy.
	if wt, ok := src.(WriterTo); ok {
		return wt.WriteTo(dst)
	}
	// Similarly, if the writer has a ReadFrom method, use it to do the copy.
	if rt, ok := dst.(ReaderFrom); ok {
		return rt.ReadFrom(src)
	}
	if buf == nil {
		size := 32 * 1024 // Here, the size is defined as 32k
		if l, ok := src.(*LimitedReader); ok && int64(size) > l.N {
			if l.N < 1 {
				size = 1
			} else {
				size = int(l.N)
			}
		}
		buf = make([]byte, size)
	}
	for {
		nr, er := src.Read(buf)
		if nr > 0 {
			nw, ew := dst.Write(buf[0:nr])
			if nw < 0 || nr < nw {
				nw = 0
				if ew == nil {
					ew = errInvalidWrite
				}
			}
			written += int64(nw)
			if ew != nil {
				err = ew
				break
			}
			if nr != nw {
				err = ErrShortWrite
				break
			}
		}
		if er != nil {
			if er != EOF {
				err = er
			}
			break
		}
	}
	return written, err
}

Obviously, if you want to customize the size of block transfer, you can implement the WriterTo method of the WriterTo interface.
One thing to note here is that the custom structure also needs to implement the Close method, because in io Transferwriter is required before copying Unwrapbody, we will be passed through HTTP The reader passed in from post unpacks and finally passes in io Copy is our custom reader.

// net/http/transfer.go  421
func (t *transferWriter) unwrapBody() io.Reader {
	if reflect.TypeOf(t.Body) == nopCloserType {
		return reflect.ValueOf(t.Body).Field(0).Interface().(io.Reader)
	}
	if r, ok := t.Body.(*readTrackingBody); ok {
		// Because we have packed before, we will unpack here and return our customized reader
		r.didRead = true
		return r.ReadCloser
	}
	return t.Body
}

summary

*** Post has a deeper understanding of how to send requests. When there is no way to start, it is most effective to read the source code. With a compliment, Golan's breakpoint debugging is still very easy to use.

When I upload, I also compare many other upload methods. Those who are interested can go to my website gitee warehouse GoTest , upload the relevant code in iotest / uploadandcalchash Go, the server code is in ginserver / API / file go.