Many languages have resource embedding schemes. In Golang, there are more open source schemes related to resource embedding. There are many use schemes for Golang resource embedding on the network, but few people analyze the principle, compare the performance of native implementation and open source implementation, and apply scenario analysis.
So this article will talk about this topic, right to throw a brick to attract jade.
Write in front
No matter which language it is, there are always some reasons why we need to embed static resources into the language compilation results. Golang is no exception. However, before the official draft of "resource embedding function" was put forward in December 2019, there were many projects in golang ecology that could provide this demand function. Until the release of Golang 1.16 in 2020, the resource embedding function was officially supported by the official.
Today, more and more articles and even open source projects that have previously implemented the resource embedding function recommend the use of the official Go embedded instruction for function implementation. Perhaps we should have a more objective understanding of the similarities and differences between the "language native function" and the three-party implementation, as well as the objective gap in the performance of technical solutions in the Go language ecology in pursuit of performance.
In the following articles, I will introduce some similar projects that have been famous or widely used in GitHub for a long time, such as packr(3.3k stars), statik (3.4k stars) and go rice (2.3k stars),go-bindata (1.5k stars),vsfgen (1k stars),esc(0.6k stars),fileb0x (0.6k stars)...
In this article, we first take the official native function go embed ded instruction as the starting point and as the standard reference system to talk about the principle, basic use and performance.
Let's talk about the principle first.
Go Embed principle
Read the latest source code of Golang 1.17 and ignore some parts related to command line parameter processing. It is not difficult to find that the main code implementations related to Embed are mainly in the following four files:
- src/embed/embed.go
- src/go/build/read.go
- src/cmd/compile/internal/noder/noder.go
- src/cmd/compile/internal/gc/main.go
embed/embed.go
embed.go mainly provides the relevant declarations and function definitions of the embedded function at runtime (the interface implementation of FS), as well as the description in the go doc document.
The implementation of FS interface is very important for accessing and operating files through the file system. For example, you want to use standard FS functions to "CRUD" files.
// lookup returns the named file, or nil if it is not present.
func (f FS) lookup(name string) *file {
...
}
// readDir returns the list of files corresponding to the directory dir.
func (f FS) readDir(dir string) []file {
...
}
func (f FS) Open(name string) (fs.File, error) {
...}
// ReadDir reads and returns the entire named directory.
func (f FS) ReadDir(name string) ([]fs.DirEntry, error) {
...
}
// ReadFile reads and returns the content of the named file.
func (f FS) ReadFile(name string) ([]byte, error) {
...
}By reading the code, it is not difficult to see that the file is set to read-only in go embed ded, but if you like, you can implement a set of readable and writable file system, which will be mentioned in the later articles.
func (f *file) Mode() fs.FileMode {
if f.IsDir() {
return fs.ModeDir | 0555
}
return 0444
}In addition to directly operating files through FS related functions, we can also mount "embedded Fs" to Go's HTTP Server or any corresponding file processing function of your favorite Go Web framework to realize a static resource server similar to Nginx.
go/build/read.go
If the former provides the availability of go: embedded when we write code, which is relatively "virtual", then build / read Go provides more practical analysis and verification before the program compilation stage.
This program mainly analyzes the content of the go: embedded instruction written in the program, deals with the effectiveness of the content, and carries out specific logical processing for the content (variables, files) to be embedded. There are two key functions:
func readGoInfo(f io.Reader, info *fileInfo) error {
...
}
func parseGoEmbed(args string, pos token.Position) ([]fileEmbed, error) {
...
}The function readGoInfo is responsible for reading our code file * Go, find the content containing go: embedded in the code, and then pass the number of lines of the corresponding file containing this content to the parseGoEmbed function to parse the functions related to the file path in the instruction into a specific file or file list.
If the file resource path is a specific file, save the file to the list of files to be processed. If it is a directory or a syntax like go: embedded image / * template / * then other calling functions will scan the content in the form of glob and save the file to the list of files to be processed.
These contents will eventually be saved in the fileInfo structure related to each program file, and then wait for go / build / build Go and other related compilers.
// fileInfo records information learned about a file included in a build.
type fileInfo struct {
name string // full name including dir
header []byte
fset *token.FileSet
parsed *ast.File
parseErr error
imports []fileImport
embeds []fileEmbed
embedErr error
}
type fileImport struct {
path string
pos token.Pos
doc *ast.CommentGroup
}
type fileEmbed struct {
pattern string
pos token.Position
}compile/internal/noder/noder.go
Compared with the first two programs, noder Go does the heaviest work, is responsible for the final parsing and content association, saves the results in the form of IR, and waits for the processing of the final compiler. In addition, it is also responsible for handling the parsing of cgo related programs (which can also be regarded as some form of embedding).
Here it is also the same as the previous read Like go, it will do some checksum judgment, such as judging whether the resources embedded by the user are really used, or whether the user uses the embedded object and its following functions, but forgets to declare the go: embedded instruction. If these unexpected events are found, it will stop the program in time to avoid entering the compilation stage and wasting time.
Relative core functions are:
func parseGoEmbed(args string) ([]string, error) {
...
}
func varEmbed(makeXPos func(syntax.Pos) src.XPos, name *ir.Name, decl *syntax.VarDecl, pragma *pragmas, haveEmbed bool) {
...
}
func checkEmbed(decl *syntax.VarDecl, haveEmbed, withinFunc bool) error {
...
}In the above function, the go: embedded instruction declared in the file will be associated with the static resources in the actual program directory in the form of IR. It can be simply understood that we have been assigned according to the variables in the context of the go: embedded instruction.
cmd/compile/internal/gc/main.go
After being processed by the above programs, the file will eventually come to the compiler. func Main(archInit func(*ssagen.ArchInfo)) {} calls the following internal functions to write the static resources directly to the disk (attach them to the file):
// Write object data to disk.
base.Timer.Start("be", "dumpobj")
dumpdata()
base.Ctxt.NumberSyms()
dumpobj()
if base.Flag.AsmHdr != "" {
dumpasmhdr()
}During file writing, we can see that the writing process is very simple for embedded static resources (the implementation part is in src/cmd/compile/internal/gc/obj.go):
func dumpembeds() {
for _, v := range typecheck.Target.Embeds {
staticdata.WriteEmbed(v)
}
}So far, we know the principle and process of Golang resource embedding. We also know what capabilities the official resource embedding function has and what capabilities it lacks (compared with other open source implementations). Then, I will expand one by one in the following articles.
Basic use
We need to talk about the basic use of embed first. On the one hand, it is to take care of the students who have not used the embedded function. On the other hand, it is to establish a standard reference system to make an objective evaluation for the subsequent performance comparison.
For the convenience and intuition of testing, in this article and subsequent articles, we give priority to the implementation of a static resource server that can perform performance testing and provide Web services, in which the static resources come from "embedded resources".
Step 1: prepare test resources
When it comes to resource embedding, we naturally need to find appropriate resources. Because it does not involve the processing of specific file types, we only need to focus on the file size here. I found two public files on the network as embedded objects.
- A front-end JavaScript file of about 100KB (94KB): Vue js
- https://cdn.jsdelivr.net/npm/vue@2.6.14/dist/vue.min.js
- One HD picture of about 20MB (17.8MB)
- https://stocksnap.io/photo/technology-motherboard-PUWNNLCU1C
If you want to try it yourself, you can use the link above to get the same test resources. After downloading the file, we can place the resources in the assets folder in the same directory of the program.
Step 2: write basic program
First initialize an empty project:
mkdir basic && cd basic go mod init solution-embed
For fairness, we first use the test code in the official warehouse of Go as the basic template.
// Copyright 2021 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package embed_test
import (
"embed"
"log"
"net/http"
)
//go:embed internal/embedtest/testdata/*.txt
var content embed.FS
func Example() {
mutex := http.NewServeMux()
mutex.Handle("/", http.FileServer(http.FS(content)))
err := http.ListenAndServe(":8080", mutex)
if err != nil {
log.Fatal(err)
}
}After simple adjustment, we can get a program to embed the assets directory in the current directory.
package main
import (
"embed"
"log"
"net/http"
)
//go:embed assets
var assets embed.FS
func main() {
mutex := http.NewServeMux()
mutex.Handle("/", http.FileServer(http.FS(assets)))
err := http.ListenAndServe(":8080", mutex)
if err != nil {
log.Fatal(err)
}
}Then we start the program or compile the program, and we can access the files in our static resource directory in localhost: 8080, for example: http://localhost:8080/assets/example.txt .
This part of the code, you can https://github.com/soulteary/awesome-golang-embed/tree/main/go-embed-official/basic Get from.
Test preparation
Before talking about performance, we first need to modify the program so that the program can be tested and give clear performance indicators.
Step 1: improve testability
Because the above code is simple enough, it is written in the same main function. In order to be tested, we need to make some simple adjustments, such as splitting the registered routing part and the startup service part.
package main
import (
"embed"
"log"
"net/http"
)
//go:embed assets
var assets embed.FS
func registerRoute() *http.ServeMux {
mutex := http.NewServeMux()
mutex.Handle("/", http.FileServer(http.FS(assets)))
return mutex
}
func main() {
mutex := registerRoute()
err := http.ListenAndServe(":8080", mutex)
if err != nil {
log.Fatal(err)
}
}To simplify test code writing, we use an open source assertion library testify, which is installed first:
go get -u github.com/stretchr/testify/assert
Then write the test code:
package main
import (
"net/http"
"net/http/httptest"
"testing"
"github.com/stretchr/testify/assert"
)
func TestStaticRoute(t *testing.T) {
router := registerRoute()
w := httptest.NewRecorder()
req, _ := http.NewRequest("GET", "/assets/example.txt", nil)
router.ServeHTTP(w, req)
assert.Equal(t, http.StatusOK, w.Code)
assert.Equal(t, "@soulteary: Hello World", w.Body.String())
}After the code is written, we execute go test. No accident, we will be able to see the following results:
# go test PASS ok solution-embed 0.219s
In addition to verifying that the function is normal, some additional operations can be added here to conduct a rough performance test, such as the time required to test 100000 times to obtain resources through HTTP:
func TestRepeatRequest(t *testing.T) {
router := registerRoute()
passed := true
for i := 0; i < 100000; i++ {
w := httptest.NewRecorder()
req, _ := http.NewRequest("GET", "/assets/example.txt", nil)
router.ServeHTTP(w, req)
if w.Code != 200 {
passed = false
}
}
assert.Equal(t, true, passed)
}This part of the code, you can https://github.com/soulteary/awesome-golang-embed/tree/main/go-embed-official/testable Obtained from.
Step 2: add performance probe
In the past, for black box programs, we can only use monitoring and comparison before and after to obtain specific performance data. When we have the ability to customize the program, we can directly use the profiler program to collect performance indicators during program operation.
With pprof's ability, we can quickly add several performance related interfaces to the Web service of the above code. Most articles will tell you to reference pprof this module, but it's not. Because reading the code( https://cs.opensource.google/go/go/+/refs/tags/go1.17.6:src/net/http/pprof/pprof.go), we can see that pprof's "automatic registration of performance monitoring interface" is only effective for the default HTTP service, not for the multiplexed (mux) http service:
func init() {
http.HandleFunc("/debug/pprof/", Index)
http.HandleFunc("/debug/pprof/cmdline", Cmdline)
http.HandleFunc("/debug/pprof/profile", Profile)
http.HandleFunc("/debug/pprof/symbol", Symbol)
http.HandleFunc("/debug/pprof/trace", Trace)
}Therefore, in order for pprof to take effect, we need to manually register these performance index interfaces and adjust the code above to get a program similar to the following.
package main
import (
"embed"
"log"
"net/http"
"net/http/pprof"
"runtime"
)
//go:embed assets
var assets embed.FS
func registerRoute() *http.ServeMux {
mutex := http.NewServeMux()
mutex.Handle("/", http.FileServer(http.FS(assets)))
return mutex
}
func enableProf(mutex *http.ServeMux) {
runtime.GOMAXPROCS(2)
runtime.SetMutexProfileFraction(1)
runtime.SetBlockProfileRate(1)
mutex.HandleFunc("/debug/pprof/", pprof.Index)
mutex.HandleFunc("/debug/pprof/cmdline", pprof.Cmdline)
mutex.HandleFunc("/debug/pprof/profile", pprof.Profile)
mutex.HandleFunc("/debug/pprof/symbol", pprof.Symbol)
mutex.HandleFunc("/debug/pprof/trace", pprof.Trace)
}
func main() {
mutex := registerRoute()
enableProf(mutex)
err := http.ListenAndServe(":8080", mutex)
if err != nil {
log.Fatal(err)
}
}After running or compiling the program again, access http://localhost:8080/debug/pprof/
This part of the relevant code can be found in https://github.com/soulteary/awesome-golang-embed/tree/main/go-embed-official/profiler See in.
Performance testing (benchmarking)
Here, I choose to use two methods for performance testing: the first is based on the sampling data of the test case, and the second is based on the throughput of the interface pressure test of the built program.
I have uploaded the relevant code to https://github.com/soulteary/awesome-golang-embed/tree/main/go-embed-official/benchmark , which can be obtained by ourselves for experiments.
Performance sampling based on test cases
We simply adjust the default test program so that it can make a large number of repeated requests (1000 small file reads and 100 large file reads) for the two resources we prepared earlier.
func TestSmallFileRepeatRequest(t *testing.T) {
router := registerRoute()
passed := true
for i := 0; i < 1000; i++ {
w := httptest.NewRecorder()
req, _ := http.NewRequest("GET", "/assets/vue.min.js", nil)
router.ServeHTTP(w, req)
if w.Code != 200 {
passed = false
}
}
assert.Equal(t, true, passed)
}
func TestLargeFileRepeatRequest(t *testing.T) {
router := registerRoute()
passed := true
for i := 0; i < 100; i++ {
w := httptest.NewRecorder()
req, _ := http.NewRequest("GET", "/assets/chip.jpg", nil)
router.ServeHTTP(w, req)
if w.Code != 200 {
passed = false
}
}
assert.Equal(t, true, passed)
}Then, write a script to help us obtain the resource consumption of files with different volumes.
#!/bin/bash go test -run=TestSmallFileRepeatRequest -benchmem -memprofile mem-small.out -cpuprofile cpu-small.out -v go test -run=TestLargeFileRepeatRequest -benchmem -memprofile mem-large.out -cpuprofile cpu-large.out -v
After execution, you can see the following output:
=== RUN TestSmallFileRepeatRequest --- PASS: TestSmallFileRepeatRequest (0.04s) PASS ok solution-embed 0.813s === RUN TestLargeFileRepeatRequest --- PASS: TestLargeFileRepeatRequest (1.14s) PASS ok solution-embed 1.331s === RUN TestStaticRoute --- PASS: TestStaticRoute (0.00s) === RUN TestSmallFileRepeatRequest --- PASS: TestSmallFileRepeatRequest (0.04s) === RUN TestLargeFileRepeatRequest --- PASS: TestLargeFileRepeatRequest (1.12s) PASS ok solution-embed 1.509s
Performance of embedding large files
Use go tool pprof - HTTP =: 8090 CPU large Out can visually display the call of program execution process and resource consumption. After executing the command, open it in the browser http://localhost:8090/ui/ , you can see a call graph similar to the following:
Embedded large file resource usage
In the call graph above, we can see that in the most time-consuming runtime The initiator of the last jump of the memmove (30.22%) function is embedded (* openFile) read (5.04%). It only takes more than 5% of the total time to obtain the nearly 20m resources we want from the embedded resources. The rest of the computation is focused on data exchange, go data length automatic expansion and data recovery.
Read embedded resources and relatively time-consuming calls
Similarly, use go tool pprof - HTTP =: 8090 MEM large Out to view the memory usage
It can be seen that after 100 calls, a total of more than 6300 MB of space has been used in the memory, which is equivalent to 360 times the consumption of our original resources. On average, we need to pay about 3.6 times the resources of the original file for each request.
Resource usage of embedded small files
After reading large files, let's take a look at the resource usage of small files. Because go tool pprof - HTTP =: 8090 CPU small After out, there is no embed related function in the call graph (consumption of resources can be ignored), so we skip the CPU call and look directly at the memory usage.
Before the final output to the user, the resource usage of io copyBuffer will be about 1.7 times that of our resources. It should be due to the gc recycling function. When the data is finally output to the user, the resource usage will be reduced to 1.4 times. Compared with large volume resources, it is not less affordable.
Throughput test using Wrk
Let's execute go build main Go, get the built program, and then execute it/ Mainstart the service, and then test the throughput of small files:
# wrk -t16 -c 100 -d 30s http://localhost:8080/assets/vue.min.js Running 30s test @ http://localhost:8080/assets/vue.min.js 16 threads and 100 connections Thread Stats Avg Stdev Max +/- Stdev Latency 4.29ms 2.64ms 49.65ms 71.59% Req/Sec 1.44k 164.08 1.83k 75.85% 688578 requests in 30.02s, 60.47GB read Requests/sec: 22938.19 Transfer/sec: 2.01GB
Without any code optimization, Go uses small embedded resources to provide services, and can handle about 20000 requests per second. Then let's look at the throughput for large files:
# wrk -t16 -c 100 -d 30s http://localhost:8080/assets/chip.jpg Running 30s test @ http://localhost:8080/assets/chip.jpg 16 threads and 100 connections Thread Stats Avg Stdev Max +/- Stdev Latency 332.75ms 136.54ms 1.32s 80.92% Req/Sec 18.75 9.42 60.00 56.33% 8690 requests in 30.10s, 144.51GB read Requests/sec: 288.71 Transfer/sec: 4.80GB
Because of the larger file size, although the number of requests seems to be reduced, the data throughput per second is more than doubled. Compared with the total data download volume, the problem has more than tripled, from 60GB to 144GB.