background information
Before 2019, Envoy runs in the form of statically compiled binary files, which means that all its extensions need to be compiled in the construction phase. Therefore, other projects (such as Istio) can only publish the custom Envoy version maintained by themselves. Once there is an update or Bug repair, they have to build a new binary version, publish and redeploy to the production environment.
Although there is no perfect solution to the above problems, some scenarios can be realized through the dynamic loadability of C + +, that is, the web assembly (wasm) module is written and delivered under a standard binary application interface (ABI),
WASM itself is a front-end technology. It is a technology born to solve the increasingly complex front-end Web applications and limited JS script interpretation performance. Through this technology, you can write code in non JavaScript programming language and run on the browser.
With the development of WASM, now WASM can not only be used in browsers, but also has been defined as a new format that is portable, small, fast loading and compatible with the Web as a portable binary format.
The WASM discussed in this paper is used to execute code written in multiple languages in a memory safe sandbox at near native speed. There are clear resource constraints and API s in the sandbox to communicate with the embedded host environment (such as Envoy).
advantage
-
Agility: WASM can be dynamically loaded into a running Envoy process without stopping or recompiling
-
Maintainability: Envoy can extend its functions without changing its own basic code base
-
Diversity: popular programming languages (such as C/C + + and TinyGo) can be compiled into WASM, so developers can choose the programming language to implement the filter
-
Reliability and isolation: the filter will be deployed in the VM sandbox, so it is isolated from the Envoy process itself; Even when the WasmFilter crashes due to a problem, it will not affect the Envoy process
-
Security: filters communicate with Envoy agents through predefined API s, so they can access and modify only a limited number of connection or request properties
shortcoming
-
Memory consumption: the use of WASM virtual machine will lock some memory
-
Performance loss: message data is copied and transcoded inside and outside the sandbox
Proxy-wasm
Proxy-Wasm It is the binary application interface (ABI) specification and standard between wasm extension module and L4/L7 agent. It clearly defines the communication interfaces between host environment and wasm virtual machine, function call, memory management and so on.
Currently proxy wasm provides AssemblyScript SDK,C++ SDK,Go (TinyGo) SDK,Rust SDK,Zig SDK SDK, support Envoy,Istio Proxy (Istio Evoy based extension) MOSN And other proxy host environments.
Overall architecture
On each Envoy worker thread (event driven), the built-in wasm runtime will create a wasm virtual machine to verify and instantiate the wasm module through the proxy wasm specification (local disk file or control panel XDS push).
When the wasm module is called through the extension interface, proxy wasm transcodes and translates through a gasket and runs on the wasm virtual machine.
Note: Envoy uses a single process multithreading architecture model. A master thread manages various trivial tasks, while some worker threads are responsible for listening, filtering and forwarding. When a listener receives a connection request, the connection binds its life cycle to a separate worker thread.
Runtime
Envoy is embedded with LLVM based WAVM and V8 C/C++ Wasm runtime, which can be selected during WASM module configuration.
Proxy-wasm-go-sdk
Go (TinyGo) SDK Is a proxy wasm implementation based on Tinygo language.
This paper is based on the project label v0 14.0.
TinyGo
TinyGo is a Go compiler designed for small scenarios such as microcontrollers, web assemblies (WASM) and command line tools. It reuses the library used by Go language tools and LLVM to provide another way to compile programs written in Go programming language.
The official Go compiler cannot generate binary files compatible with proxy wasm, and another major difference between TinyGo and Go is the binary size.
According to TinyGo official Description: for the simplest "Hello world" program, under the blessing of the strip command (removing all symbol flags and debugging information), the Go compiler generates 837kb binary, while TinyGo is 10kb, which is close to 1% size reduction efficiency.
Using TinyGo also has some limitations and constraints:
-
Some Go libraries are not available (can be imported, but run-time exceptions)
-
Some system calls are not available, such as crypto/rand packages
-
Reflection is not supported
-
Some language features are not supported, such as recover and goroutine
Although it does not support the creation of coroutines, proxy wasm defines the OnTick function, which is similar to the timer trigger function, and can be used to handle some asynchronous call tasks.
term
-
Virtual machine (Wasm VM): Wasm virtual machines in envoy are created in each worker thread and isolated from each other
-
Plug in: filter types in envoy (Http Filter, Network(Tcp) Filter, and Wasm Service). Each extension module can be configured. The same Wasm module in a single virtual machine forms multiple plug-ins after different configurations
-
Http Filter: handles the HTTP protocol in the worker thread virtual machine, and can operate the header, body and tail contents of HTTP requests
-
Network Filter: process Tcp protocol in the worker thread virtual machine, and operate Tcp data frames and connection information
-
Wasm Service: it runs in the single instance virtual of Envoy main thread and can be used to concurrently process some additional tasks, such as integrating metrics, logs, etc
-
Envoy configuration
Wasm filter in Envoy is configured as follows:
vm_config:
vm_id: "foo"
runtime: "envoy.wasm.runtime.v8"
configuration:
"@type": type.googleapis.com/google.protobuf.StringValue
value: '{"my-vm-env": "dev"}'
code:
local:
filename: "example.wasm"
configuration:
"@type": type.googleapis.com/google.protobuf.StringValue
value: '{"my-plugin-config": "bar"}'| field | describe |
|---|---|
| vm_config | Configure Wasm virtual machine |
| vm_config.vm_id | id of the virtual machine, which can be used for configuration Cross virtual machine communication |
| vm_config.runtime | Wasm runtime type, such as envoy wasm. runtime. v8. |
| vm_config.configuration | Virtual machine configuration, which can be read dynamically at run time to configure different virtual machine contexts |
| vm_config.code | Wasm binary location |
| configuration | Plug in configuration, which can be read dynamically at run time to configure different plug-in contexts |
Field VM_ When all attributes in config are the same value, multiple plug-ins share a Wasm virtual machine, which will have a certain impact on resource usage and startup delay.
Http Filter
Handle HTTP events, that is, HTTP protocol traffic. The plug-in reference is envoy filter. http. Wasm, for example:
http_filters:
- name: envoy.filters.http.wasm
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
config:
vm_config: { ... }
# ... plugin config follows
- name: envoy.filters.http.routerNetwork Filter
Handle TCP events, that is, all TCP traffic (including http traffic), and the plug-in reference is envoy filter. network. Wasm, for example:
filter_chains:
- filters:
- name: envoy.filters.network.wasm
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.wasm.v3.Wasm
config:
vm_config: { ... }
# ... plugin config follows
- name: envoy.tcp_proxyNote: the difference between Http Filter and Network Filter is only through different configurations, which act on TCP stream or HTTP stream respectively.
Wasm Service
It works on the main thread and is configured in bootstrap_ In extensions, the plug-in reference is envoy bootstrap. Wasm, for example:
bootstrap_extensions:
- name: envoy.bootstrap.wasm
typed_config:
"@type": type.googleapis.com/envoy.extensions.wasm.v3.WasmService
singleton: true
config:
vm_config: { ... }
# ... plugin config followsThe singleton attribute is usually configured as true to represent the virtual machine that reuses the main thread. At this time, the main thread will not block the operation of the plug-in in the working thread.
Go SDK API
Context (Contexts)
The set of interfaces in the Go SDK has four types of contexts: VMContext, PluginContext, TcpContext and HttpContext. The relationship table is as follows:
Wasm Virtual Machine
(.vm_config.code)
┌────────────────────────────────────────────────────────────────┐
│ Your program (.vm_config.code) TcpContext │
│ │ ╱ (Tcp stream) │
│ │ 1: 1 ╱ │
│ │ 1: N ╱ 1: N │
│ VMContext ────────── PluginContext │
│ (Plugin) ╲ 1: N │
│ ╲ │
│ ╲ HttpContext │
│ (Http stream) │
└────────────────────────────────────────────────────────────────┘-
VMContext: corresponds to in the configuration vm_config.code, there is only one VMContext in each virtual machine. As the parent of PluginContexts, you can create any number of PluginContexts
-
PluginContext: used to configure configuration is responsible for instantiating specific plug-ins. As the parent of TcpContex and HttpContext, you can create multiple TcpContex and HttpContext
-
TcpContext: Processing Tcp data stream
-
HttpContext: process Http data stream
The source code of VMContext is defined as follows:
// VMContext is equivalent to the configuration of Wasm virtual machine and is the entry point for extending network agent. Its life cycle is the same as that of Wasm virtual machine
type VMContext interface {
// When Wasm virtual machine is created, OnVMStart is called, during which API GetVMConfiguration can be used to retrieve VMS in the configuration_ config. Configuration property
// This function is mainly used for Wasm virtual machine level initialization
OnVMStart(vmConfigurationSize int) OnVMStartStatus
// Create PluginContext according to the plug-in configuration
NewPluginContext(contextID uint32) PluginContext
}The plug-in context PluginContext source code is defined as follows:
// PluginContext is equivalent to each different plug-in configuration (config.configuration)
// Each configuration is usually created in the http/tcp filter of a listener, so PluginContext is equivalent to creating a network filter instance
type PluginContext interface {
// After the OnVmStart call occurs, OnPluginStart will be called, during which API GetPluginConfiguration can be used to retrieve config. In the configuration Configuration property
OnPluginStart(pluginConfigurationSize int) OnPluginStartStatus
// onPluginDone is called when the plug-in ends running in the host
// Returning false means that it is in pending status, and there are still some legacy work to be completed
// In this case, the method PluginDone() must be called to tell the host that the work is complete and the context can be cleared
OnPluginDone() bool
// When the plug-in calls API RegisterQueue, other plug-ins will put data into the queue, and OnQueueReady of this plug-in is called
OnQueueReady(queueID uint32)
// When the timing period is set through API SetTickPeriodMilliSeconds and the time has expired, OnTick of this plug-in is called
// This method can be used to process other tasks in parallel during stream processing
OnTick()
// The developer must implement one of the following two extension entry points for real stream data
//
// NewTcpContext is used to create TcpContext. Returning nil means that this plug-in is not applicable to TcpContext
NewTcpContext(contextID uint32) TcpContext
// NewHttpContext is used to create HttpContext. Returning nil means that this plug-in is not applicable to HttpContext
NewHttpContext(contextID uint32) HttpContext
}HttpContext and TcpContext will not be expanded specifically, but can be expanded through context.go View details.
Host call API
The host call API is a series of methods provided by proxy wasm to interact with network plug-ins. For example, GetHttpRequestHeaders API can be called in HttpContext to obtain Http request header data, and the LogInfo API can be used to add print information to the log.
All available API s are available through hostcall.go View details.
entry point
When Envoy creates Wasm virtual machine, it will call the main function in the program before he creates VMContext. Therefore, the user-defined VMContext must be implemented in the main function.
Proxywasm The SetVMContext in the package is the entry point for creating VMContext Proxywasm The DefaultVMContext and main functions provided by the package are generally as follows:
func main() {
proxywasm.SetVMContext(&vmContext{})
}
type vmContext struct {
// Embed the virtual machine context provided by default, so you don't have to implement all the methods in the VMContext interface
types.DefaultVMContext
}
// Override NewPluginContext method in DefaultVMContext
func (*vmContext) NewPluginContext(contextID uint32) types.PluginContext {
return &pluginContext{}
}
type pluginContext struct {
// Embed the plug-in context provided by default
types.DefaultPluginContext
}
// Override the NewTcpContext method in DefaultPluginContext
func (ctx *pluginContext) NewTcpContext(contextID uint32) types.TcpContext {
return &networkContext{}
}
type networkContext struct {
// Embed default provided Tcp context
types.DefaultTcpContext
}
// Override the method OnNewConnection of DefaultTcpContext
func (ctx *networkContext) OnNewConnection() types.Action {
... ... Cross virtual machine communication
As mentioned above, the built-in wasm runtime on each Envoy worker thread will create a wasm virtual machine. In some specific scenarios, we may need to communicate with other virtual machines in the current virtual machine, such as integration status information, cache data, etc.
At present, two schemes are provided to realize cross virtual machine communication.
shared data
Shared data is a scheme based on key value pair storage to share data across virtual machines or threads.
Shared data is applicable to scenarios such as:
-
A global request counter is used to count requests in multiple Wasm virtual machines
-
Multiple Wasm virtual machines need to share cached data
A shared storage area is created through vm_config.vm_id configuration, which means that the same wasm binary (specified by vm_config.code) is not a necessary condition.
As shown in the figure above, although the two virtual machines use hello Wasm and bye Wasm two binaries, because they use the same vm_id (foo), but they share the same data store.
The API s available for sharing data are as follows:
// GetSharedData retrieves the specified key // The returned "cas" is the value set in the method SetSharedData to ensure thread safe update func GetSharedData(key string) (value []byte, cas uint32, err error) // SetSharedData is used to set key value pairs in shared storage // If the CAS value does not match the current value, errorstatus casmismatch is returned, which means that other Wasm virtual machines have set a value on this key, so the current CAS value is updated incrementally. Therefore, it is very necessary to add retry logic to the change logic // When CAS is set to 0, CAS value comparison will not be performed, and success will always be returned func SetSharedData(key string, data []byte, cas uint32) error
The API is relatively simple and uses Compare-And-Swap Scheme to ensure thread safety.
Shared queue
A shared queue is a first in first out (FIFO) queue.
Shared queues are applicable to scenarios such as:
-
Parallel integration of metrics information in multiple Wasm virtual machines
-
Push cross virtual machine and complete set of information to remote host
A shared queue passes through the VM in the configuration_ config. vm_ ID and a queue name (vm_id, name). Through these two generation, a queue ID(queue_id) can be generated for the out / in queue.
The API s available for shared queues are as follows:
// ResolveSharedQueue via vm_id and queue name to generate ququeue ID, which is used for Enqueue/DequeueSharedQueue methods func ResolveSharedQueue(vmID, queueName string) (ququeID uint32, err error) // Queue through queueID func EnqueueSharedQueue(queueID uint32, data []byte) error // Out of queue through queueID func DequeueSharedQueue(queueID uint32) ([]byte, error) // RegisterSharedQueue is used to register a shared queue in the plug-in context // Registration means that the OnQueueReady method of the current plug-in context is called when data is queued func RegisterSharedQueue(name string) (ququeID uint32, err error)
Generally, RegisterSharedQueue and DequeueSharedQueue are called by "consumer", ResolveSharedQueue and EnqueueSharedQueue are used by "producer":
-
RegisterSharedQueue is the "consumer" through vm_id and name to create a shared queue, which is usually called in PluginContext
-
ResolveSharedQueue is used by the producer to put data into the queue after resolving the queue ID
Therefore, both methods return the queue ID.
stay Environment context As mentioned in the section, the PluginContext contains an API OnQueueReady, which is the mechanism used to notify the "consumer" when data is queued. When other plug-ins queue data, OnQueueReady of this plug-in is called.
It is recommended to create a shared queue in the Wasm Service of the singleton (such as the main thread of Envoy). Otherwise, when OnQueueReady is called, the Tcp/Http stream processing of the current worker thread will be blocked.
As shown in the above figure, the main thread wasm virtual machine (vm_id="foo", my-singleton.wasm) creates and registers two shared queues (named "Http" and "Tcp" respectively) through RegisterQueue. The "producers" of the two shared queues instantiate HttpContext and TcpContext in the wasm virtual machine of their respective worker threads to process Http and Tcp data streams. When they queue data to their respective queues, the PluginContext in the main thread automatically calls the OnQueueReady method to obtain queue data.
Sample test
Due to the rich functions supported by Wasm filter, this section only carries out a simple data stream printing test to verify its role in Tcp data stream.
Deploy sample services
Deploy the following services and inject them into the side car to verify the normal operation of the service:
-
A simple sleep service used to simulate sending http requests
-
The goserver server is used to respond to http requests. The 8081 port of the person in the container is mapped to the 9091 port of the Service
-
For producers and consumers who call regularly under dubbo protocol, ServiceEntry is used instead of Service resource
[root@linux ~]# kubectl -nwasm get po
NAME READY STATUS RESTARTS AGE
goserver-7c5cc7cf6-lslcz 2/2 Running 2 4d2h
sleep-558cdddbdb-g4wwd 2/2 Running 2 3d6h
[root@linux ~]# kubectl -nwasm get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
goserver NodePort 10.107.169.228 <none> 9091:16814/TCP
sleep ClusterIP 10.96.185.32 <none> 80/TCP
[root@linux ~]# kubectl -nwasm exec -it goserver-7c5cc7cf6-lslcz -c goserver -- bash
bash-4.4# curl 127.0.0.1:8081/healthz
{"status":"healthy","hostName":"goserver-7c5cc7cf6-lslcz"}
[root@linux ~]# kubectl -ndubbo get po -owide
NAME READY STATUS IP
dubbo-sample-consumer-6958b44b75-lf7sr 2/2 Running 10.244.104.60
dubbo-sample-provider-v1-cfdcf7768-ptbld 2/2 Running 10.244.122.140
[root@linux ~]# kubectl -ndubbo get se dubbo-samples-demoservice -o yaml
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
...
spec:
addresses:
- 240.240.0.5
endpoints:
- address: 10.244.122.140
ports:
tcp-dubbo: 20880
[root@linux ~]# kubectl -ndubbo logs -f deploy/dubbo-sample-consumer
...
Hello Aeraki, response from dubbo-sample-provider-v1-cfdcf7768-ptbld/10.244.122.140Build Wasm filter
according to web-assembly-hub The tutorial installs the wasm client tool and initializes the project
[root@linux ~]# wasm init --language tinygo --platform istio --platform-version 1.9.x tcp-stream-data INFO[0000] extracting 1416 bytes to /path/tcp-stream-data [root@linux ~]# tree . . ├── go.mod ├── go.sum ├── main.go └── runtime-config.json 0 directories, 4 files
Modify go Mod, using the latest SDK:
go 1.16 require github.com/tetratelabs/proxy-wasm-go-sdk v0.14.0
Modify its default generated main Go to make it conform to the usage of the new SDK (the default application is version v0.1.0). Add a custom print function:
...
// Print tcp connection properties
func (ctx *networkContext) PrintConnectionAttrs() error {
addr, err := proxywasm.GetProperty([]string{"source", "address"})
......
proxywasm.LogInfof("source address: %s", string(addr))
dest, err := proxywasm.GetProperty([]string{"destination", "address"})
......
proxywasm.LogInfof("destination address: %s", string(dest))
return nil
}
// Print envoy upstream properties
func (ctx *networkContext) PrintUpstreamAttrs() error {
addr, err := proxywasm.GetProperty([]string{"upstream", "address"})
......
proxywasm.LogInfof("upstream address: %s", string(addr))
return nil
}Reference the above custom functions in ondownstreamdata (client request data) and onupstreamdata (server response data) respectively for printing:
func (ctx *networkContext) OnDownstreamData(dataSize int, endOfStream bool)types.Action{
......
_ = ctx.PrintConnectionAttrs()
_ = ctx.PrintUpstreamAttrs()
data, err := proxywasm.GetDownstreamData(0, dataSize)
......
proxywasm.LogInfof(">>>>>> downstream data received >>>>>>\n%s", string(data))
return types.ActionContinue
}
func (ctx *networkContext) OnUpstreamData(dataSize int, endOfStream bool) types.Action {
......
_ = ctx.PrintConnectionAttrs()
_ = ctx.PrintUpstreamAttrs()
data, err := proxywasm.GetUpstreamData(0, dataSize)
......
proxywasm.LogInfof("<<<<<< upstream data received <<<<<<\n%s", string(data))
return types.ActionContinue
}Among them, Envoy supports the referenced environment context attribute. See its Official website.
Compile the code in linux environment to generate binary files:
# Configure the network agent as needed [root@linux ~]# export http_proxy=proxyIP:proxyPort [root@linux ~]# export GOPROXY=https://goproxy.cn,https://goproxy.io,direct # Compile (complete the compilation in the container image quay.io/solo-io/ee-builder:0.0.33, and map the compilation results to the local disk) [root@linux ~]# wasm build tinygo . -t tcp-steam-data:test --store ./build/ Building with tinygo...go: downloading github.com/tetratelabs/proxy-wasm-go-sdk v0.14.0 INFO[0007] adding image to cache... filter file=/tmp/wasme551366072/filter.wasm tag="tcp-steam-data:test" INFO[0007] tagged image digest="sha256:fc1563eb463aeb31119104a923509d4e885063ad0bd64fcfd2f6dd4da79c2196" image="docker.io/library/tcp-steam-data:test" [root@linux ~]# ls -l build/79ada3a6417713a07a6c89d400f62306/ -rw-r--r--. 1 root root 225 Aug 9 16:57 descriptor.json -rw-r--r--. 1 root root 255K Aug 9 16:57 filter.wasm -rw-r--r--. 1 root root 37 Aug 9 16:57 image_ref -rw-r--r--. 1 root root 126 Aug 9 16:57 runtime-config.json
Apply Wasm filter
For simplicity, the storage is mounted in the hostPath mode so that the sleep service sidecar container can access the locally generated Wasm binary file:
apiVersion: apps/v1
kind: Deployment
metadata:
name: sleep
......
template:
metadata:
annotations:
sidecar.istio.io/userVolume: '[{"name":"host","host": {"path":"/host/path"}}]'
sidecar.istio.io/userVolumeMount: '[{"mountPath":"/mount/path","name":"host"}]'Modify the log level of the client side car container to Info, and the default is Warn, while proxywasm is used in the above code Loginfof enter Info log:
[root@linux ~]# kubectl -nwasm exec deploy/sleep -- curl -X POST http://localhost:15000/logging?level=info active loggers: admin: info ...... wasm: info [root@linux ~]# kubectl -ndubbo exec deploy/dubbo-sample-consumer -- curl -X POST http://localhost:15000/logging?level=info active loggers: admin: info ...... wasm: info
Because this paper applies Wasm module to SIDECAR_OUTBOUND environment, so it is necessary to adjust the client side vehicle log level.
Write the EnvoyFilter resource so that the Wasm filter is inserted into the last filter envoy filters. network. tcp_ Before proxy:
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: goserver-wasm
namespace: wasm
spec:
configPatches:
- applyTo: NETWORK_FILTER
match:
context: SIDECAR_OUTBOUND
listener:
name: 10.107.169.228_9091 # (goserver) svcIP_svcPort
filterChain:
filter:
name: envoy.filters.network.tcp_proxy # tcp traffic
patch:
operation: INSERT_BEFORE
value:
name: envoy.filters.network.wasm
typed_config:
'@type': type.googleapis.com/envoy.extensions.filters.network.wasm.v3.Wasm
config:
name: tcp-stream-data
configuration:
'@type': type.googleapis.com/google.protobuf.StringValue
value: "empty string"
vm_config:
vm_id: "same_vm_id"
runtime: "envoy.wasm.runtime.v8"
code:
local:
filename: "/mount/path/filter.wasm"EnvoyFilter configuration of Dubbo protocol except listener Name needs to be replaced with serviceentry Spec.addresses is almost the same as others. It is not expanded here.
http traffic
Enter the sleep service and send an http request:
[root@linux ~]# kubectl -nwasm exec -it deploy/sleep -c sleep -- sh
/ # curl goserver:9091/healthz
{"status":"healthy","hostName":"goserver-7c5cc7cf6-lslcz"}Note that you need to use svcName:svcPort at this time, because the Host domain name will be matched according to the requested Host message header according to the invoke Rule.
At this time, observe the sidecar container log of sleep service:
According to the above results, the addresses and port data printed in ondownstreamdata (client request data) and onupstreamdata (server response data) are consistent. Since Http traffic also belongs to Tcp traffic, the message header and body in Http request are completely printed.
dubbo(tcp) traffic
Since the deployed client requests the server data regularly, it is not necessary to trigger the request sending manually.
In the dubbo protocol traffic, the request and response data are also printed normally. Because the protocol header customized by dubbo is in non plain text character format, some garbled codes appear in the data.
However, the interface, method, message type and other data in the request can still be seen in the body.
Summary
The above application of Wasm filter in Istio service grid, from its architecture principle, standard use specification to sample test, completely shows its pluggable and scalable characteristics in Envoy agent.
However, problems in its application in Istio will still be found in practice:
-
Wasm filter cannot operate normally in INBOUND traffic, especially when its type is defined as Network Filter, the side car container will report an error
-
After the Envoyfilter takes effect, if the service instance pod restarts, the service may become unavailable
It is believed that the official will support the Wasm extension of Envoy written in more and more languages. We can easily choose our own familiar language to implement such functions as measurement, observability, transformation, data loss prevention, compliance verification or other functions.