Envoy configuration use
Envoy agents have two common uses. One is used as a sidecar and the other is used as a gateway.
When used as a sidecar, Envoy is a four or seven layer application agent next to the service, which can generate indicators, application policies and control traffic.
When used as an API gateway, Envoy acts as a "front-end agent" to receive inbound traffic, check the information in the request and direct it to the destination. The example in this article will demonstrate how to use Envoy as the front agent. We will write a static configuration to return static data that will not change, such as HTTP and IPv4. As you will see in this example, this purpose is simple and suitable for dealing with almost unchanged information.
envoy basic request process
- listener: bind with the port and listen for inbound requests to the gateway. The listening addresses of listeners are mutually exclusive. Two listeners cannot listen to the same socket address
- filter chain: how the request should be handled after being accepted by the listener and entered into Envoy. It consists of some filters, which decide whether the request can be passed to the next filter or short circuit occurs. filter chains are network filter s that can have multiple groups.
- Route: after the request passes through all filter s, the route will direct the request to the correct service
Write yaml file
#Declare static resources
static_resources:
listeners:
- address:
socket_address:
address: 0.0.0.0
port_value: 8080
filter_chains: #Filtering chain and routing
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: backend
domains:
- "*"
routes:
- match:
prefix: "/service/1"
route:
cluster: service1
- match:
prefix: "/service/2"
route:
cluster: service2
http_filters:
- name: envoy.filters.http.router
typed_config: {}
clusters: #Set up cluster
- name: service1
connect_timeout: 0.25s
type: strict_dns
lb_policy: round_robin
http2_protocol_options: {}
load_assignment:
cluster_name: service1
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 10.244.1.1
port_value: 8000
- name: service2
connect_timeout: 0.25s
type: strict_dns
lb_policy: round_robin
http2_protocol_options: {}
load_assignment:
cluster_name: service2
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: service2
port_value: 8000
admin:
access_log_path: "/dev/null"
address:
socket_address:
address: 0.0.0.0
port_value: 8001
envoy dynamic configuration method
In addition to static configuration in the configuration file, listeners and clusters can also be dynamically obtained from the control plane. The service that issues dynamic configuration is called the control plane. Its address is statically configured in the configuration file in the form of cluster, and then in dynamic_ This resource is referenced in the cluster. The configurations that can be dynamically distributed mainly include CDS, LDS, EDS, RDS and SDS. Only the discovery addresses of CDS and LDs are in dynamic_ Specified in resources.
Only envoy with the same ID and Cluster in the configuration file will accept the configuration issued by this control plane.
File based approach
The file based approach requires that the configuration file path of xDS be specified in the envy container. Envy will use inotify to monitor the changes of the file system and resolve the DiscoveryResponse prototype in the file when updating.
node:
id: id_1
cluster: test
dynamic_resources:
cds_config:
path: /var/lib/envoy/cds.yaml
lds_config:
path: /var/lib/envoy/lds.yaml
admin:
access_log_path: "/dev/null"
address:
socket_address:
address: 0.0.0.0
port_value: 19000
Where cds_config and Las_ The config content is the same as the static configuration method:
#cds content
resources:
- "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster
name: example_proxy_cluster
connect_timeout: 1s
type: strict_dns
http2_protocol_options: {}
load_assignment:
cluster_name: example_proxy_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: service1
port_value: 8080
-----
#lds content
resources:
- "@type": type.googleapis.com/envoy.config.listener.v3.Listener
name: listener_0
address:
socket_address:
address: 0.0.0.0
port_value: 10001
filter_chains:
- filters:
name: envoy.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
http_filters:
- name: envoy.router
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains:
- "*"
routes:
- match:
prefix: "/"
route:
cluster: example_proxy_cluster
In this way, although the envoy route can be modified dynamically, the xds configuration file still needs to be changed manually, and envoy needs to be restarted after each modification to refresh.
Control plane based approach
Issue configuration with XDS
First, statically configure the address of the control plane in the configuration file, and then in cds_config and LDS_ The previously configured XDS is referenced in config_ cluster.
dynamic_resources:
cds_config:
api_config_source:
api_type: GRPC
grpc_services:
envoy_grpc:
cluster_name: xds_cluster
lds_config:
api_config_source:
api_type: GRPC
grpc_services:
envoy_grpc:
cluster_name: xds_cluster
Distribute configuration with ADS
Cds_config and las_config is separate, which means that the cluster and listener configurations can be obtained from different control planes. In this way, the configuration will not be synchronized. Even if they use the same control plane, they may not be synchronized due to different arrival orders. For example, cluster A is used in listener a, but the configuration of listener a may be distributed to envoy before cluster A, making envoy think that listener a uses a nonexistent cluster a.
ADS It supports the distribution of all types of dynamic configurations, and handles the dependencies between configurations to ensure the distribution order of configurations. It is the preferred configuration discovery method. See the description of the configuration distribution protocol for the implementation principle xDS REST and gRPC protocol.
node:
cluster: test-cluster
id: test-id
dynamic_resources:
ads_config:
api_type: GRPC
transport_api_version: V3
grpc_services:
- envoy_grpc:
cluster_name: xds_cluster
cds_config:
resource_api_version: V3
ads: {}
lds_config:
resource_api_version: V3
ads: {}
static_resources:
clusters:
- connect_timeout: 1s
type: strict_dns
http2_protocol_options: {}
name: xds_cluster
load_assignment:
cluster_name: xds_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: go-control-plane
port_value: 18000
admin:
access_log_path: /dev/null
address:
socket_address:
address: 0.0.0.0
port_value: 19000
lds/cds/rds/sds/eds of envoy
dynamic_ Only in the resources configuration lds_config,cds_config and ads_config , corresponding to listener, cluster and aggregation discovery respectively. But not only listener and cluster can be dynamically distributed.
The endpoint in the cluster, the secret used in tls, and the route used in HttpConnectionManager can also be dynamically distributed. The corresponding discovery services are eds,sds,rds . These discovery services are not in dynamic_ They are configured in resources, but independently or in filter s that use them.
Control plane implementation
Basic usage of go control plane
go-control-plane Is a framework that provides a configuration distribution interface, based on which other functions can be developed according to their own needs.
Dynamic configuration is associated with envoy, and the dynamic configuration of each envoy should be maintained separately.
There is a node configuration in the envoy configuration file, which indicates the cluster and id to which the current envoy belongs. Envoy only accepts dynamic configuration using the same cluster and id. the control plane implemented by go control plane should maintain a configuration for each envoy instance:
node := &core.Node{
Id: "envoy-64.58",
Cluster: "test",
}
node_config := &NodeConfig{
node: node,
endpoints: []cache.Resource{}, //[]*api_v2.ClusterLoadAssignment
clusters: []cache.Resource{}, //[]*api_v2.Cluster
routes: []cache.Resource{}, //[]*api_v2.RouteConfiguration
listeners: []cache.Resource{}, //[]*api_v2.Listener
}
Definition and distribution of filter in go control plane
Listener is the most complex configuration in envoy, and filter is the most cumbersome in listener. The request received from the socket passes first listener_filters Processing, and then by filter_chains The filter contained in the former is called listener filter, and the filter contained in the latter is called network filter. Because listener_filters work first, so it can modify the requested information to affect the filter_ Match of chains. Filters fall into the following categories:
Listener filters Network filters HTTP filters Thrift filters Common access log types Common fault injection types Dubbo filters
Filter is filled in the listener and distributed as the listener configuration. The listener go-control-plane/envoy/api/v2/lds.pb.go The interface of filter is defined in envoy/api/v2/listener/listener.pb.go The implementation of filter is defined in go-control-plane/envoy/config/filter/ in