With the rise of micro service architecture, maintaining service security has become more challenging than before. Multiple back-end server instances using a single static key to access the database server will bring huge risks. If the key certificate is leaked, the whole system will be affected. In order to solve the impact of key certificate disclosure, the key certificate can only be revoked. Revoking the key certificate will lead to large-scale service interruption. For developers, large-scale service interruption is the last thing developers want to see.
Although we cannot predict in advance which security vulnerabilities will appear in the future, we can control the impact scope of these security vulnerabilities by configuring multiple keys. To avoid this situation, like HashiCorp Vault (hereinafter referred to as Vault) this key certificate solution came into being. This article demonstrates how to integrate Vault with the JWT auth plug-in of Apache APISIX to provide services with excellent performance of high concurrency and low latency, while safeguarding the security of services.
What is Vault
Vault is designed to help users manage access to service keys and securely transfer them between multiple services. The key can be any form of certificate. Because the key can be used to unlock sensitive information, it needs to be closely controlled. The key can be in the form of password, API key, SSH key, RSA token or OTP. In fact, key leakage is very common: the key is usually stored in the configuration file or as a variable in the code. If it is not properly saved, the key will even appear in the publicly visible code bases such as GitHub, BitBucket or GitLab, which poses a major threat to security. Vault solves this problem by centralizing the key. It provides encrypted storage for static keys, generates dynamic keys with TTL leases, and authenticates users to ensure that they have access to specific keys. Therefore, even in the case of security vulnerabilities, the scope of influence is much smaller and can be better controlled.
Vault provides a user interface for key management, making it easy to control and manage permissions. Moreover, it also provides a flexible and detailed audit log function, which can track the historical access records of all users.
[external link image transfer failed. The source station may have anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-snn47ey4-1645609372957)( https://tfzcfxawmk.feishu.cn/...)]
Introduction to JWT auth plug-in
JWT auth is an authentication plug-in that can be attached to any APIs IX route to perform JWT authentication before the request is forwarded to the upstream URI. Typically, the publisher uses a private key or text key to sign the JWT. The recipient of the JWT will verify the signature to ensure that the token has not been changed after being signed by the issuer. The overall integrity of the entire JWT mechanism depends on the signing key (or the text key of the RSA key pair). This makes it difficult for unauthenticated sources to guess the signature key and try to change the declaration in JWT.
Therefore, it is very critical to store these keys in a secure environment. If the key falls into the wrong hands, it may endanger the security of the whole infrastructure. Although Apache apisik has taken all means to follow the standard SecOps practice, it is also a good thing to have a centralized key management solution in the production environment. For example, Vault has detailed audit log, regular key rotation, key revocation and other functions. If you have to update the Apache apifix configuration every time the key rotation occurs in the whole infrastructure, it will be a very troublesome problem.
How to integrate Vault and Apache apifix
In order to integrate with Vault, Apache APIs IX needs to config.yaml Load the relevant configuration information of the Vault in the file.
Apache APIs IX and Vault server KV Secrets Engine v1 HTTP APIs Communicate. Since most enterprise solutions tend to use KV Secrets Engine - Version 1 in the production environment, we only use this version in the initial stage of apisik vault support. In future versions, we will add support for K/V - Version 2.
The main concern of using Vault instead of APISIX etcd backend is security in a low trust environment. Because the Vault access token is small-scale, limited permissions can be granted to apisik server.
Configure Vault
This section shares best practices for using Vault in the Apache apisik ecosystem. If you already have a Vault instance running with the necessary permissions, skip this section.
Step 1: start the Vault server
Here, you have many choices. You can choose docker, precompiled binary package or build from source code. For communication with the Vault server, you need a Vault CLI client. Run the following command to start the server:
$ vault server -dev -dev-root-token-id=root ... WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory and starts unsealed with a single unseal key. The root token is already authenticated to the CLI, so you can immediately begin using Vault. You may need to set the following environment variable: export VAULT_ADDR='http://127.0.0.1:8200' The unseal key and root token are displayed below in case you want to seal/unseal the Vault or re-authenticate. Unseal Key: 12hURx2eDPKK1tzK+8TkgH9pPhPNJFpyfc/imCLgJKY= Root Token: root Development mode should NOT be used in production installations!
Set the Vault CLI client with the correct environment variables.
$ export VAULT_ADDR='http://127.0.0.1:8200' $ export VAULT_TOKEN='root'
Enable the key engine backend of vault k/v version 1 with an appropriate path prefix. In this demonstration, we will choose the kv path so that it will not conflict with the key path of vault's default kv version 2.
$ vault secrets enable -path=kv -version=1 kv Success! Enabled the kv secrets engine at: kv/ # To reconfirm the status, run $ vault secrets list Path Type Accessor Description ---- ---- -------- ----------- cubbyhole/ cubbyhole cubbyhole_4eeb394c per-token private secret storage identity/ identity identity_5ca6201e identity store kv/ kv kv_92cd6d37 n/a secret/ kv kv_6dd46a53 key/value secret storage sys/ system system_2045ddb1 system endpoints used for control, policy and debugging
Step 2: generate a Vault access token for Apache APIs IX
This article is about using Vault in JWT auth plug-ins. Therefore, for an apifix consumer jack, JWT auth plug-in will be in < vault prefix inside config.yaml>/consumer/<consumer. Username > / JWT auth to find (if vault configuration is enabled) secret/s to the vault key value pair store. In this case, if you specify the kV / apisik namespace (Vault path) as config Vault in yaml Prefix is used to retrieve all apifix related data. We recommend that you create a policy for the path kV / apifix / consumer /. The last asterisk (*) ensures that the policy allows reading of any path with the prefix kV / apifix / consumer.
Create a policy file with HashiCorp configuration language (HCL).
$ tee apisix-policy.hcl << EOF
path "kv/apisix/consumer/*" {
capabilities = ["read"]
}
EOFApply policies to Vault instances.
$ vault policy write apisix-policy apisix-policy.hcl Success! Uploaded policy: apisix-policy
Generate a token with the newly defined policy, which has been configured as a small access boundary.
$ vault token create -policy="apisix-policy" Key Value --- ----- token s.KUWFVhIXgoRuQbbp3j1eMVGa token_accessor nPXT3q0mfZkLmhshfioOyx8L token_duration 768h token_renewable true token_policies ["apisix-policy" "default"] identity_policies [] policies ["apisix-policy" "default"]
In this demo, s.KUWFVhIXgoRuQbbp3j1eMVGa is your access token.
Add Vault configuration in Apache apifix
Apache APIs IX communicates with Vault instances through Vault HTTP APIs. The necessary configuration must be added to the config.yaml Yes.
Here is a brief information about the different fields you can use.
- Host: the address of the host where the Vault server is running.
- Timeout: HTTP timeout for each request.
Token: the token generated from the vault instance can grant the permission to read data from the vault.
- Prefix: enabling prefix can better implement policies, generate a limited range of tokens, and strictly control the data that can be accessed from apifix. Valid prefixes are (kV / apisik, secret, etc.).
vault: host: 'http://0.0.0.0:8200' timeout: 10 token: 's.KUWFVhIXgoRuQbbp3j1eMVGa' prefix: 'kv/apisix'
Create an apisik consumer
Apisid has a consumer level abstraction, juxtaposed with authentication schemes. In order to enable authentication of any APISIX route, a consumer with a configuration suitable for that particular type of authentication service is required. Then, only apisid can forward the request to the upstream URI by successfully performing authentication in terms of consumer configuration. Apisex consumers have two fields: one is username (required), which is used to identify consumers, and the other is plugins, which is used to save the specific plug-in configuration used by consumers.
Here, in this article, we will create a consumer using the JWT auth plug-in. It executes for the respective route or service JWT certification.
Run the following command to enable JWT auth for Vault configuration.
$ curl http://127.0.0.1:9080/apisix/admin/consumers -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"username": "jack",
"plugins": {
"jwt-auth": {
"key": "test-key",
"vault": {}
}
}
}'In this example, the plug-in finds the key secret in the Vault path of the consumer user jack mentioned in the consumer configuration (< Vault. Prefix from conf.yaml > / consumer / jack / jwt auth) and uses it for subsequent signature and jwt authentication. If the key is not found in the same path, the plug-in logs an error and cannot perform jwt authentication.
Set up an upstream server for testing
To test this behavior, you can create a route for an upstream (a simple ping handler that returns a pong). You can set it up with a normal go HTTP server.
// simple upstream server
package main
import "net/http"
func ping(w http.ResponseWriter, req *http.Request) {
w.Write([]byte("secure/pong\n"))
}
func main() {
http.HandleFunc("/secure/ping", ping)
http.ListenAndServe(":9999", nil)
}Here, the plug-in finds the key secret in the Vault path (< Vault. Prefix from conf.yaml > / consumer / jack / jwt auth) of the consumer jack mentioned in the consumer configuration, and uses it for subsequent signature and jwt verification. If the key is not found in the same path, the plug-in logs an error and cannot perform jwt authentication.
Create an apisex route with authentication enabled
Use this secure ping HTTP server and the JWT auth authentication plug-in to create an apisex route.
$ curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"plugins": {
"jwt-auth": {}
},
"upstream": {
"nodes": {
"127.0.0.1:9999": 1
},
"type": "roundrobin"
},
"uri": "/secure/ping"
}'Generate token from JWT auth plug-in
Now sign a jwt ciphertext from apifix, which can be used and passed to the apifix server http://localhost:9080/secure/ping The proxy routes the request.
$ curl http://127.0.0.1:9080/apisix/plugin/jwt/sign\?key\=test-key -i HTTP/1.1 200 OK Date: Tue, 18 Jan 2022 07:50:57 GMT Content-Type: text/plain; charset=utf-8 Transfer-Encoding: chunked Connection: keep-alive Server: APISIX/2.11.0 eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJ0ZXN0LWtleSIsImV4cCI6MTY0MjU3ODY1N30.nkyev1_KUapVgY_QVYETsSApA6gEkDWS8tsHFV1EpD8
In the previous step, if you see information like jwt signing failure, please make sure you have a private key stored in the vault kV / apifix / consumers / Jack / jwt auth path.
# example $ vault kv put kv/apisix/consumer/jack/jwt-auth secret=$ecr3t-c0d3 Success! Data written to: kv/apisix/consumer/jack/jwt-auth
Send request to apisex server
Now, issue a request to route / secure/ping to the APIs IX agent. After successful verification, it will forward the request to our go HTTP server.
$ curl http://127.0.0.1:9080/secure/ping -H 'Authorization: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJ0ZXN0LWtleSIsImV4cCI6MTY0MjU3ODU5M30.IYudBr7FTgRme70u4rEBoYNtGmGByzgfGlt8hctI__Q' -i HTTP/1.1 200 OK Content-Type: text/plain; charset=utf-8 Content-Length: 12 Connection: keep-alive Date: Tue, 18 Jan 2022 08:00:04 GMT Server: APISIX/2.11.0 secure/pong
Any invalid jwt request will throw an HTTP 401 unauthorized error.
$ curl http://127.0.0.1:9080/secure/ping -i
HTTP/1.1 401 Unauthorized
Date: Tue, 18 Jan 2022 08:00:33 GMT
Content-Type: text/plain; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Server: APISIX/2.11.0
{"message":"Missing JWT token in request"}Different use cases where Vault can be integrated with APIs IX JWT auth plug-ins
The Apache APIs IX JWT auth plug-in can be configured to obtain simple text keys and RS256 public-private key pairs from Vault storage.
Note: for earlier versions supported by the integration, the plug-in wants the key name stored in the vault path to be between [* secret*, *public_key*, *private_key *] to successfully use the key. In future releases, we will add support for referencing custom named keys.
- You have stored the HS256 signature key in the vault. You want to use it for jwt signature and verification.
$ curl http://127.0.0.1:9080/apisix/admin/consumers -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"username": "jack",
"plugins": {
"jwt-auth": {
"key": "key-1",
"vault": {}
}
}
}'Here, the plug-in finds the key secret in the Vault path (< Vault. Prefix from conf.yaml > / consumer / jack / jwt auth) of the consumer user name jack mentioned in the consumer configuration, and uses it for subsequent signature and jwt verification. If the key is not found in the same path, the plug-in logs an error and cannot perform jwt validation.
- RS256 RSA key pair, public key and private key are stored in Vault.
$ curl http://127.0.0.1:9080/apisix/admin/consumers -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"username": "jim",
"plugins": {
"jwt-auth": {
"key": "rsa-keypair",
"algorithm": "RS256",
"vault": {}
}
}
}'The plug-in finds public for the user jim mentioned in the plug-in vault configuration in the vault key value pair path (< vault.prefix from conf.yaml > / consumer / jim / JWT auth)_ Key and private_key. If not found, authentication fails.
Use this command if you are not sure how to store the public and private keys in a Vault key value pair
# provided, your current directory contains the files named "public.pem" and "private.pem" $ vault kv put kv/apisix/consumer/jim/jwt-auth public_key=@public.pem private_key=@private.pem Success! Data written to: kv/apisix/consumer/jim/jwt-auth
- In the Vault, the public key of the consumer is configured.
$ curl http://127.0.0.1:9080/apisix/admin/consumers -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"username": "john",
"plugins": {
"jwt-auth": {
"key": "user-key",
"algorithm": "RS256",
"public_key": "-----BEGIN PUBLIC KEY-----\n......\n-----END PUBLIC KEY-----"
"vault": {}
}
}
}'This plug-in uses the RSA public key from the consumer configuration and the private key obtained directly from the Vault.
Disable Vault plug-in
Now, to disable the jwt auth plug-in's vault query, simply delete the empty vault object (jack in this case) from the consumer plug-in configuration. This will enable the jwt plug-in to incorporate the lookup signature key (including HS256/HS512 or RS512 key pair) into the plug-in configuration in subsequent requests for URI routing with jwt auth configuration enabled. Even if you're in apifix config Vault configuration is enabled in yaml, and no requests are sent to the vault server.
The APIs IX plug-in is hot loaded, so there is no need to restart APIs IX.
$ curl http://127.0.0.1:9080/apisix/admin/consumers -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"username": "jack",
"plugins": {
"jwt-auth": {
"key": "test-key",
"secret": "my-secret-key"
}
}
}'