Friendly tip: to download the source code for transformation, this paper is based on Sentinel-1.8.2.
The following contents are extracted from personal technical documents, and relevant materials are mainly from spring cloud Alibaba and GitHub sentinel. The text is part of the excerpts for reference only.
4.1.2. Gateway flow control
Gateway flow restriction rules are customized flow restriction rules for API Gateway scenarios. They can limit flow for different route s or customized API packets, and support customized flow restriction for parameters, headers, source IP, etc. in the request.
Similar to normal application flow control, gateway flow control will return such results after triggering.
{
"code":429,
"message":"Sentinel block exception",
"route":"/"
}
4.1.2.1. Infrastructure
Front end code
- Front end page: webapp / resources / APP / views / gateway / flow html
- JS file: webapp / resources / APP / scripts / Controllers / gateway / flow js
Background code
- Package: com.alibaba.csp.sentinel.dashboard.controller.gateway
- Class: GatewayApiCountroller
- GET /gateway/api/list.json - query flow control rules
- POST /gateway/api/new.json - add flow control rule
- PUT /gateway/api/save.json - modify flow control rules
- DELETE /gateway/api/delete.json - delete flow control rule
4.1.2.2. Query flow control rules
Interface input parameter
| Parameter name | type | Rule constraint | explain |
|---|---|---|---|
| app | String | String is not empty | Gateway gateway ApplicationName, corresponding to spring in gateway configuration file application. The value of the name attribute |
| ip | String | String is not empty | Gateway server IP |
| port | Integer | Non empty | Gateway server port, corresponding to sentinel in gateway configuration file transport. Value of port |
Internal logic
4.1.2.3. Add flow control rule
Interface input parameter
| Parameter name | type | Rule constraint | explain |
|---|---|---|---|
| app | String | String is not empty | apply name |
| ip | String | String is not empty | Application IP |
| port | Integer | Application port number | |
| resource | String | String is not empty | Resources targeted by flow control |
| resourceMode | Integer | Non empty, RESOURCE_MODE_ROUTE_ID=0； RESOURCE_MODE_CUSTOM_API_NAME=1 | Resource model |
| grade | Integer | Non empty, 0 is the number of threads, 1 is qps | The current limit indicator dimension is the same as the grade field of the current limit rule |
| count | Double | Greater than or equal to 0 | Current limiting threshold |
| interval | Long | Greater than 0 | Statistics interval: 1 by default |
| intervalUnit | Integer | Second, minute, hour, day | The statistical interval unit is seconds by default |
| controlBehavior | Integer | Non empty, 0-fast failure, 2-uniform queuing | The control effect of traffic shaping is the same as that of the controlBehavior field of the flow restriction rule. At present, two modes of fast failure and uniform queuing are supported. The default is fast failure. |
| burst | Integer | Required for quick failure, greater than or equal to 0 | Number of additional requests allowed in response to sudden requests |
| maxQueueingTimeoutMs | Integer | Required when queuing at constant speed, greater than or equal to 0 | The longest queuing time in milliseconds under the uniform queuing mode. It only takes effect under the uniform queuing mode |
| paramItem | GPFIV | Parameter current limiting configuration |
GPFIV: GatewayParamFlowItemVo if not provided, it means that the current is not limited for the parameters, and the gateway rule will be converted into a common flow control rule; Otherwise, it will be converted to a hotspot rule.
| Parameter name | type | Rule constraint | explain |
|---|---|---|---|
| parseStrategy | Integer | 0-ClientIP 1-Remote Host 2-Header 3-URL parameter4-cookie | Policy for extracting parameters from requests |
| fieldName | String | When the parameter property is 2-Header 3-URL 4-Cookie, the parameter name is required | If the extraction policy selects header mode or URL parameter mode, you need to specify the corresponding header name or URL parameter name |
| pattern | String | Parameter value matching mode. Only the request attribute value matching the mode will be included in statistics and flow control; If it is blank, all values of the request attribute will be counted. (supported from version 1.6.2) | |
| matchStrategy | Integer | PARAM_MATCH_STRATEGY_EXACT, PARAM_MATCH_STRATEGY_CONTAINS, and PARAM_MATCH_STRATEGY_REGEX | Matching strategy for parameter values (supported from version 1.6.2) |
Internal logic
4.1.2.4. Modify flow control rules
Interface input parameter
Except that some parameters are replaced by id, other parameters are the same as the input parameters of the new interface
| Parameter name | type | Rule constraint | explain |
|---|---|---|---|
| id | Long | Non empty | Rule id |
| grade | Integer | Non empty, 0 is the number of threads, 1 is qps | The current limit indicator dimension is the same as the grade field of the current limit rule |
| count | Double | Greater than or equal to 0 | Current limiting threshold |
| interval | Long | Greater than 0 | Statistics interval: 1 by default |
| intervalUnit | Integer | Second, minute, hour, day | The statistical interval unit is seconds by default |
| controlBehavior | Integer | Non empty, 0-fast failure, 2-uniform queuing | The control effect of traffic shaping is the same as that of the controlBehavior field of the flow restriction rule. At present, two modes of fast failure and uniform queuing are supported. The default is fast failure. |
| burst | Integer | Required for quick failure, greater than or equal to 0 | Number of additional requests allowed in response to sudden requests |
| maxQueueingTimeoutMs | Integer | Required when queuing at constant speed, greater than or equal to 0 | The longest queuing time in milliseconds under the uniform queuing mode. It only takes effect under the uniform queuing mode |
| paramItem | GPFIV | Parameter current limiting configuration |
Internal logic (same as adding)
4.1.2.5. Delete flow control rule
Interface input parameter
| Parameter name | type | Rule constraint | explain |
|---|---|---|---|
| id | Long | Non empty | Rule id |
Internal logic (roughly the same as adding and modifying, and will not be repeated)
5. Dynamic data source
5.1. Pre knowledge
5.1.1. What is a dynamic data source
In the open source framework (at least as of version 1.8.2), all rules are stored in memory (in the form of Map in the JVM). However, in the production environment, we usually want to have a fixed configuration file or data source to complete the persistence and use of these rules, which is the dynamic data source.
This project is Sentinel Dashboard, not Sentinel. Since Dashboard provides relatively complete rule management functions, we expect to configure rules through Dashboard instead of modifying text through configuration centers such as Nacos. Then, in order to meet such requirements, it is necessary to integrate the dynamic data source into the Dashboard, so as to persist the rules configured in the Dashboard to the dynamic data source.
Sentinel (including Dashboard) provides two dynamic data source schemes:
-
In pull mode, the client (integrated Sentinel application) actively polls a rule management center for pull rules regularly. This rule center can be RDBMS, files, or even VCS. The way to do this is simple, but the disadvantage is that changes cannot be obtained in time;
-
Push mode, the rule center pushes uniformly, and the client (integrating Sentinel's application) monitors changes at any time by registering a listener, such as using Nacos, Zookeeper and other configuration centers. This method has better real-time and consistency guarantee.
Sentinel currently supports the following data source extensions:
- Pull based: dynamic file data source Consul, Eureka
- Push-based: ZooKeeper, Redis, Nacos, Apollo, etcd
5.1.2. Pull mode expansion
The simplest way to implement a pull pattern data source is inheritance AutoRefreshDataSource Abstract class, and then implement the readSource() method, in which the configuration data in string format is read from the specified data source. such as File based data source.
5.1.3. Push mode expansion
The simplest way to implement a data source in push mode is inheritance AbstractDataSource Abstract class, add a listener to its constructor, and implement readSource() to read configuration data in string format from the specified data source. such as Nacos based data source.
The console usually needs to be modified to directly push the rules of the application dimension to the configuration center. Function examples can be referred to Rule push function of AHAS Sentinel console . Modification guidelines can be referred to Using Sentinel console in production environment.
5.1.4. Register data source
This section describes the steps you need to do when using Sentinel alone. If you use spring cloud Alibaba to integrate Sentinel, you don't need to.
You usually need to call the following method to register the data source in the specified rule manager:
ReadableDataSource<String, List<FlowRule>> flowRuleDataSource = new NacosDataSource<>(remoteAddress, groupId, dataId, parser); FlowRuleManager.register2Property(flowRuleDataSource.getProperty());
If you do not want to register the data source manually, you can use Sentinel's InitFunc SPI extension interface. You only need to implement your own InitFunc interface and write the logic to register the data source in the init method. For example:
package com.test.init;
public class DataSourceInitFunc implements InitFunc {
@Override
public void init() throws Exception {
final String remoteAddress = "localhost";
final String groupId = "Sentinel:Demo";
final String dataId = "com.alibaba.csp.sentinel.demo.flow.rule";
ReadableDataSource<String, List<FlowRule>> flowRuleDataSource = new NacosDataSource<>(remoteAddress, groupId, dataId,
source -> JSON.parseObject(source, new TypeReference<List<FlowRule>>() {}));
FlowRuleManager.register2Property(flowRuleDataSource.getProperty());
}
}
Then add the corresponding class name to com.inf under META-INF/services directory under resource directory (usually resource directory) alibaba. csp. sentinel. init. Initfunc file, for example:
com.test.init.DataSourceInitFunc
In this way, Sentinel can automatically register the corresponding data source when accessing any resource for the first time.
5.1.5. Official Expansion example
Sentinel's official source code provides an example of dynamic data source expansion for the application of flow control rules. Sentinel open source version provides a memory based rule engine. In order to use the data source for persistence, it is necessary to transform it.
The official has written a special sample code for Nacos as a dynamic data source, which is aimed at flow control rules.
Review the background code of flow control rules:
- Package: com.alibaba.csp.sentinel.dashboard.controller
- Class: FlowControllerV1
- GET /v1/flow/rules - query flow control rules
- POST /v1/flow/rule - add flow control rule
- PUT /v1/flow/save.json - modify flow control rules
- DELETE /v1/flow/delete.json - delete flow control rule
The sample code is:
- Package: com.alibaba.csp.sentinel.dashboard.controller.v2
- Class: FlowControllerV2
- GET /v2/flow/rules - query flow control rules
- POST /v2/flow/rule - add flow control rule
- PUT /v2/flow/save.json - modify flow control rules
- DELETE /v2/flow/delete.json - delete flow control rule
Sentinel official provides key samples of Nacos (test/com.alibaba.csp.sentinel.dashboard.rule.nacos) in the dashboard test package, including Apollo and Zookeeper. The following mainly introduces how to operate based on Nacos as a dynamic data source.
5.1.5.1. Copy sample class
At test / com alibaba. csp. sentinel. dashboard. rule. Under Nacos, there are four classes:
- FlowRuleNacosProvider
- FlowRuleNacosPublisher
- NacosConfig
- NacosConfigUtil
Copy them to SentinelDashboard. Since the Nacos service address filled in NacosConfig is localhost, we need to change it to the format of ip:port. Usually, we will define a configuration item, move it to the configuration file for definition or specify it with startup parameters.
5.1.5.2. Add data source dependency
Since the official set the data source dependency to the test range, we need to open it for use, as long as POM Just comment out the following tags in XML.
<!-- for Nacos rule publisher sample -->
<dependency>
<groupId>com.alibaba.csp</groupId>
<artifactId>sentinel-datasource-nacos</artifactId>
<scope>test</scope>
</dependency>
5.1.5.3. Modify front-end route
The default front-end access back-end interfaces mentioned above are:
- GET /v1/flow/rules - query flow control rules
- POST /v1/flow/rule - add flow control rule
- PUT /v1/flow/save.json - modify flow control rules
- DELETE /v1/flow/delete.json - delete flow control rule
Now I want to change it to / v2 /... So I need to transform the front end. Although the front end is developed by Angular, you don't have to worry about the whole stack to see what's going on.
Angular is similar to Vue and needs node JS to compile and package, so you need to install node JS, it is recommended to use 11.15.0, otherwise you may encounter some strange problems. If you want to manage multiple node versions at the same time, you can refer to this article Blog.
node related commands:
# Installation package npm -install # Operation (Development) npm start # Packaging (production) npm run build
The structure of the front-end project is roughly as follows (only the necessary structures are excerpted, others have been omitted):
-
webapp/resources/
-
app
- scripts
- controllers
- flow_v1.js --------------------------------- original flow control front end js
- flow_v2.js --------------------------------- dynamic data source front end js
- services
- flow_service_v1.js -------------------------- original flow control backend API definition
- flow_service_v2.js -------------------------- dynamic data source backend API definition
- directives
- sidebar
- sidebar.html ----------------------- define route
- sidebar
- app.js ---------------------------------------------- defines the global control, which is the core configuration
- controllers
- views
- flow_v1.html --------------------------------- original flow control rule configuration page
- flow_v2.html --------------------------------- dynamic data source flow control rule configuration page
- scripts
-
dist ------------------------------------------------------- store the files compiled and packaged in the production environment
-
tmp -------------------------------------------------------- store the compiled and packaged files of the development environment
-
gulpfile. JS ------------------------------------------------------- the place defined in the service file (to be used for subsequent extension of new code)
-
There are two key here, app JS and sidebar HTML, in app JS defines two state s that we can use directly. The contents are as follows:
.state('dashboard.flowV1', {
templateUrl: 'app/views/flow_v1.html',
url: '/flow/:app',
controller: 'FlowControllerV1',
resolve: {
loadMyFiles: ['$ocLazyLoad', function ($ocLazyLoad) {
return $ocLazyLoad.load({
name: 'sentinelDashboardApp',
files: [
'app/scripts/controllers/flow_v1.js',
]
});
}]
}
})
.state('dashboard.flow', {
templateUrl: 'app/views/flow_v2.html',
url: '/v2/flow/:app',
controller: 'FlowControllerV2',
resolve: {
loadMyFiles: ['$ocLazyLoad', function ($ocLazyLoad) {
return $ocLazyLoad.load({
name: 'sentinelDashboardApp',
files: [
'app/scripts/controllers/flow_v2.js',
]
});
}]
}
})
Correspondingly, the connection in the navigation on the left side of the console is in the sidebar Defined in HTML:
<!-- Original flow control rule navigation -->
<li ui-sref-active="active" ng-if="!entry.isGateway">
<a ui-sref="dashboard.flowV1({app: entry.app})">
<i class="glyphicon glyphicon-filter"></i> Flow control rules</a>
</li>
<!-- Change to dynamic data source flow control rule navigation -->
<li ui-sref-active="active" ng-if="!entry.isGateway">
<a ui-sref="dashboard.flow({app: entry.app})">
<i class="glyphicon glyphicon-filter"></i> Flow control rules</a>
Obviously, the dashboard in the tag Flowv1 is app JS.
After the front end is modified, you can start it directly or start it after packaging to debug to see whether it is successful.
5.2. Based on spring cloud gateway
Spring cloud gateway is relatively simple to integrate because Sentinel officials have made adaptation.
The gateway related codes are basically in the gateway package or directory, which is easy to find.
In order to retain the original code, basically new classes, pages, js, etc. are created for extension, and the original code will not be changed directly. The following is only for flow control process extension for reference.
5.2.1. New Publisher and Provider
According to the original GatewayFlowRuleController, you can know that the entity objects used in the interface are GatewayFlowRuleEntity, so you can directly provide a converter for it and write Publisher and Provider.
Provider
import com.alibaba.csp.sentinel.dashboard.config.NacosConfigUtil;
import com.alibaba.csp.sentinel.dashboard.datasource.entity.gateway.GatewayFlowRuleEntity;
import com.alibaba.csp.sentinel.datasource.Converter;
import com.alibaba.csp.sentinel.util.StringUtil;
import com.alibaba.nacos.api.config.ConfigService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import java.util.ArrayList;
import java.util.List;
@Component("gatewayFlowRuleNacosProvider")
public class GatewayFlowRuleNacosProvider implements DynamicRuleProvider<List<GatewayFlowRuleEntity>> {
@Autowired
private ConfigService configService;
// For the converter of rule objects, the obtained data needs to be converted by different converters according to the different data types used
@Autowired
private Converter<String, List<GatewayFlowRuleEntity>> converter;
@Override
public List<GatewayFlowRuleEntity> getRules(String appName) throws Exception {
String rules = configService.getConfig(appName + NacosConfigUtil.FLOW_DATA_ID_POSTFIX,
NacosConfigUtil.GROUP_ID, 3000);
if (StringUtil.isEmpty(rules)) {
return new ArrayList<>();
}
return converter.convert(rules);
}
}
Publisher
import com.alibaba.csp.sentinel.dashboard.config.NacosConfigUtil;
import com.alibaba.csp.sentinel.dashboard.datasource.entity.gateway.GatewayFlowRuleEntity;
import com.alibaba.csp.sentinel.datasource.Converter;
import com.alibaba.csp.sentinel.util.AssertUtil;
import com.alibaba.nacos.api.config.ConfigService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import java.util.List;
/**
* <h3>Sentinel Gateway flow control rule publisher for dynamic data source</h3>
*
* <p>Publish the rules configured through the console to the dynamic data source (nacos). Currently, the push mode is used,
* After the rules are pushed to nacos, the gateway will automatically obtain the latest rules through the listener and update them to the local buffer of the gateway
* In storage.
* </p>
*/
@Component("gatewayFlowRuleNacosPublisher")
public class GatewayFlowRuleNacosPublisher implements DynamicRulePublisher<List<GatewayFlowRuleEntity>> {
// Configuration service of data source
@Autowired
private ConfigService configService;
/**
* <p>Converter for data communication.
* <p>The Spring Bean object declared in the NacosConfig class under the config package.
* <p>Responsible for converting entity objects to json formatted strings</p>
*/
@Autowired
private Converter<List<GatewayFlowRuleEntity>, String> converter;
@Override
public void publish(String app, List<GatewayFlowRuleEntity> rules) throws Exception {
AssertUtil.notEmpty(app, "app name cannot be empty");
if (rules == null) {
return;
}
/*
* Publish the rules to the dynamic data source for persistence. The first parameter is app + suffix. Here, the suffix of - flow rules is used;
* The second parameter is the nacos group id, which can use the sentinel reservation provided by default; The last parameter is data conversion
* After the object is converted into a unified format, it is transmitted to nacos through the network.
*/
configService.publishConfig(app + NacosConfigUtil.FLOW_DATA_ID_POSTFIX,
NacosConfigUtil.GROUP_ID, converter.convert(rules));
}
}
5.2.2. Add Converter
Add a converter < list < gatewayflowruleentity >, string > type converter, which can be directly added to the NacosConfig copied earlier:
@Bean
public Converter<List<GatewayFlowRuleEntity>, String> gatewayFlowRuleEntityEncoder() {
return JSON::toJSONString;
}
@Bean
public Converter<String, List<GatewayFlowRuleEntity>> gatewayFlowRuleEntityDecoder() {
return s -> JSON.parseArray(s, GatewayFlowRuleEntity.class);
}
5.2.3. GatewayFlowRuleControllerV2
-
A new GatewayFlowRuleControllerV2 is added, and the content can be directly copied to the GatewayFlowRuleController.
-
Change the original interface address: / gateway / flow - > / V2 / gateway / flow
-
Remove the original sentinelApiClient
@Autowired private SentinelApiClient sentinelApiClient;
-
Join Provider and Publisher
@Autowired @Qualifier("gatewayFlowRuleNacosProvider") private DynamicRuleProvider<List<GatewayFlowRuleEntity>> ruleProvider; @Autowired @Qualifier("gatewayFlowRuleNacosPublisher") private DynamicRulePublisher<List<GatewayFlowRuleEntity>> rulePublisher; -
Modify publishRules() method
private boolean publishRules(String app, String ip, Integer port) { List<GatewayFlowRuleEntity> rules = repository.findAllByMachine(MachineInfo.of(app, ip, port)); return sentinelApiClient.modifyGatewayFlowRules(app, ip, port, rules); } // Change to /** * <h3>Unified logic of Publishing Rules</h3> * * <p>Rules are stored in local memory. First, get all the rules applied by the rules to be published from memory, which is a list</p> * <p>Publish the full amount of rules to the data source in a certain format for unified update</p> * * @param app apply name * @param ip Application IP * @param port Application port * @throws Exception If you publish remotely, an exception will occur. You need to handle the exception */ private void publishRules(String app, String ip, Integer port) throws Exception { List<GatewayFlowRuleEntity> rules = repository.findAllByMachine(MachineInfo.of(app, ip, port)); rulePublisher.publish(app, rules); } -
Modify the logic in the controller, mainly for reading rules and saving rules:
-
queryFlowRules
List<GatewayFlowRuleEntity> rules = sentinelApiClient.fetchGatewayFlowRules(app, ip, port).get(); // Change to List<GatewayFlowRuleEntity> rules = ruleProvider.getRules(app);
-
addFlowRule
try { entity = repository.save(entity); } catch (Throwable throwable) { logger.error("add gateway flow rule error:", throwable); return Result.ofThrowable(-1, throwable); } if (!publishRules(app, ip, port)) { logger.warn("publish gateway flow rules fail after add"); } // Change to try { entity = repository.save(entity); publishRules(entity.getApp(), entity.getIp(), entity.getPort()); } catch (Throwable throwable) { logger.error("add gateway flow rule error:", throwable); return Result.ofThrowable(-1, throwable); } -
updateFlowRule (similar to addFlowRule)
-
deleteFlowRule (similar to addFlowRule)
-
5.2.4. Front end transformation
-
On app JS to define a new gateway state
// New state .state('dashboard.gatewayFlowV2', { templateUrl: 'app/views/gateway/flow_v2.html', url: '/gateway/flow/:app', controller: 'GatewayFlowCtlV2', resolve: { loadMyFiles: ['$ocLazyLoad', function ($ocLazyLoad) { return $ocLazyLoad.load({ name: 'sentinelDashboardApp', files: [ 'app/scripts/controllers/gateway/flow_v2.js', ] }); }] } }) // This is the original, at the bottom .state('dashboard.gatewayFlow', { templateUrl: 'app/views/gateway/flow.html', url: '/gateway/flow/:app', controller: 'GatewayFlowCtl', resolve: { loadMyFiles: ['$ocLazyLoad', function ($ocLazyLoad) { return $ocLazyLoad.load({ name: 'sentinelDashboardApp', files: [ 'app/scripts/controllers/gateway/flow.js', ] }); }] } }); -
Retrofit sidebar HTML, add new navigation to the new controller, and comment out the old one
<li ui-sref-active="active" ng-if="entry.isGateway"> <a ui-sref="dashboard.gatewayFlowV2({app: entry.app})"> <i class="glyphicon glyphicon-filter"></i> Gateway flow control rules</a> </li> -
Add a new gateway flow control page, views/gateway/flow_v2.html
-
Copy Controllers / gateway / flow js -> controllers/gateway/flow_ V2. JS, just change the place with V2
app.controller('GatewayFlowCtlV2', ['$scope', '$stateParams', 'GatewayFlowServiceV2', 'GatewayApiService', 'ngDialog', 'MachineService', function ($scope, $stateParams, GatewayFlowServiceV2, GatewayApiService, ngDialog, MachineService) { -
Copy services / gateway / flow_ service_ v1. js -> services/gateway/flow_ service_ v2. js
app.service('FlowServiceV2', ['$http', function ($http) {Change FlowService to FlowServiceV2, and then prefix all request paths with / v2
-
gulpfile.js to add the definition of FlowServiceV2:
const JS_APP = [ ......, 'app/scripts/services/gateway/flow_service_v2.js', ]
-
Execute npm run build, empty the browser cache, and then test.