1, Problem description
After upgrading the platform store on site, it was found that the service did not start successfully. The error log is as follows:
Caused by: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Abnormal network communication at com.zaxxer.hikari.pool.HikariPool.throwPoolInitializationException(HikariPool.java:576) at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:562) at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115) at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:81) at com.baomidou.dynamic.datasource.creator.HikariDataSourceCreator.createDataSource(HikariDataSourceCreator.java:90) at com.baomidou.dynamic.datasource.creator.DefaultDataSourceCreator.createDataSource(DefaultDataSourceCreator.java:68) at com.baomidou.dynamic.datasource.provider.AbstractDataSourceProvider.createDataSourceMap(AbstractDataSourceProvider.java:44) at com.baomidou.dynamic.datasource.provider.YmlDynamicDataSourceProvider.loadDataSources(YmlDynamicDataSourceProvider.java:42) at com.baomidou.dynamic.datasource.DynamicRoutingDataSource.afterPropertiesSet(DynamicRoutingDataSource.java:229) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1837) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1774) ... 121 common frames omitted Caused by: dm.jdbc.driver.DMException: Abnormal network communication Communication error at dm.jdbc.driver.DBError.throwException(DBError.java:683) at dm.jdbc.c.a.a(DBAccess.java:764) at dm.jdbc.c.a.r(DBAccess.java:143) at dm.jdbc.driver.DmdbConnection.openConnection(DmdbConnection.java:660) at dm.jdbc.driver.DmDriver.do_connect(DmDriver.java:183) at dm.jdbc.driver.DmDriver.connect(DmDriver.java:458) at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:136) at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:369) at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:198) at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:467) at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:541) ... 130 common frames omitted Caused by: java.io.IOException: The software in your host has aborted an established connection. at sun.nio.ch.SocketDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at dm.jdbc.util.Buffer$Node.load(Buffer.java:1181) at dm.jdbc.util.Buffer$Node.access$6(Buffer.java:1164) at dm.jdbc.util.Buffer.load(Buffer.java:351) at dm.jdbc.c.a.c(DBAccess.java:855) at dm.jdbc.c.a.a(DBAccess.java:757) ... 139 common frames omitted
In the on-site multi tenant environment, there are occasional phenomena. Some tenants can be successfully upgraded and started, and some tenants cannot be successfully started.
Network communication anomaly is a difficult anomaly to locate, especially using the less commonly used Dameng database. However, after confirming that the Damon service is running normally and other tenants have successfully upgraded, the problem is easier to locate. The database times out and most connections are rejected.
Background let's introduce the content of this upgrade again: the store newly supports Shentong Jincang database and uses the dynamic data source of mybatis plus.
// mybatis-plus
implementation group: 'com.baomidou', name: 'mybatis-plus-boot-starter', version: '3.4.3.1'
// Multiple data sources
implementation group: 'com.baomidou', name: 'dynamic-datasource-spring-boot-starter', version: '3.4.0'
2, Problem elimination
1. To change the timeout configuration of a data source:
Let's stick to the current configuration first. The data source is roughly configured as follows
spring:
application:
name: store
# profiles:
# include: ca
datasource:
type: com.zaxxer.hikari.HikariDataSource
hikari:
#Minimum number of free connections
minimum-idle: 5
#Connections returned from the pool are automatically committed by default
auto-commit: true
#Maximum idle connection time, 10 seconds
idle-timeout: 10000
#Connection pool name
pool-name: StoreHikariCP
#Maximum lifetime of connections in the pool
max-lifetime: 1800000
#Timeout for database connection
connection-timeout: 30000
# Old single data source configuration
# url: jdbc:dm://localhost
# username: LALA
# password: LALALALA123
# driver-class-name: dm.jdbc.driver.DmDriver
# Add multi data source configuration
dynamic:
primary: dm #Set the default data source or data source group. The default value is master
strict: true #Strictly match the data source. The default is false True throws an exception when it does not match the specified data source, and false uses the default data source
datasource:
dm:
url: jdbc:dm://localhost
username: LALA
password: LALALALA123
driver-class-name: dm.jdbc.driver.DmDriver
# kingbasees:
# url: jdbc:kingbase8://localhost:54321/springbootv2
# username: LALA
# password: LALALALA123
# driver-class-name: com.kingbase8.Driver
# oscar:
# url: jdbc:oscar://locahost:2003/OSRDB?serverTimezone=UTC&useSSL=FALSE
# username: LALA
# password: LALALALA123
# driver-class-name: com.oscar.Driver
# mysql:
# url: jdbc:mysql://localhost:3306/am?characterEncoding=UTF-8&serverTimezone=UTC
# username: LALA
# password: LALALALA123
# driver-class-name: com.mysql.cj.jdbc.DriverBy changing the above data source connection time, the database timeout is doubled, and the problem remains unsolved.
2. Check the number of database connections
The default Damon database is used. View the Damon database configuration DM ini
#IO DIRECT_IO = 0 #Flag For Io Mode(Non-Windows Only), 0: Using File System Cache; 1: Without Using File System Cache IO_THR_GROUPS = 2 #The Number Of Io Thread Groups(Non-Windows Only) HIO_THR_GROUPS = 2 #The Number Of Huge Io Thread Groups(Non-Windows Only) #database MAX_SESSIONS = 100 #Maximum number of concurrent sessions MAX_CONCURRENT_TRX = 0 #Maximum number of concurrent transactions MAX_SESSION_STATEMENT = 20000 #Maximum number of statement handle of one session MAX_CONCURRENT_OLAP_QUERY = 0 #Maximum number of concurrent OLAP queries BIG_TABLE_THRESHHOLD = 1000 #Threshhold value of a big table in 10k MAX_EP_SITES = 64 #Maximum number of EP sites for MPP PORT_NUM = 5236 #Number Of Database Server Listening Port
Pay attention to Max above_ The default maximum number of sessions is 100. Looking back at the program data source configuration, the minimum number of idle connections is 5
15 tenants, 15 * 5 = 75, should not be a problem with the number of connections. Make sure that the number of connections is normal. Execute sql
select * from v$sessions
View results
There were 10 sessions, which did not meet the expected 5. That is, the previously configured hikari is not effective.
View the connection pool configuration in the official document
It needs to be configured in spring dataresource. Under dynamic level, add the configuration here
spring:
application:
name: store
# profiles:
# include: ca
datasource:
###Omit some
# Add multi data source configuration
dynamic:
hikari: # Global hikariCP parameter, all values are consistent with the default. (the supported parameters are as follows. Don't set them randomly if you don't know the meaning)
max-pool-size: 5
min-idle: 5
primary: dm #Set the default data source or data source group. The default value is master
strict: true #Strictly match the data source. The default is false True throws an exception when it does not match the specified data source, and false uses the default data source
datasource:
dm:
url: jdbc:dm://localhost
username: LALA
password: LALALALA123
driver-class-name: dm.jdbc.driver.DmDriver
###Omit someNote: the default maximum number of connections and minimum number of idle connections are 10.
After changing the configuration, restart and query again, and the configuration takes effect. The number of free connections is updated to 5.