BoneCP Connection Pool Reconnection Mechanism Analysis

1. Background

BoneCP used by friend company Mysql connection pool, the application accesses Mysql as domain name, which is configured as follows:


All middleware access is through the Dns server on the intranet.


The machine on which Mysql was recently used for an application has become very old. It has gone down several times and is ready to migrate the database to a new machine. To reduce the number of operations, you do not want to modify the application configuration (dozens of online machines). Ask if you can only modify the domain name without restarting the machine.


The steps are as follows:

1. Point the domain name to the new machine;

2. Turn off old instances.


Data consistency and data migration are not discussed here.



2. Problem Analysis

Here are two things to confirm:

1. Will BoneCP reconnect if it fails?

That is, whether BoneCP s can catch connection failures/unexpected execution errors and then close open connections in the connection pool.


2. Does DNS have a cache?

Because the program is equipped with a domain name, Mysql is actually connected through a four-tier TCP protocol with a DNS resolution in between, but DNS is usually cached.


For the first question, officially supported, but how to support it. Look at the code, the key code is the PreparedStatementHandle::execute method:

/**   * {@inheritDoc}   *    * @see java.sql.PreparedStatement#execute()   */  // @Override  public boolean execute() throws SQLException {    checkClosed();    try {      if (this.logStatementsEnabled){        logger.debug(PoolUtil.fillLogParams(this.sql, this.logParams));      }      long queryStartTime = queryTimerStart();
      if (this.connectionHook != null){        this.connectionHook.onBeforeStatementExecute(this.connectionHandle, this, this.sql, this.logParams);      }
      boolean result = this.internalPreparedStatement.execute();
      if (this.connectionHook != null){        this.connectionHook.onAfterStatementExecute(this.connectionHandle, this, this.sql, this.logParams);      }

      queryTimerEnd(this.sql, queryStartTime);
      return result;    } catch (SQLException e) {      throw this.connectionHandle.markPossiblyBroken(e);


Focus on handling exceptions, the call to markPossiblyBroken

**    * Given an exception, flag the connection (or database) as being potentially broken. If the exception is a data-specific exception,   * do nothing except throw it back to the application.    *    * @param e SQLException e   * @return SQLException for further processing   */  protected SQLException markPossiblyBroken(SQLException e) {      String state = e.getSQLState();      boolean alreadyDestroyed = false;
    ConnectionState connectionState = this.getConnectionHook() != null ? this.getConnectionHook().onMarkPossiblyBroken(this, state, e) : ConnectionState.NOP;     if (state == null){ // safety;      state = "08999";     }
    if (((sqlStateDBFailureCodes.contains(state) || connectionState.equals(ConnectionState.TERMINATE_ALL_CONNECTIONS)) && this.pool != null) && this.pool.getDbIsDown().compareAndSet(false, true) ){      logger.error("Database access problem. Killing off this connection and all remaining connections in the connection pool. SQL State = " + state);      this.pool.connectionStrategy.terminateAllConnections();      this.pool.destroyConnection(this);      this.logicallyClosed.set(true);      alreadyDestroyed = true;
      for (int i=0; i < this.pool.partitionCount; i++) {        // send a signal to try re-populating again.        this.pool.partitions[i].getPoolWatchThreadSignalQueue().offer(new Object()); // item being pushed is not important.      }    }
   //If an error code is specified, connection timeouts typically occur, and so on, connections in the connection pool are closed    if (state.equals("08003") || sqlStateDBFailureCodes.contains(state) || e.getCause() instanceof SocketException) {        if (!alreadyDestroyed) {      this.pool.destroyConnection(this);      this.logicallyClosed.set(true);      getOriginatingPartition().getPoolWatchThreadSignalQueue().offer(new Object()); // item being pushed is not important.        }    }         char firstChar = state.charAt(0);    if (connectionState.equals(ConnectionState.CONNECTION_POSSIBLY_BROKEN) || state.equals("40001") ||         state.startsWith("08") ||  (firstChar >= '5' && firstChar <='9') /*|| (firstChar >='I' && firstChar <= 'Z')*/){      this.possiblyBroken = true;    }
    // Notify anyone who's interested    if (this.possiblyBroken  && (this.getConnectionHook() != null)){      this.possiblyBroken = this.getConnectionHook().onConnectionException(this, state, e);    }
    return e;  }


You can see that the code detects errors such as connection timeout, closes the connection pool, and re-establishes the new connection next time.


The second is about DNS parsing. By analyzing the code, BoneCP's connection multiplexed Jdbc code, the connection was established by the StandardSocketFactory class in the Jdbc package:

/**   * @see com.mysql.jdbc.SocketFactory#createSocket(Properties)   */  public Socket connect(String hostname, int portNumber, Properties props)      throws SocketException, IOException {
      if ( != null) {        if (!(wantsLocalBind || wantsTimeout || needsConfigurationBeforeConnect)) {          //Find IP based on domain name          InetAddress[] possibleAddresses = InetAddress              .getAllByName(;
          Throwable caughtWhileConnecting = null;
          // Need to loop through all possible addresses, in case          // someone has IPV6 configured (SuSE, for example...)
          for (int i = 0; i < possibleAddresses.length; i++) {            try {              this.rawSocket = new Socket(possibleAddresses[i],                  port);
              configureSocket(this.rawSocket, props);
              break;            } catch (Exception ex) {              caughtWhileConnecting = ex;            }          }
          if (rawSocket == null) {            unwrapExceptionToProperClassAndThrowIt(caughtWhileConnecting);          }        } else {                  }
        return this.rawSocket;      }    }
    throw new SocketException("Unable to create socket");  }


You can see that InetAddres.getAllByName is ultimately called to get the IP address based on the domain name.


getAllByName calls getAddressesFromNameService to get IP:>

 private static InetAddress[] getAddressesFromNameService(String host, InetAddress reqAddr)        throws UnknownHostException    {        InetAddress[] addresses = null;        boolean success = false;        UnknownHostException ex = null;
              if ((addresses = checkLookupTable(host)) == null) {            try {                // This is the first thread which looks up the addresses                // this host or the cache entry for this host has been                // expired so this thread should do the lookup.                for (NameService nameService : nameServices) {                    try {                        /*                         * Do not put the call to lookup() inside the                         * constructor.  if you do you will still be                         * allocating space when the lookup fails.                         */
                        addresses = nameService.lookupAllHostAddr(host);                        success = true;                        break;                    } catch (UnknownHostException uhe) {                        if (host.equalsIgnoreCase("localhost")) {                            InetAddress[] local = new InetAddress[] { impl.loopbackAddress() };                            addresses = local;                            success = true;                            break;                        }                        else {                            addresses = unknown_array;                            success = false;                            ex = uhe;                        }                    }                }
                // More to do?                if (reqAddr != null && addresses.length > 1 && !addresses[0].equals(reqAddr)) {                    // Find it?                    int i = 1;                    for (; i < addresses.length; i++) {                        if (addresses[i].equals(reqAddr)) {                            break;                        }                    }                    // Rotate                    if (i < addresses.length) {                        InetAddress tmp, tmp2 = reqAddr;                        for (int j = 0; j < i; j++) {                            tmp = addresses[j];                            addresses[j] = tmp2;                            tmp2 = tmp;                        }                        addresses[i] = tmp2;                    }                }                //Key code, caching query results                cacheAddresses(host, addresses, success);
                if (!success && ex != null)                    throw ex;
            } finally {                // Delete host from the lookupTable and notify                // all threads waiting on the lookupTable monitor.                updateLookupTable(host);            }        }
        return addresses;    }


It calls cacheAddresses to save the address in the cache. How long does the bottom buffer last?

The code is not pasted anymore; it gets the cache time by calling InetAddressCachePolicy.

 static {        Integer var0 = (Integer)AccessController.doPrivileged(new PrivilegedAction<Integer>() {            public Integer run() {                String var1;                try {                    var1 = Security.getProperty("networkaddress.cache.ttl");                    if(var1 != null) {                        return Integer.valueOf(var1);                    }                } catch (NumberFormatException var3) {                    ;                }
                try {                    var1 = System.getProperty("");                    if(var1 != null) {                        return Integer.decode(var1);                    }                } catch (NumberFormatException var2) {                    ;                }
                return null;            }        });        if(var0 != null) {            cachePolicy = var0.intValue();            if(cachePolicy < 0) {                cachePolicy = -1;            }
            propertySet = true;        } else if(System.getSecurityManager() == null) {            cachePolicy = 30;        }


You can see that the InetAddressCachePolicy is read from several configurations, and if it cannot be read and there is no SecurityManager configured, the default is 30 seconds, which is the case in this example.



4. Actual Verification

The above is our analysis process, how can we verify it?


1. Run the program;

2. Point the domain name to the new machine;

3. Show the connections on the old mysql machine with show processlist, and kill them with kill;

4. See if there is any connection on the new mysql machine and if there is any error in the program.


Unsurprisingly, there will be a minor error in the program and it will return to normal, with all mysql connections pointing to the new machine.



Removing redis mechanism from an online failure

PHP Memory Pool Analysis

Improper use of cache

Advertising Alliance Design Trample


Published an original article. Praise 0. Visits 142


Added by navtheace on Sun, 15 Mar 2020 04:17:09 +0200