Project background

A task system written in Java language uses a thread pool with fixed thread size to process long-time tasks. The specific business logic of the task is to periodically call an external http interface. The CloseableHttpClient is used to call the external interface

Problem phenomenon

One day, the external http interface called by the task is switched to the new domain name. It is found that there are occasional task threads in the thread pool for several consecutive days. The business log is not printed (as if the task thread exited abnormally), but the corresponding task status is not terminated when querying the database. The execution code of the task submitted by the thread pool is as follows:

Future future = executorService.submit(() -> {
    try {
        //Perform tasks
    } catch (InterruptedException e) {
        //Update task status 1
    } catch (Exception e) {
        //Update task status 2
    } finally {
        //Submit database
    }
});

Cause analysis

1. If there is an exception in the task execution code, you should go to finally update the database task record status. The database task record is not updated, indicating that the thread task is still executing (make sure that the submitted database code in finally has no exceptions).

2. Since the thread is still executing and the corresponding task does not output the business log, you need to think about whether the thread is blocked.

3. Query jvm thread information through jstack -l ${pid} and find no thread sharing object interlock. Locate the thread corresponding to an abnormal task according to the task code. It is found that the status is RUNNABLE, but the stack call is displayed on this method

"pool-2-thread-209" #476 prio=5 os_prio=0 cpu=5035.61ms elapsed=356247.91s tid=0x00007fcb90031000 nid=0x129c runnable  [0x00007fcb199de000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(java.base@11.0.2/Native Method)
	at java.net.SocketInputStream.socketRead(java.base@11.0.2/SocketInputStream.java:115)
	at java.net.SocketInputStream.read(java.base@11.0.2/SocketInputStream.java:168)
	at java.net.SocketInputStream.read(java.base@11.0.2/SocketInputStream.java:140)
	at sun.security.ssl.SSLSocketInputRecord.read(java.base@11.0.2/SSLSocketInputRecord.java:448)
	at sun.security.ssl.SSLSocketInputRecord.decodeInputRecord(java.base@11.0.2/SSLSocketInputRecord.java:237)
	at sun.security.ssl.SSLSocketInputRecord.decode(java.base@11.0.2/SSLSocketInputRecord.java:190)
	at sun.security.ssl.SSLTransport.decode(java.base@11.0.2/SSLTransport.java:108)
	at sun.security.ssl.SSLSocketImpl.decode(java.base@11.0.2/SSLSocketImpl.java:1152)
	at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(java.base@11.0.2/SSLSocketImpl.java:1063)
	at sun.security.ssl.SSLSocketImpl.startHandshake(java.base@11.0.2/SSLSocketImpl.java:402)
	- locked <0x000000070a6b62d8> (a sun.security.ssl.TransportContext)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)

Guess this method is at Java net. SocketInputStream. socketRead0(java. base@11.0.2 /Native method) consistent blocking?

4. Query stackoverflow and find that some netizens mentioned the version of openjdk 8. There is a probability that socketRead0 has been blocked, but the subsequent versions have fixed this problem. The current online environment uses openjdk 11, so this possibility is excluded.

5. Then why is socketRead0 blocked all the time? Is there no timeout set? After reading the JDK document, I found that if the socket timeout is not set (the default is 0), socketRead0 will block until the data is read.

6. The current project calls the third-party http interface, and the CloseableHttpClient found that the timeout time is not set

setSocketTimeout and setConnectTimeout, which will be used when the underlying socket connects and reads the contents of the socket buffer. The default is 0, which means that it is blocked until the contents are read.

Deal with it like this first, go online urgently and see the follow-up effect