Phenomenon: the database service and monitoring run normally, but the function module is very slow to open;
Process:
I. check the database alert? Oaxx.log log
Log switching is not high, no error is reported; database load is not large;
II. Check database status
The service and monitoring are running normally, sqlplus / as sysdba login is normal, and the number of sessions is normal
Check system start time
Initial installation date: 2015 / 5 / 13, 12:09:51
System start time: June 14, 2018, 3:11:06
Check the network connection and find a large number of close_wait caused by process 327000, which is judged as 360tray
TCP 10.25.136.5:52667 221.130.200.53:80 CLOSE_WAIT 327000 TCP 10.25.136.5:52668 221.130.200.53:80 CLOSE_WAIT 327000 TCP 10.25.136.5:52669 221.130.200.53:80 CLOSE_WAIT 327000 TCP 10.25.136.5:52672 221.130.200.53:80 CLOSE_WAIT 327000 TCP 10.25.136.5:52674 221.130.200.53:80 CLOSE_WAIT 327000 TCP 10.25.136.5:52675 221.130.200.53:80 CLOSE_WAIT 327000 TCP 10.25.136.5:52676 221.130.200.53:80 CLOSE_WAIT 327000 TCP 10.25.136.5:52677 221.130.200.53:80 CLOSE_WAIT 327000 TCP 10.25.136.5:52678 221.130.200.53:80 CLOSE_WAIT 327000 TCP 10.25.136.5:52680 221.130.200.53:80 CLOSE_WAIT 327000 TCP 10.25.136.5:52681 221.130.200.53:80 CLOSE_WAIT 327000 TCP 10.25.136.5:52682 221.130.200.53:80 CLOSE_WAIT 327000 TCP 10.25.136.5:52683 221.130.200.53:80 CLOSE_WAIT 327000 TCP 10.25.136.5:52684 221.130.200.53:80 CLOSE_WAIT 327000 TCP 10.25.136.5:52686 221.130.200.53:80 CLOSE_WAIT 327000 TCP 10.25.136.5:52688 221.130.200.53:80 CLOSE_WAIT 327000 TCP 10.25.136.5:52690 221.130.200.53:80 CLOSE_WAIT 327000 TCP 10.25.136.5:52691 221.130.200.53:80 CLOSE_WAIT 327000 TCP 10.25.136.5:52692 221.130.200.53:80 CLOSE_WAIT 327000
III. check database waiting events
No waiting;
4. Tnspin findings:
C:\Users\Administrator>tnsping oadb TNS Ping Utility for 64-bit Windows: Version 11.2.0.1.0 - Production on 08-5 month - 2019 15:05:04 Copyright (c) 1997, 2010, Oracle. All rights reserved. //Used parameter file: Y:\oracle\product\11.2.0\dbhome_1\network\admin\sqlnet.ora //TNSNAMES adapter used to resolve alias //Try to connect (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST =10.xx.1xx.5)(PORT = 15 22)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = oadb))) OK (51910 Millisecond)
High latency, 360tray disabled
C:\Users\Administrator>tnsping oadb TNS Ping Utility for 64-bit Windows: Version 11.2.0.1.0 - Production on 08-5 month - 2019 14:58:23 Copyright (c) 1997, 2010, Oracle. All rights reserved. //Used parameter file: Y:\oracle\product\11.2.0\dbhome_1\network\admin\sqlnet.ora //TNSNAMES adapter used to resolve alias //Try to connect (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = WIN-5K0CGNM3EQV)(PORT = 1522)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = oadb))) OK (24950 Millisecond)
It's down to 25 seconds, but it's still very high. ping the host shows that the delay is very low. If ipv6 is disabled, the effect is the same;
C:\Users\Administrator>ping 10.25.136.5
Ping 10.25.136.5 with 32 bytes of data:
Reply from 10.25.136.5: byte = 32 time < 1ms TTL = 64
Reply from 10.25.136.5: byte = 32 time < 1ms TTL = 64
It is also found that the application log has a network connection timeout
url=jdbc:oracle:thin:@10.2x.1x6.x:xxx:oadb;user=bxxx;pwd=xxx java.sql.SQLException: Io abnormal: Connection refused(DESCRIPTION=(TMP=)(VSNNUM=186 646784)(ERR=12505)(ERROR_STACK=(ERROR=(CODE=12505)(EMFI=4)))) at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:189) at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:231)
Check that sqlnet.ora is normal;
V. check the database monitoring, restart the monitoring, no improvement. Check the monitoring log and find that the log is not updated after 2:00 on April 6, 2019
There are a lot of trace:
** DBGRL Error: Text Alert Log ** DBGRL Error: SLERC_OERC, 48184 ** DBGRL Error: OSD-00002: Additional error messages O/S-Error: (OS 87) Parameter error. ** DBGRL Error: 06-4 month -2019 02:01:39 * (CONNECT_DATA=(SID=xxdb)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))) * (ADDRESS=(PROTOCOL=tcp)(HOST=1xxxxx)(PORT=60839)) * establish * oadb * 0 Trace file y:\oracle\diag\tnslsnr\WIN-5K0CGNM3EQV\listener\trace\ora_895324_893720.trc ** DBGRL Error: Text Alert Log ** DBGRL Error: SLERC_OERC, 48184 ** DBGRL Error: OSD-00002: Additional error messages O/S-Error: (OS 87) Parameter error. ** DBGRL Error: The system parameter file is Y:\oracle\product\11.2.0\dbhome_1\network\admin\listener.ora
Check listner.log and find that it has 4GB. There are the following problems under win:
Windows: listener hang & lsnrctl commands are slow or hang (document ID 1319797.1)
Bug 9879101 : THE CONNECT THROUGH LISTENER WAS SLOW WHEN LISTNER LOG GROWED 4GB
Treatment method:
1 stop listening, rename the original listening file,
y:\oracle\diag\tnslsnr\WIN-5K0CGNM3EQV\listener\trace\ listener.bak.log
Create a new listener.log