Phenomenon:
On the web side of ZABBIX, JMX displays red and prompts cannot reslove 192.168.100.107192.168.100.108...
At that time, the first thought was to keep the ip after JavaGateway in / etc / ZABBIX / ZABBIX [server. Conf on the server side only 192.168.100.107, and then restart ZABBIX server.
But JMX still displays red, and there is a problem of ", (connection referred): Service: JMX: RMI: / / / JNDI / RMI: / / 192.168.100.107: 9999 / jmxrmi"
resolvent:
Go down to 192.168.100.107
[root@web1 ~]# /usr/local/tomcat/bin/shutdown.sh
Using CATALINA_BASE: /usr/local/tomcat
Using CATALINA_HOME: /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME: /usr/local/jdk1.8/jre
Using CLASSPATH: /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
Jul 07, 2019 10:48:46 PM org.apache.catalina.startup.Catalina stopServer
SEVERE: Could not contact [localhost:8005] (base port [8005] and offset [0]). Tomcat may not be running.
Jul 07, 2019 10:48:46 PM org.apache.catalina.startup.Catalina stopServer
SEVERE: Error stopping Catalina
java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:513)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:390)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:480)
[root@web1 ~]# !net
netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:9999 0.0.0.0:* LISTEN 9629/java
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1/systemd
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 6387/nginx: master
tcp 0 0 0.0.0.0:20048 0.0.0.0:* LISTEN 15753/rpc.mountd
tcp 0 0 0.0.0.0:37457 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:49366 0.0.0.0:* LISTEN 14929/rpc.statd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 2993/sshd
tcp 0 0 0.0.0.0:44247 0.0.0.0:* LISTEN 9629/java
tcp 0 0 0.0.0.0:34332 0.0.0.0:* LISTEN 9629/java
tcp 0 0 0.0.0.0:2049 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:10050 0.0.0.0:* LISTEN 10386/zabbix_agentd
tcp 0 0 0.0.0.0:10052 0.0.0.0:* LISTEN 10320/java
tcp6 0 0 :::43533 :::* LISTEN -
tcp6 0 0 :::111 :::* LISTEN 15627/rpcbind
tcp6 0 0 :::20048 :::* LISTEN 15753/rpc.mountd
tcp6 0 0 :::34298 :::* LISTEN 14929/rpc.statd
tcp6 0 0 :::2049 :::* LISTEN -
tcp6 0 0 :::10050 :::* LISTEN 10386/zabbix_agentd I find that tomcat can't be turned off at all, so only kill will be turned off.
[root@web1 ~]# kill 9629 [root@web1 ~]# netstat -lntp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1/systemd tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 6387/nginx: master tcp 0 0 0.0.0.0:20048 0.0.0.0:* LISTEN 15753/rpc.mountd tcp 0 0 0.0.0.0:37457 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:49366 0.0.0.0:* LISTEN 14929/rpc.statd tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 2993/sshd tcp 0 0 0.0.0.0:2049 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:10050 0.0.0.0:* LISTEN 10386/zabbix_agentd tcp 0 0 0.0.0.0:10052 0.0.0.0:* LISTEN 10320/java tcp6 0 0 :::43533 :::* LISTEN - tcp6 0 0 :::111 :::* LISTEN 15627/rpcbind tcp6 0 0 :::20048 :::* LISTEN 15753/rpc.mountd tcp6 0 0 :::34298 :::* LISTEN 14929/rpc.statd tcp6 0 0 :::2049 :::* LISTEN - tcp6 0 0 :::10050 :::* LISTEN 10386/zabbix_agentd
Restart tomcat again. It is better to restart ZABBIX Java gateway and ZABBIX agent at the same time.
[root@web1 ~]# /usr/local/tomcat/bin/startup.sh Using CATALINA_BASE: /usr/local/tomcat Using CATALINA_HOME: /usr/local/tomcat Using CATALINA_TMPDIR: /usr/local/tomcat/temp Using JRE_HOME: /usr/local/jdk1.8/jre Using CLASSPATH: /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar Tomcat started. [root@web1 ~]# netstat -lntp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:41125 0.0.0.0:* LISTEN 10506/java tcp 0 0 0.0.0.0:8009 0.0.0.0:* LISTEN 10506/java tcp 0 0 0.0.0.0:9999 0.0.0.0:* LISTEN 10506/java tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1/systemd tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 10506/java tcp 0 0 0.0.0.0:40496 0.0.0.0:* LISTEN 10506/java tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 6387/nginx: master tcp 0 0 0.0.0.0:20048 0.0.0.0:* LISTEN 15753/rpc.mountd tcp 0 0 0.0.0.0:37457 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:49366 0.0.0.0:* LISTEN 14929/rpc.statd tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 2993/sshd tcp 0 0 0.0.0.0:2049 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:10050 0.0.0.0:* LISTEN 10386/zabbix_agentd tcp 0 0 0.0.0.0:10052 0.0.0.0:* LISTEN 10320/java tcp6 0 0 :::43533 :::* LISTEN - tcp6 0 0 :::111 :::* LISTEN 15627/rpcbind tcp6 0 0 :::20048 :::* LISTEN 15753/rpc.mountd tcp6 0 0 :::34298 :::* LISTEN 14929/rpc.statd tcp6 0 0 :::2049 :::* LISTEN - tcp6 0 0 :::10050 :::* LISTEN 10386/zabbix_agentd
In the same way, restart other problematic tomcat machines, and it should be OK.
Conclusion:
When ZABBIX fails, check the configuration in server / usr/zabbix/zabbix_server.conf, and then check the tomcat status of each client.