1. Preface:
Our monitoring system is zabbix. We have just finished monitoring jvm and tomcat recently. We need to spit out the following:
The official jvm and tomcat monitoring templates provided by zabbix have pits in the writing format of the item, which prevents many content from being monitored.
zabbix monitoring java articles written by netizens, 99% only build environment, no zabbix item key value, so on environmental monitoring, item key value does not know how to write.
This article monitors the jvm as follows:
- Memory usage status: Heap memory and non-Heap memory, including used, maximum, committed;
- Heap memory pool: memory usage status of the new generation (eden space), survivor space, and old gen;
Non-heap memory pool: Code cache, meta space, compressed class space;
Class Loading: Total number of loads, loaded, unloaded.
Java threads: always open threads, active threads, peak threads.
This article monitors tomcat as follows:
- Tomcat requests: Include requests per second, number of errors per second;
- Tomcat network traffic statistics: Include incoming and outgoing traffic statistics;
- Tomcat threads: Includes the maximum number of threads, the current number of threads, the current number of busy threads.
Instead of adding garbage collection (gc) monitoring to the zabbix, I added GC day input to the catalina configuration for development analysis, set up as follows.
CATALINA_OPTS="-XX:ParallelGCThreads=4-XX:+PrintGCDetails-Xloggc:Log Storage Path"
The monitoring effect is as follows:
2. Monitoring environment setup
Introduction to the environment:
[root@tomcat-01 ~]# /usr/local/tomcat/bin/version.sh Server version: Apache Tomcat/8.0.23 Server built: May 19 2015 14:58:38 UTC Server number: 8.0.23.0 OS Name: Linux OS Version: 2.6.32-573.22.1.el6.x86_64 Architecture: amd64 JVM Version: 1.8.0_65-b17 JVM Vendor: Oracle Corporation
2.1.Server side configuration
1. Install the java jdk environment on the Zabbix server side and turn on java Gateway to support java monitoring.
Java Gateway is enabled as follows:
Zabbix is installed via the rpm package: just install the zabbix-java-gateway package.
Zabbix is installed by compiling: you need to add -enable-java at compile time to support jmx monitoring, if you have not previously recompiled it.
2. Modify the zabbix-java-gateway configuration file
[root@zabbix ~]# vim /etc/zabbix/zabbix_java_gateway.conf LISTEN_IP="192.168.10.3" LISTEN_PORT=10052 PID_FILE="/var/run/zabbix/zabbix_java.pid" START_POLLERS=5
Modify zabbix-server configuration file
[root@zabbix ~]# vim /etc/zabbix/zabbix_server.conf JavaGateway=192.168.10.3 JavaGatewayPort=10052 StartJavaPollers=5
Start the zabbix-java-gateway service
[root@zabbix ~]# /etc/init.d/zabbix-java-gateway start
2.2.Tomcat server configuration
1. Download the catalina-jmx-remote.jar package to the lib directory under the tomcat installation directory
wget -O /usr/local/tomcat/lib/catalina-jmx-remote.jar http://archive.apache.org/dist/tomcat/tomcat-8/v8.0.23/bin/extras/catalina-jmx-remote.jar
2. Modify catalina.sh to add the following
CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8090 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"
3. Restart tomcat and monitor test
(1) Download cmdline-jmxclient-0.10.3.jar file, download address http://crawler.archive.org/cm...
(2) Execute the following command locally to view tomcat's heap memory information
[root@tomcat-01 ~]# java -jar /root/cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 java.lang:type=Memory HeapMemoryUsage 11/04/2016 15:23:16 +0800 org.archive.jmx.Client HeapMemoryUsage: committed: 2146959360 init: 2147483648 max: 2146959360 used: 407611808
3. Monitoring data collection
3.1. Heap memory
Tomcat views heap memory information locally:
[root@tomcat-01 ~]# java -jar cmdline-jmxclient-0.10.3.jar controlRole:tomcat 127.0.0.1:8090 java.lang:type=Memory HeapMemoryUsage 11/04/2016 15:36:58 +0800 org.archive.jmx.Client HeapMemoryUsage: committed: 2145910784 init: 2147483648 max: 2145910784 used: 741540536
zabbix Monitor Heap Memory Key Value:
Maximum heap memory: jmx["java.lang:type=Memory","HeapMemoryUsage.max"] Used heap memory: jmx["java.lang:type=Memory","HeapMemoryUsage.used"] Heap memory committed: jmx["java.lang:type=Memory","HeapMemoryUsage.committed"]
A complete zabbix item is filled out as follows, with different keys:
3.2. Memory pool eden space:
Tomcat local view eden space:
java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 java.lang:type=MemoryPool,name=PS\ Eden\ Space Usage
zabbix monitors eden region key values:
Maximum space: JMX ["java.lang:type=MemoryPool, name=PS Eden Space, Usage.max] Used space: JMX ["java.lang:type=MemoryPool, name=PS Eden Space, Usage.used] Commit Space: JMX ["java.lang:type=MemoryPool, name=PS Eden Space, Usage.committed]
3.3. Memory pool survivor space:
Tomcat views the Survivor space area locally:
java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 java.lang:type=MemoryPool,name=PS\ Survivor\ Space Usage
zabbix monitors Survivor key values:
jmx["java.lang:type=MemoryPool,name=PS Survivor Space",Usage.committed] jmx["java.lang:type=MemoryPool,name=PS Survivor Space",Usage.max] jmx["java.lang:type=MemoryPool,name=PS Survivor Space",Usage.used]
3.4. Memory pool old gen:
Tomcat locally views the old gen region using:
java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 java.lang:name=PS\ Old\ Gen,type=MemoryPool Usage
zabbix monitors old gen key values:
jmx["java.lang:type=MemoryPool,name=PS Old Gen",Usage.committed] jmx["java.lang:type=MemoryPool,name=PS Old Gen",Usage.max] jmx["java.lang:type=MemoryPool,name=PS Old Gen",Usage.used]
3.5. Non-heap memory:
Tomcat local view of non-heap memory usage:
java -jar cmdline-jmxclient-0.10.3.jar controlRole:tomcat 127.0.0.1:8090 java.lang:type=Memory NonHeapMemoryUsag1
zabbix monitors non-heap memory usage
1jmx["java.lang:type=Memory","NonHeapMemoryUsag.committed"] jmx["java.lang:type=Memory","NonHeapMemoryUsag.used"]
3.6. Memory pool meta space:
Tomcat local view meta space area uses:
java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 java.lang:type=MemoryPool,name=Metaspace Usage
zabbix monitors mete space region key values
jmx["java.lang:type=MemoryPool,name=Metaspace",Usage.committed] jmx["java.lang:type=MemoryPool,name=Metaspace",Usage.used]
3.7. Memory pool code cache:
Tomcat local view code cache area uses:
java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 java.lang:type=MemoryPool,name=Code\ Cache Usage
zabbix monitors code cache region usage:
jmx["java.lang:type=MemoryPool,name=Code Cache",Usage.committed] jmx["java.lang:type=MemoryPool,name=Code Cache",Usage.max] jmx["java.lang:type=MemoryPool,name=Code Cache",Usage.used]
3.8. Memory pool compressed class space:
Tomcat views compressed class space locally using:
java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 java.lang:type=MemoryPool,name=Compressed\ Class\ Space Usage zabbix Monitor compressed class space Areas use key values: jmx["java.lang:type=MemoryPool,name=Compressed Class Space",Usage.committed] jmx["java.lang:type=MemoryPool,name=Compressed Class Space",Usage.max] jmx["java.lang:type=MemoryPool,name=Compressed Class Space",Usage.used]
3.9. Class loading:
Tomcat local view class load information:
Total number of loads: java-jar cmdline-jmxclient-0.10.3.jar control lRole: Tomcat 127.0.0.1:8090 java.lang:type=ClassLoading TotalLoadedClassCoun Loaded: java-jar cmdline-jmxclient-0.10.3.jar control lRole: Tomcat 127.0.0.1:8090 java.lang:type=ClassLoading LoadedClassCount Uninstalled: java-jar cmdline-jmxclient-0.10.3.jar control lRole: Tomcat 127.0.0.1:8090 java.lang:type=ClassLoading Unloaded ClassCount
Zabbix monitoring class load key values:
Total number of loads: jmx["java.lang:type=ClassLoading","TotalLoadedClassCount"] Loaded: jmx["java.lang:type=ClassLoading","LoadedClassCount"] Uninstalled: jmx["java.lang:type=ClassLoading","UnloadedClassCount"]
3.10.java threads:
tomcat local view java threads:
Total open threads: java-jar cmdline-jmxclient-0.10.3.jar control lRole: Tomcat 127.0.0.1:8090 java.lang:type=Threading TotalStartedThreadCount Active Threads: java-jar cmdline-jmxclient-0.10.3.jar control lRole: Tomcat 127.0.0.1:8090 java.lang:type=Threading PeakThreadCount Peak Thread: java-jar cmdline-jmxclient-0.10.3.jar control lRole: Tomcat 127.0.0.1:8090 java.lang:type=Threading PeakThreadCount
Zabbix monitors java thread key values:
Always open threads: jmx["java.lang:type=Threading","TotalStartedThreadCount"] Active Threads: jmx["java.lang:type=Threading","ThreadCount"] Thread peak: jmx["java.lang:type=Threading","PeakThreadCount"]
3.11.tomcat thread:
View tomcat thread information locally:
Maximum threads: java-jar cmdline-jmxclient-0.10.3.jar-127.0.0.1:8090 Catalina:name="http-nio-8080\", type=ThreadPool maxThreads Current thread: java-jar cmdline-jmxclient-0.10.3.jar-127.0.0.1:8090 Catalina:name="http-nio-8080\", type=ThreadPool current ThreadCount Busy Threads: java-jar cmdline-jmxclient-0.10.3.jar-127.0.0.1:8090 Catalina:name="http-nio-8080\", type=ThreadPool current ThreadsBusy
Zabbix monitors tomcat thread key values:
Maximum threads: JMX ["Catalina:type=ThreadPool, name=" http-nio-8080\", maxThreads] Current thread: JMX ["Catalina:type=ThreadPool, name=" http-nio-8080\", currentThreadCount] Busy thread JMX ["Catalina:type=ThreadPool, name=" http-nio-8080\", currentThreadsBusy]
3.12. Network traffic:
Tomcat sees the bytes received locally:
Bytes received: java-jar cmdline-jmxclient-0.10.3.jar-127.0.0.1:8090 Catalina:name="http-nio-8080\", type=GlobalRequestProcessor bytesReceived Bytes sent: java-jar cmdline-jmxclient-0.10.3.jar-127.0.0.1:8090 Catalina:name="http-nio-8080\", type=GlobalRequestProcessor bytesSent
Zabbix Monitor tomcat accepts byte key values:
Bytes received: JMX ["Catalina:type=GlobalRequestProcessor, name=" http-nio-8080\", bytesReceived] Bytes sent: JMX ["Catalina:type=GlobalRequestProcessor, name=" http-nio-8080\", bytesSent]
3.13.tomcat request, error request:
Number of Tomcat requests viewed locally by tomcat: Number of tomcat requests: java-jar cmdline-jmxclient-0.10.3.jar-192.168.10.46:8090 Catalina:name="http-nio-8080\", type=GlobalRequestProcessor requestCount tomcat error request: java-jar cmdline-jmxclient-0.10.3.jar-192.168.10.46:8090 Catalina:name="http-nio-8080\", type=GlobalRequestProcessor errorCount zabbix monitors the number of tomcat requests: Number of tomcat requests: JMX ["Catalina:type=GlobalRequestProcessor, name=" http-nio-8080\", requestCount] tomcat error request: JMX ["Catalina:type=GlobalRequestProcessor, name=" http-nio-8080\", errorCount]
4. Answers to questions
Recently, a netizen contacted me and said that after watching my blog build up the environment, I could not get the data. I helped to sort out the problem and found out what it was, so I record it.
Netizens reported the following errors:
# java -jar /root/cmdline-jmxclient-0.10.3.jar - 127.0.0.1:9080 java.lang:type=MemoryPool,name=PS\ Eden\ Space Usag 11/11/2016 10:03:37 +0800 org.archive.jmx.Client java.lang:name=PS Eden Space,type=MemoryPool is not a registered bean
4.1. Solution ideas
- If you can't get data using command line monitoring, use jconsole first to see if there is data.
- If jconsole has data, look down and check your environment instead.
- If jconsole has data, but the command line has no data, then there is a problem with Object Name or attributes of Mbeans. Netizens copy me, but their local environment and my environment are different so that they can not get data. There are two methods to query their local Mbeans, one is graphics and the other is command line, the other is netizenIf there's something wrong with your Eden Space, I'll post a way to view the Object Name and attributes of the local memory pool.
(1) View through jconsole:
(2) View from the command line: directly use the java-jar cmdline-jmxclient-0.10.3.jar-127.0.0.1:8090 command to get all the Mbean information, output too much here will not paste the output results.I get monitoring information for all memory pools using the grep command as follows.
[root@tomcat-01 ~]# java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 | grep MemoryPool java.lang:name=Compressed Class Space,type=MemoryPool java.lang:name=Metaspace,type=MemoryPool java.lang:name=PS Old Gen,type=MemoryPool java.lang:name=PS Eden Space,type=MemoryPool java.lang:name=PS Survivor Space,type=MemoryPool java.lang:name=Code Cache,type=MemoryPool
Then I want to get all the attribute information for Eden Space as follows: (Some special characters need to be escaped)
[root@tomcat-01 ~]# java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 java.lang:name=PS\ Eden\ Space,type=MemoryPool Attributes: Usage: Usage (type=javax.management.openmbean.CompositeData) PeakUsage: PeakUsage (type=javax.management.openmbean.CompositeData) MemoryManagerNames: MemoryManagerNames (type=[Ljava.lang.String;) UsageThreshold: UsageThreshold (type=long) UsageThresholdExceeded: UsageThresholdExceeded (type=boolean) UsageThresholdCount: UsageThresholdCount (type=long) UsageThresholdSupported: UsageThresholdSupported (type=boolean) CollectionUsageThreshold: CollectionUsageThreshold (type=long) CollectionUsageThresholdExceeded: CollectionUsageThresholdExceeded (type=boolean) CollectionUsageThresholdCount: CollectionUsageThresholdCount (type=long) CollectionUsage: CollectionUsage (type=javax.management.openmbean.CompositeData) CollectionUsageThresholdSupported: CollectionUsageThresholdSupported (type=boolean) Valid: Valid (type=boolean) Name: Name (type=java.lang.String) Type: Type (type=java.lang.String) ObjectName: ObjectName (type=javax.management.ObjectName) Operations: resetPeakUsage: resetPeakUsage Parameters 0, return type=void
So we're looking at the usage of Eden Space.You'll see the data coming out
[root@tomcat-01 ~]# java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 java.lang:name=PS\ Eden\ Space,type=MemoryPool Usage 12/03/2016 08:33:58 +0800 org.archive.jmx.Client Usage: committed: 712507392 init: 537395200 max: 712507392 used: 396006304
