Monitoring tomcat with zabbix (including jvm monitoring)

1. Preface:

Our monitoring system is zabbix. We have just finished monitoring jvm and tomcat recently. We need to spit out the following:
The official jvm and tomcat monitoring templates provided by zabbix have pits in the writing format of the item, which prevents many content from being monitored.
zabbix monitoring java articles written by netizens, 99% only build environment, no zabbix item key value, so on environmental monitoring, item key value does not know how to write.

This article monitors the jvm as follows:

  • Memory usage status: Heap memory and non-Heap memory, including used, maximum, committed;
  • Heap memory pool: memory usage status of the new generation (eden space), survivor space, and old gen;

Non-heap memory pool: Code cache, meta space, compressed class space;
Class Loading: Total number of loads, loaded, unloaded.
Java threads: always open threads, active threads, peak threads.

This article monitors tomcat as follows:

  • Tomcat requests: Include requests per second, number of errors per second;
  • Tomcat network traffic statistics: Include incoming and outgoing traffic statistics;
  • Tomcat threads: Includes the maximum number of threads, the current number of threads, the current number of busy threads.

Instead of adding garbage collection (gc) monitoring to the zabbix, I added GC day input to the catalina configuration for development analysis, set up as follows.
CATALINA_OPTS="-XX:ParallelGCThreads=4-XX:+PrintGCDetails-Xloggc:Log Storage Path"
The monitoring effect is as follows:

2. Monitoring environment setup

Introduction to the environment:

[root@tomcat-01 ~]# /usr/local/tomcat/bin/
Server version: Apache Tomcat/8.0.23
Server built: May 19 2015 14:58:38 UTC
Server number:
OS Name: Linux
OS Version: 2.6.32-573.22.1.el6.x86_64
Architecture: amd64
JVM Version: 1.8.0_65-b17
JVM Vendor: Oracle Corporation

2.1.Server side configuration
1. Install the java jdk environment on the Zabbix server side and turn on java Gateway to support java monitoring.
Java Gateway is enabled as follows:
Zabbix is installed via the rpm package: just install the zabbix-java-gateway package.
Zabbix is installed by compiling: you need to add -enable-java at compile time to support jmx monitoring, if you have not previously recompiled it.
2. Modify the zabbix-java-gateway configuration file

[root@zabbix ~]# vim /etc/zabbix/zabbix_java_gateway.conf

Modify zabbix-server configuration file

[root@zabbix ~]# vim /etc/zabbix/zabbix_server.conf

Start the zabbix-java-gateway service

[root@zabbix ~]# /etc/init.d/zabbix-java-gateway start

2.2.Tomcat server configuration
1. Download the catalina-jmx-remote.jar package to the lib directory under the tomcat installation directory

wget -O /usr/local/tomcat/lib/catalina-jmx-remote.jar

2. Modify to add the following


3. Restart tomcat and monitor test
(1) Download cmdline-jmxclient-0.10.3.jar file, download address
(2) Execute the following command locally to view tomcat's heap memory information

[root@tomcat-01 ~]# java -jar /root/cmdline-jmxclient-0.10.3.jar - java.lang:type=Memory HeapMemoryUsage
11/04/2016 15:23:16 +0800 org.archive.jmx.Client HeapMemoryUsage:
committed: 2146959360
init: 2147483648
max: 2146959360
used: 407611808

3. Monitoring data collection

3.1. Heap memory
Tomcat views heap memory information locally:

[root@tomcat-01 ~]# java -jar cmdline-jmxclient-0.10.3.jar controlRole:tomcat java.lang:type=Memory HeapMemoryUsage
11/04/2016 15:36:58 +0800 org.archive.jmx.Client HeapMemoryUsage:
committed: 2145910784
init: 2147483648
max: 2145910784
used: 741540536

zabbix Monitor Heap Memory Key Value:

Maximum heap memory: jmx["java.lang:type=Memory","HeapMemoryUsage.max"]
Used heap memory: jmx["java.lang:type=Memory","HeapMemoryUsage.used"]
Heap memory committed: jmx["java.lang:type=Memory","HeapMemoryUsage.committed"]

A complete zabbix item is filled out as follows, with different keys:

3.2. Memory pool eden space:
Tomcat local view eden space:

java -jar cmdline-jmxclient-0.10.3.jar - java.lang:type=MemoryPool,name=PS\ Eden\ Space Usage

zabbix monitors eden region key values:

Maximum space: JMX ["java.lang:type=MemoryPool, name=PS Eden Space, Usage.max]
Used space: JMX ["java.lang:type=MemoryPool, name=PS Eden Space, Usage.used]
Commit Space: JMX ["java.lang:type=MemoryPool, name=PS Eden Space, Usage.committed]

3.3. Memory pool survivor space:
Tomcat views the Survivor space area locally:

java -jar cmdline-jmxclient-0.10.3.jar - java.lang:type=MemoryPool,name=PS\ Survivor\ Space Usage

zabbix monitors Survivor key values:

jmx["java.lang:type=MemoryPool,name=PS Survivor Space",Usage.committed]
jmx["java.lang:type=MemoryPool,name=PS Survivor Space",Usage.max]
jmx["java.lang:type=MemoryPool,name=PS Survivor Space",Usage.used]

3.4. Memory pool old gen:
Tomcat locally views the old gen region using:

java -jar cmdline-jmxclient-0.10.3.jar - java.lang:name=PS\ Old\ Gen,type=MemoryPool Usage

zabbix monitors old gen key values:

jmx["java.lang:type=MemoryPool,name=PS Old Gen",Usage.committed]
jmx["java.lang:type=MemoryPool,name=PS Old Gen",Usage.max]
jmx["java.lang:type=MemoryPool,name=PS Old Gen",Usage.used]

3.5. Non-heap memory:
Tomcat local view of non-heap memory usage:

java -jar cmdline-jmxclient-0.10.3.jar controlRole:tomcat java.lang:type=Memory NonHeapMemoryUsag1

zabbix monitors non-heap memory usage


3.6. Memory pool meta space:
Tomcat local view meta space area uses:

java -jar cmdline-jmxclient-0.10.3.jar - java.lang:type=MemoryPool,name=Metaspace Usage

zabbix monitors mete space region key values


3.7. Memory pool code cache:
Tomcat local view code cache area uses:

java -jar cmdline-jmxclient-0.10.3.jar - java.lang:type=MemoryPool,name=Code\ Cache Usage

zabbix monitors code cache region usage:

jmx["java.lang:type=MemoryPool,name=Code Cache",Usage.committed]
jmx["java.lang:type=MemoryPool,name=Code Cache",Usage.max]
jmx["java.lang:type=MemoryPool,name=Code Cache",Usage.used]

3.8. Memory pool compressed class space:
Tomcat views compressed class space locally using:

java -jar cmdline-jmxclient-0.10.3.jar - java.lang:type=MemoryPool,name=Compressed\ Class\ Space Usage
zabbix Monitor compressed class space Areas use key values:
jmx["java.lang:type=MemoryPool,name=Compressed Class Space",Usage.committed]
jmx["java.lang:type=MemoryPool,name=Compressed Class Space",Usage.max]
jmx["java.lang:type=MemoryPool,name=Compressed Class Space",Usage.used]

3.9. Class loading:
Tomcat local view class load information:

Total number of loads: java-jar cmdline-jmxclient-0.10.3.jar control lRole: Tomcat java.lang:type=ClassLoading TotalLoadedClassCoun
 Loaded: java-jar cmdline-jmxclient-0.10.3.jar control lRole: Tomcat java.lang:type=ClassLoading LoadedClassCount
 Uninstalled: java-jar cmdline-jmxclient-0.10.3.jar control lRole: Tomcat java.lang:type=ClassLoading Unloaded ClassCount

Zabbix monitoring class load key values:

Total number of loads: jmx["java.lang:type=ClassLoading","TotalLoadedClassCount"]
Loaded: jmx["java.lang:type=ClassLoading","LoadedClassCount"]
Uninstalled: jmx["java.lang:type=ClassLoading","UnloadedClassCount"] threads:
tomcat local view java threads:

Total open threads: java-jar cmdline-jmxclient-0.10.3.jar control lRole: Tomcat java.lang:type=Threading TotalStartedThreadCount
 Active Threads: java-jar cmdline-jmxclient-0.10.3.jar control lRole: Tomcat java.lang:type=Threading PeakThreadCount
 Peak Thread: java-jar cmdline-jmxclient-0.10.3.jar control lRole: Tomcat java.lang:type=Threading PeakThreadCount

Zabbix monitors java thread key values:

Always open threads: jmx["java.lang:type=Threading","TotalStartedThreadCount"]
Active Threads: jmx["java.lang:type=Threading","ThreadCount"]
Thread peak: jmx["java.lang:type=Threading","PeakThreadCount"]

3.11.tomcat thread:
View tomcat thread information locally:

Maximum threads: java-jar cmdline-jmxclient-0.10.3.jar- Catalina:name="http-nio-8080\", type=ThreadPool maxThreads
 Current thread: java-jar cmdline-jmxclient-0.10.3.jar- Catalina:name="http-nio-8080\", type=ThreadPool current ThreadCount
 Busy Threads: java-jar cmdline-jmxclient-0.10.3.jar- Catalina:name="http-nio-8080\", type=ThreadPool current ThreadsBusy

Zabbix monitors tomcat thread key values:

Maximum threads: JMX ["Catalina:type=ThreadPool, name=" http-nio-8080\", maxThreads]
Current thread: JMX ["Catalina:type=ThreadPool, name=" http-nio-8080\", currentThreadCount]
Busy thread JMX ["Catalina:type=ThreadPool, name=" http-nio-8080\", currentThreadsBusy]

3.12. Network traffic:
Tomcat sees the bytes received locally:

Bytes received: java-jar cmdline-jmxclient-0.10.3.jar- Catalina:name="http-nio-8080\", type=GlobalRequestProcessor bytesReceived
 Bytes sent: java-jar cmdline-jmxclient-0.10.3.jar- Catalina:name="http-nio-8080\", type=GlobalRequestProcessor bytesSent

Zabbix Monitor tomcat accepts byte key values:

Bytes received: JMX ["Catalina:type=GlobalRequestProcessor, name=" http-nio-8080\", bytesReceived]
Bytes sent: JMX ["Catalina:type=GlobalRequestProcessor, name=" http-nio-8080\", bytesSent]

3.13.tomcat request, error request:

Number of Tomcat requests viewed locally by tomcat:
Number of tomcat requests: java-jar cmdline-jmxclient-0.10.3.jar- Catalina:name="http-nio-8080\", type=GlobalRequestProcessor requestCount
 tomcat error request: java-jar cmdline-jmxclient-0.10.3.jar- Catalina:name="http-nio-8080\", type=GlobalRequestProcessor errorCount
 zabbix monitors the number of tomcat requests:
Number of tomcat requests: JMX ["Catalina:type=GlobalRequestProcessor, name=" http-nio-8080\", requestCount]
tomcat error request: JMX ["Catalina:type=GlobalRequestProcessor, name=" http-nio-8080\", errorCount]

4. Answers to questions

Recently, a netizen contacted me and said that after watching my blog build up the environment, I could not get the data. I helped to sort out the problem and found out what it was, so I record it.
Netizens reported the following errors:

# java -jar /root/cmdline-jmxclient-0.10.3.jar - java.lang:type=MemoryPool,name=PS\ Eden\ Space Usag
11/11/2016 10:03:37 +0800
org.archive.jmx.Client java.lang:name=PS Eden Space,type=MemoryPool is not a registered bean

4.1. Solution ideas

  • If you can't get data using command line monitoring, use jconsole first to see if there is data.
  • If jconsole has data, look down and check your environment instead.
  • If jconsole has data, but the command line has no data, then there is a problem with Object Name or attributes of Mbeans. Netizens copy me, but their local environment and my environment are different so that they can not get data. There are two methods to query their local Mbeans, one is graphics and the other is command line, the other is netizenIf there's something wrong with your Eden Space, I'll post a way to view the Object Name and attributes of the local memory pool.

(1) View through jconsole:

(2) View from the command line: directly use the java-jar cmdline-jmxclient-0.10.3.jar- command to get all the Mbean information, output too much here will not paste the output results.I get monitoring information for all memory pools using the grep command as follows.

[root@tomcat-01 ~]# java -jar cmdline-jmxclient-0.10.3.jar - | grep MemoryPool
java.lang:name=Compressed Class Space,type=MemoryPool
java.lang:name=PS Old Gen,type=MemoryPool
java.lang:name=PS Eden Space,type=MemoryPool
java.lang:name=PS Survivor Space,type=MemoryPool
java.lang:name=Code Cache,type=MemoryPool

Then I want to get all the attribute information for Eden Space as follows: (Some special characters need to be escaped)

[root@tomcat-01 ~]# java -jar cmdline-jmxclient-0.10.3.jar - java.lang:name=PS\ Eden\ Space,type=MemoryPool
Usage: Usage (
PeakUsage: PeakUsage (
MemoryManagerNames: MemoryManagerNames (type=[Ljava.lang.String;)
UsageThreshold: UsageThreshold (type=long)
UsageThresholdExceeded: UsageThresholdExceeded (type=boolean)
UsageThresholdCount: UsageThresholdCount (type=long)
UsageThresholdSupported: UsageThresholdSupported (type=boolean)
CollectionUsageThreshold: CollectionUsageThreshold (type=long)
CollectionUsageThresholdExceeded: CollectionUsageThresholdExceeded (type=boolean)
CollectionUsageThresholdCount: CollectionUsageThresholdCount (type=long)
CollectionUsage: CollectionUsage (
CollectionUsageThresholdSupported: CollectionUsageThresholdSupported (type=boolean)
Valid: Valid (type=boolean)
Name: Name (type=java.lang.String)
Type: Type (type=java.lang.String)
ObjectName: ObjectName (
resetPeakUsage: resetPeakUsage
Parameters 0, return type=void

So we're looking at the usage of Eden Space.You'll see the data coming out

[root@tomcat-01 ~]# java -jar cmdline-jmxclient-0.10.3.jar - java.lang:name=PS\ Eden\ Space,type=MemoryPool Usage
12/03/2016 08:33:58 +0800 org.archive.jmx.Client Usage:
committed: 712507392
init: 537395200
max: 712507392
used: 396006304

Keywords: Linux Java Tomcat Zabbix jvm

Added by xuelun on Mon, 02 Sep 2019 06:20:10 +0300