Version 2.0 of Zabbix has a native support to monitor applications using JMX. The actor that monitors the JMX application is a Java daemon called the Zabbix Java gateway. Basically, it works like a gateway. When Zabbix wishes to know the value of a JMX counter, it simply asks the Java gateway, and the gateway will do all the work for Zabbix. All the queries are done using the JMX management API from Oracle.
The Zabbix Java gateway is in the early stages of development, thus providing great functionality but still experiencing some challenges.
The distinguishing characteristic of this method is that the application only needs to be started with the JMX remote console enabled and doesn't need to implement or extend the class or write new code to handle the Zabbix request because the entire request is a standard JMX.
The default way to enable the JMX console is to start the Java application with the following parameters:
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=<put-your-port-number-here> -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
With these parameters, you are going to configure the JMX interface on the application's side. As usual, you need to define a port, the authentication method, and the encryption.
This basic setup is the simplest and easiest way, but unfortunately, it is not the safest and most secure configuration.
Now, as you are going to open a door in your application, you are basically exposing your application to a security attack. The JMX console, on most of the widely diffused application servers, is not only an entry point to get values from the counter, but also something that is a lot more sophisticated. Basically, with a JMX console open in an application server, you can deploy an application. Start it, stop it, and so on, as you can figure out what a hacker can deploy, and start an application, or cause an issue on the running one. The JMX console can be called from the application server looping itself, using the post
and get
methods. Adding malicious content in the HEAD
section of a web page results in the server that has a JMX console that is not secured and is easily hackable, which is the weakest point of your infrastructure. Once an application server is compromised, your entire network is potentially exposed, and you need to prevent all this. This can be done through the following steps:
-Dcom.sun.management.jmxremote.authenticate=true
-Dcom.sun.management.jmxremote.password.file=/etc/java-6-penjdk/management/jmxremote.password
There are potential security issues with password authentication for JMX remote connectors. Once the client obtains the remote connector from an insecure RMI registry (the default), such as for all the man-in-the-middle attacks, an attacker can start a bogus RMI registry on the target server right before the valid original one is started and can then steal the client's passwords.
-Dcom.sun.management.jmxremote.access.file=/etc/java-6-penjdk/management/jmxremote.access
access
file, for instance, should contain something similar to the following:monitorRole readonly controlRole readwrite
monitorRole <monitor-password-here> controlRole <control-password-here>
-Dcom.sun.management.jmxremote.ssl=true
-Djavax.net.ssl.keyStore=<Keystore-location-here> -Djavax.net.ssl.keyStorePassword=<Default-keystore-password> -Djavax.net.ssl.trustStore=<Trustore-location-here> -Djavax.net.ssl.trustStorePassword=<Trustore-password-here> -Dcom.sun.management.jmxremote.ssl.need.client.auth=true
The -D
parameter will be written in the startup file of your application or application server as, after this configuration, your startup file will contain sensitive data (your keyStore
and trustStore
passwords) that needs to be protected and not be readable from other accounts in the same group or by other users.
To compile the Java gateway, perform the following steps:
$ yum install java-devel
$ ./configure --enable-java
Enable Java gateway: yes Java gateway details: Java compiler: javac Java archiver: jar
$ make && make install
$PREFIX/sbin/zabbix_java
bin/zabbix-java-gateway-2.0.5.jar
lib/logback-classic-0.9.27.jar lib/logback-core-0.9.27.jar lib/android-json-4.3_r3.1.jar lib/slf4j-api-1.6.1.jar
lib/logback-console.xml lib/logback.xml
shutdown.sh startup.sh
settings.sh
-Djavax.net.ssl.*
JavaGateway=<ip-address-here> JavaGatewayPort=10052
StartJavaPollers=5
startup.sh
command.The logs will be available at /tmp/zabbix_java.log
with the verbosity "info"
.
As the Zabbix Java gateway uses the logback
library, you can change the log level or the log file location by simply changing the lib/logback.xml
file. In particular, the following XML tags need to be changed:
<fileNamePattern>/tmp/zabbix_java.log.%i</fileNamePattern><root level="info">
Here, you can change all the logrotation
parameters as well.
If you need to debug a Zabbix Java Gateway issue, another useful thing to know is that you can start the Java gateway in console mode. To do that, you simply need to comment out the PID_FILE
variable contained in settings.sh
. If the startup.sh
script doesn't find the PID_FILE
parameter, it starts the Java gateway as a console application, and Logback uses the lib/logback-console.xml
file instead. This configuration file, other than enabling the log on console, changes even the log level to debug. Anyway, if you're looking for more details about logging on the Zabbix Java gateway, you can refer directly to the SLF4J user manual available at http://www.slf4j.org/manual.html.
Now it is time to create a JMX monitored host with its relatively monitored JMX items. To do that, inside the host configuration, you need to add a JMX interface and address, as shown in the following screenshot:
Once you have done that for each of the JMX counters you want to acquire, you need to define an item of the JMX agent type. Inside the definition of the JMX agent, you need to specify the username, password, and the JMX query string. The JMX key is composed of the following:
The following screenshot shows the Item configuration window:
Data type in this configuration window permits us to store the unsigned integer values (such as 0
and 1
) as numbers or as Boolean values (such as true
or false
).
MBean is quite a simple string defined in your Java application. The other component is a bit more complex indeed; the attribute can return primitive data types or composite data.
The primitive data types are simple types, such as integers and strings. For instance, you can have a query such as the following:
jmx[com.example:Type=Test,weight]
This will return the weight expressed as a numerical floating point value.
If the attribute returns composite data, it is a bit more complicated but is handled since dots are supported. For instance, you can have a pen that can have two values that represent color and the remaining ink, usually dot separated, as shown in the following code:
jmx[com.example:Type=Test,pen.remainink] jmx[com.example:Type=Test,pen.color]
Now, if you have an attribute name that includes a dot in its name, such as all.pen
, you need to escape the dot, as shown in the following code:
jmx[com.example:Type=Test,all.pen.color]
If your attribute name also contains a backslash (), this needs to be escaped twice, as shown in the following code:
jmx[com.example:Type=Test,c:\utility]
If the object name or attribute name contains spaces or commas, it needs to be double-quoted:
jmx[com.example:type=Hello,""c:\documents and settings""]
Unfortunately, JMX support is not as flexible and customizable as it should be; at least at the time of writing this book, JMX still had some problems.
For instance, from my personal experience, I know that JBoss, which is one of the most widely used application servers, can't be successfully enquired. The JMX endpoint is currently hardcoded into JMXItemChecker.java
as follows:
service:jmx:rmi:///jndi/rmi://" + conn + ":" + port + "/jmxrmi"
Some applications use different endpoints for their JMX management console. JBoss is one of them. The endpoint is not configurable as per the host or frontend, and you can't add a parameter to specify this endpoint on the host's configuration window.
Anyway, the development is really active and things are getting better and are improving every day. At the moment, the status is that the Zabbix Java gateway needs some improvement. Also, the current implementation of the Zabbix Java gateway suffers because of the workload; if you have more than 100 JMX items per host, the gateway needs to be restarted periodically. It is possible that you face some errors of the following kind:
failed: another network error, wait for 15 seconds
This is followed by:
connection restored
Also, there is another aspect to consider: in a real-word scenario, it might happen that you have multiple JVMs running on the same hosts. In this case, you need to configure each JMX port creating multiple items and host aliases, one for each network interface; well, this scenario can't be resolved with low-level discovery and requires a lot of manually redundant configuration work. It is fundamental that the implementer of the Zabbix monitoring infrastructure knows not only all the strong points of the product but also the cons and limitations. The implementer can then choose whether they want to develop something in-house, use an open source alternative, try to fix the possible issues, or ask the Zabbix team for a new functionality or fix.