Logging and monitoring
In this chapter, we describe how to configure the logging and monitoring destinations by using in the Liberty server environment in CICS.
We also describe where the logs and other diagnostics are written, and how to use the output to debug issues in practice. We also describe the key tools that can be used to monitor the JVM server status, including CICS statistics and monitoring, CICS Performance Analyzer, IBM Health Center, and Javametrics.
Finally, we review how to use CICS task runaway detection and use CICS policies to monitor transaction activity.
This chapter includes the following topics:
8.1 Message and log files
This section describes the output files that are related to a Liberty JVM server.
8.1.1 CICS logs
Several CICS logs are available for messages that are related to the JVM server.
MSGUSR DD file
Most errors that occur in a JVM server result in a message that is issued to the CICS MSGUSR DD log. This log is one of the first places to look for any CICS-related errors, warnings, or abends. A Java-based AJ05 abend in the CSMT queue that was produced by a NullPointerException in the Liberty CJSA transaction is shown in Figure 8-1.
Figure 8-1 AJ05 Abend message on MSGUSER sysout.
SYSPRINT and SYSOUT DD files
By default, the JVM System.out and System.err streams are sent to the Language Environment stdout and stderr streams. When run under batch MVS, the Language Environment stdout and stderr streams default to the SYSPRINT (stdout) and SYSOUT (stderr) destinations.
If either DD destinations are not defined in JCL, a dynamic SYSnnnn is created in the CICS JOBLOG for that stream.
To prevent a dynamic SYSnnnn being created, you can define the DD cards as shown in the following example:
//SYSPRINT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
|
Note: The SYSOUT DD is distinct from the SYSOUT parameter. The SYSOUT =* parameter is a redirection parameter and in this case, it sends output for the DD to the default output class on your JCL.
|
SYSOUT DD
The SYSOUT DD is a good place to look for any errors from the initialization of the JVM server. These messages are sent to the Language Environment stderr stream.
By introducing a JVM profile error in one of our CICS regions (SC8CICS3), a mis-configuration of the Java directory JAVA_HOME=/usr/lpp/java/J8.0_64_SR44 produces a dynamically created SYSOUT file that is named SYS00007, as shown in Figure 8-2 on page 221.
Figure 8-2 SYSOUT generated by a profile error
When the JVM is initialized, the error is detected and the SYS0007 shows the error message. As when the Java library is wrong, the JVM fails to initialize. Figure 8-3 shows the contents of the SYS00007 file.
Figure 8-3 Error showing the wrong Java library
8.1.2 Java logs
The JVM server provides the following specific log destinations:
•STDOUT
When the JVM server initializes, output from the System.out stream is written to the STDOUT destination that is set by the STDOUT option within the JVM server profile. If the file exists, the output is appended to the end of the file.
If the console log option is enabled, more information is recorded in this file. For more information, see 8.1.3, “Liberty server logs” on page 222. By default, this file is in the WORK_DIR/<applid>/<jvmserver> directory in zFS. In our scenario, the file was redirected to JES setting STDOUT=//DD:JVMOUT in the JVM profile.
•STDERR
The STDERR destination is the primary location to which Java exceptions and stack traces are written. These exceptions are typically a result of errors in Java applications or components. The location is set by using the STDERR option in the JVM server profile.
If the file exists, output is appended to the end of the file. By default, this file is in the WORK_DIR/<applid>/<jvmserver> directory in zFS. In our scenario, the file was redirected to JES setting STDOUT=//DD:JVMERR in the JVM profile.
Figure 8-4 shows the first part of a Java stack trace that was written to the stderr, which was caused by a NullPointerException in our restapp application.
Figure 8-4 Abend AJ05 with NullPointerException written to stderr
8.1.3 Liberty server logs
The following log files are written to by the Liberty server:
•Liberty message log
The message.log file contains all messages that are written or captured by the Liberty logging component. All messages that are written to this file contain more information, such as the message time stamp and the ID of the thread that wrote the message.
Each message features a unique message identifier, and includes an explanation of the problem and details of any action that you can take to resolve the problem. Liberty messages are logged from various sources, including application server components and applications.
The messages.log file is in the logs directory. You can control the size and number of log files by using the following system properties, which can be set in the JVM profile:
com.ibm.ws.logging.max.files
com.ibm.ws.logging.log.directory
com.ibm.ws.logging.max.file.name
com.ibm.ws.logging.max.file.size
This messages.log file can also be copied to JES by using an MSGLOG DD destination, as shown in the following example:
//MSGLOG DD SYSOUT=*
Figure 8-5 shows the Liberty messages that are related to UMASK value, included files, applications started, feature manager messages, and HTTP and HTTPS port allocation output to the MSGLOG DD file in the CICS job log.
Figure 8-5 MSGLOG output
•Liberty trace
The trace.log file contains all trace entries that are written or captured by Liberty. This file is created only if you enable Liberty tracing (usually at the request of IBM level 2 support). Within the JVM profile, selectively enable this trace and specify the file name by using the following parameters:
com.ibm.ws.logging.trace.file.name=trace.log
com.ibm.ws.logging.trace.specification=SSLChannel=all:SSL=all
The format of the log detail level specification is <component> = <level> where <component> is the component for which to set a log detail level, and <level> is one of the valid logger levels (off, fatal, severe, warning, audit, info, config, detail, fine, finer, finest, or all). Separate multiple log detail level specifications by using colons (:).
By default, this file is in the Liberty logs directory along with the messages.log file.
•HTTP access log
The HTTP access log file contains a record of all inbound client requests that are handled by HTTP endpoints in Liberty. You can enable it in the Liberty server for each defined httpEndpoint by using the <httpAccessLogging> element. Example 8-1 on page 224 shows how it can be enabled in server.xml.
Example 8-1 Access logging configuration
<httpEndpoint id="defaultHttpEndpoint"
host="wtsc80"
httpPort="57080"
httpsPort="57443"
accessLoggingRef="accessLogging" />
<httpAccessLogging id="accessLogging"
filepath="${server.output.dir}/logs/http_access.log"
logFormat="%t %a %i %r %s %u %D %B" />
The logFormat attribute includes the following directives for a customized request:
– %t: Time and date
– %a: Remote IP address
– %i: Contents of the Header header-name in the request
– %r: First line of the request URL
– %s: HTTP status code
– %u: Remote user
– %D: Response time of the request in ms
– %B: Response size in Bytes, excluding headers
The information that is contained in the access log file after running our sample restapp application, the Javametrics dashboard, and the CICS defaultapp is shown in Figure 8-6.
Figure 8-6 HTTP access log
One line per HTTP request is available, which provides an easy way of determining the requests were received by Liberty and the HTTP status code was for each request.
•FFDC file
The First Failure Data Capture (FFDC) feature runs in the background and collects events and errors that occur during JVM runtime. The information that it collects are written to log files in the logs/ffdc directory.
Summary files are shown in Figure 8-7 on page 225. One entry for each exception occurred in the system, as shown in Figure 8-8 on page 225. The FFDC can be useful in your problem determination efforts. Also, they might be required by the Liberty server support team if you open a PMR.
Figure 8-7 FFDC exception summary file
Figure 8-8 FFDC Exception log
•Console log
The Liberty console log is not enabled by default in a CICS Liberty JVM server. It can be enabled by using the following system property:
com.ibm.ws.logging.console.log.level={INFO|AUDIT|WARNING|ERROR|OFF}
The ERROR level is the highest level of logging and causes all INFO, AUDIT WARNING, and ERROR messages to be written to the JVM server stdout destination. These messages are the same as the messages that are written to the messags.log file but do not include the time stamp and thread identity information.
When written to the stdout destination, the messages can be redirected to JES and prefixed with the identity of the JVM server by using the IDENTITY_PREFIX option in the JVM server profile, as shown in Figure 8-9.
Figure 8-9 Console log output with INFO level
8.1.4 JVM server trace output
In addition to the logging that is produced by the JVM server, CICS provides some standard trace points in the SJ (JVM server) and AP domains. These trace points trace the actions that CICS takes in setting up and managing JVM servers and running Java transactions.
The SJ component traces exceptions and processing in SJ domain to the internal trace table when set at level 2. SJ level 3 and beyond produces Java logging that is written to a trace file in zFS. The name and location of the trace file is determined by the JVMTRACE option in the JVM profile. If you want to capture the maximum amount of trace, setting the SJ component to ALL ensures that both CICS internal trace and the zFS trace files are produced.
You also can control the format of the trace by using the following JVM server system property (choose the option for your own purposes):
com.ibm.cics.jvmserver.trace.format={FULL|SHORT|ABBREV}
The default option is SHORT and this option is used when you send diagnostic information to IBM service.
Activating trace
You can modify and control CICS tracing by using the CETR transaction. Enter CETR at a CICS workstation and after you see the map that is shown in Figure 8-10 on page 227. Then, press PF4 for trace components.
Figure 8-10 CETR trace control facility transaction
On the first trace options page, set the trace values for AP domain to 1-2, and for SJ domain to ALL, as shown in Figure 8-11 and Figure 8-12.
Figure 8-11 Setting AP domain trace level
Figure 8-12 Setting SJ domain trace level
Then, press Enter and PF3 to return, and enter STARTED in the Auxiliary Trace Status field.
To demonstrate this trace, we called our restapp application taskInformation service, and used the following URL to show how the trace information is output for a CICS Liberty Java task:
wtsc80:52080/com.ibm.cicsdev.restappext/rest/taskInformation
The taskInformation service returned the following JSON response, which provided more information about the transaction ID, user ID, and task number:
{"transid":"CJSA","userid":"CICSUSER","tasknum":"339"}
The start of the JVM server trace file, which was in the zFS file /var/cicsts/SC8CICS2/ITSOJVM1/D20171129.T160838.dfhjvmtrc, is shown in Figure 8-13.
Figure 8-13 JVM server zFS trace
In this trace output, we can see the URL, the HTTP request type of GET, and the client IP address. This output is followed by the call to buildTransaction(), which sets the transaction ID to CJSA, and then attaches the thread to the CICS task 339. Finally, the thread name is switched from the RUN_SERVICE_Thread-28 to DFHSJTHP.TASK339.CJSA.
In Figure 8-14, we can see the CICS AUX trace for the SJ=ALL and AP=1-2 components, which are formatted in short format by using the DFHTU710 trace utility program, and filtered by using task number 339. This trace shows how the buildTransaction() method starts the task in CICS on the T8 TCB T802E and then starts to call the ASSIGN command that is driven from the JCICS Task.getTask() method call.
Figure 8-14 CICS AUX trace
Stopping trace
To stop CICS writing an internal trace to the auxiliary trace dataset, you can set Auxiliary Trace Status to OFF in the CETR transaction.
To stop JVM server tracing, the values for the AP and SJ domains must be back to 1 in the CETR transaction because this value is not affected by the AUX or SYSTEM trace settings.
8.2 Monitoring tools
In this section, we describe several methods that can be used to monitor CICS JVM server activity, including CICS statistics and CICS performance class data. These methods can be extracted from system management facilities (SMF) 110 records.
8.2.1 CICS statistics records
CICS collects statistics for the JVM servers and JVM programs. You can use these statistics to manage and tune the Java workloads that are running in your CICS region. The JVM server statistics provide information about the JVM and activity within a particular JVM server. The JVM program statistics provide information about the use of CICS Java programs. In the context of a Liberty JVM server, these programs are the only Java programs that are started by using the Link to Liberty function (for more information, see Chapter 8, “Logging and monitoring” on page 219).
To view the statics online, use the sample STAT transaction that prints the statistics to the CICS log by using the DFH0STAT program. You also can develop your own reporting application by using the EXEC CICS EXTRACT STATISTICS command. If you use the CICS Explorer, the JVM Servers view also displays all the same statistics fields.
For batch reporting, the sample utility program DFHSTUP provides a simple method to report on any type of CICS statistics. A sample JCL for running this program to report on our JVM server is shown in Example 8-2.
Example 8-2 DFHSTUP sample JCL
//JOB ...
//STUP1 EXEC PGM=DFHSTUP,REGION=0M
//STEPLIB DD DISP=SHR,DSN=CICSTS54.CICS.SDFHLOAD
// DD DISP=SHR,DSN=CICSTS54.CICS.SDFHAUTH
//DFHSTATS DD DISP=SHR,DSN=CICSUSER.CICS.SMF110(0)
//SYSIN DD *
SELECT APPLID=(SC8CICS7)
COLLECTION TYPE=ALL
SELECT TYPE=(JVMSERVER)
DATE START=10/06/2017,STOP=10/06/2017
TIME START=00.00.00,STOP=23.00.00,ELAPSED
/*
//DFHSTWRK DD UNIT=SYSDA,SPACE=(CYL,(8,4))
//SORTWK01 DD UNIT=SYSDA,SPACE=(CYL,(4))
//SORTWK02 DD UNIT=SYSDA,SPACE=(CYL,(4))
//SORTWK03 DD UNIT=SYSDA,SPACE=(CYL,(4))
//SORTWK04 DD UNIT=SYSDA,SPACE=(CYL,(4))
//DFHPRINT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
//SYSABEND DD DUMMY
The output from DFHSTUP regarding our JVMSERVER ITSOJVM1 is shown in Figure 8-15 on page 231. The statistic period is set for one-hour intervals (for 10/19/2017 from 16:00:00 to 17:00:00).
Figure 8-15 JVMSERVER DFHSTUP output
The report shows the following information:
•Our Liberty JVM server was used 1466 times, which equates to all the HTTP and JMS MDB requests that were received.
•The thread limit of the JVM server was set to a maximum of 50.
•There are 13 active threads, which are the threads available in the pool.
•The peak thread usage was 27 threads.
•No requests are needed to wait for a thread; therefore, the limit of 50 threads appears to be fine for our JVM server.
•Seven calls were made to the system thread, most of which were caused by the use of CEMT SET JVMSERVER commands.
•The heap size is 97 MB out of a maximum of 256 MB.
•The heap size after the last garbage collection (GC) was 92 MB. This result is known as the occupancy and often remains stable in a well-tuned system.
•The GC policy in use is gencon. This policy splits the Java heap into two areas, the new or nursery area for short lived objects, and the old or tenured area for long-lived objects.
•One major GC event (for the tenured area) and 91 minor GC events for the short-lived objects were observed in the nursery area.
This report can be generated daily by using a statistics period of one hour to track the performance of the JVM server over time.
|
Tip: Another useful metric is total GC heap freed divided by the number of requests. This formula gives an average garbage per request figure, which is useful to track on across application changes.
|
For more information about how to use the DFHSTUP and DFH0STAT reporting programs, see IBM Knowledge Center.
8.2.2 CICS performance records
CICS provides the following programs for processing any CICS monitoring data that is written to SMF data sets:
•DFHMNDUP
This utility program generates a performance dictionary record in a sequential data set for use with monitoring data that is extracted from SMF data sets. The JCL for running this program is shown in Example 8-3.
Example 8-3 DFHMNDUP sample JCL
//JOB …
//MNDUP EXEC PGM=DFHMNDUP
//STEPLIB DD DSN=CICSTS54.CICS.SDFHLOAD,DISP=SHR
//SYSUT4 DD DSN=CICSUSER.CICS.MNDUPREC,DISP=(NEW,CATLG),
// UNIT=SYSDA,SPACE=(TRK,(1,1))
//SYSPRINT DD SYSOUT=A
//SYSUDUMP DD SYSOUT=A
//SYSIN DD *
MCT=NO
SYSID=SC80
GAPPLID=SC8CICS7
//* 
•DFH$MOLS
This print program is for CICS monitoring data. DFH$MOLS is a sample program that you can modify or adapt to your own purposes. The JCL for running this program is shown in Example 8-4.
Example 8-4 DFH$MOLS sample JCL
//JOB …
//EXPAMND EXEC PGM=DFH$MOLS
//STEPLIB DD DSN=CICSTS54.CICS.SDFHLOAD,DISP=SHR
//INPUT DD DISP=OLD,DSN=CICSUSER.CICS.MNDUPREC
// DD DISP=SHR,DSN=CICSUSER.CICS.SMF110(0)
//SYSPRINT DD SYSOUT=A
//SYSOUT DD SYSOUT=A
//SORTDIAG DD SYSOUT=A
//SYSIN DD *
SELECT APPLID=SC8CICS7
SELECT TRANID=CJSA,CJSU,JDB2,JRES,JJMS
PRINT PER
RESOURCE ALL
//*
The output report from DFH$MOLS that is shown in Figure 8-16 on page 234 displays the following information for a single CICS task from an HTTP request to our Java web application:
•The transaction name (TRAN), in this case it is CJSA, which is the default JVM server request processor transaction.
•The start and stop time stamps for the CICS task (START and STOP).
•The number of performance class records that is recorded is 1 (PERRCENT).
•The CICS task number (TRANNUM) and transaction priority (TRANPRI).
•The IP address (9.27.245.24) of the browser (CLIPADDR).
•The dispatch time (USRDISPT) across all CICS TCBs, which is 1081 ms. This dispatch time equates approximately to the response time.
•The CPU usage timers, including:
– The total CPU time on all types of CPU (USRCPUT), which is 810 ms.
– The amount of CPU that is used by the task on the Java T8 TCB (T8CPUT), which is also 810 ms.
– The amount of CPU that is used by the task on the QR TCB (QRCPUT), which is minimal at 5 μs.
– The time that is spent waiting for dispatch on to the QR TCB (QRMODDLY), which is insignificant at 4 μs.
– The CPU time on a general-purpose CPU (CPUTONCP), which is 3.895 ms.
– CPU time that was eligible for zIIP offload, but ran on a general-purpose CPU (OFFLCPUT), which is 3.160 ms.
|
zIIP analysis: From the metrics that are shown in Figure 8-16 on page 234, we can calculate the CPU time that is used on a zIIP engine for our task is USRCPUT-CPUTONCP, which is 806 ms.
We can also calculate the total time spent zIIP eligible, which is the amount of time that is spent on a zIIP engine (806 ms as shown in Figure 8-16 on page 234), plus the extra 3.16 ms, which was zIIP eligible but ran on a general-purpose CP (OFFLCPUT), which makes 809 ms.
Therefore, the zIIP offload eligible fraction is the amount of zIIP-eligible CPU time (809 ms) divided by the total CPU time (810 ms), or 99.9% for this task.
|
Figure 8-16 DFH$MOLS output for CJSA transaction
All of these fields are taken from the SMF 110.1 record (subtype 1). For more information about programming the structure of CICS SMF type 110, and how the monitoring data is packaged in the SMF records, see IBM Knowledge Center.
8.2.3 CICS Performance Analyzer
CICS Performance Analyzer (CICS PA) is a reporting tool that provides information about the performance of your CICS systems and applications. It helps you tune, manage, and plan your CICS systems effectively. By using the SMF records from our current scenario, CICS PA can be used to easily generate more detailed performance reports.
A CICS PA report form that includes the fields that are related to JVM usage and CPU time is shown in Figure 8-17 on page 235. The report definition also allows transaction selection. By using the edit form facility, you can select the fields of your choice to create your own report.
Figure 8-17 CICS performance analyzer report form
The fields that we selected in our example report show the following information:
•TRAN: Transaction identifier and task number.
•RESPONSE: Transaction response time.
•DISPATCH: Total user task dispatch time.
•CPU: CPU time across all CICS TCBs.
•JVMSUSP: Time in which the user task was suspended.
•JVMTHDWT: Time waiting for a JVM server thread.
•JVMTIME: Elapsed time that is spent in the CICS JVM by the user task.
•CPUONSP: CPU time that was offloaded to zIIP speciality processors.
•CPUONCP: CPU time on general-purpose CPU.
•CPUONCPE: CPU time on general-purpose CPU that can be offloaded but was not.
•CUPONCPN: CPU time on general-purpose CPU that was not eligible for offload.
•CPUISSPE: CPU time that is converted to Service Units.
•ABCODEC: Task abend code, if any.
After the job is submitted, the CICS PA report shows results for selected transactions, as shown in Figure 8-18 on page 236.
|
Note: The use of CICS PA for performance reporting can be easier than the use of DFH$MOLS because several other fields are automatically calculated, such as CPUONSP.
|
Figure 8-18 CICS PA performance list
For more information about the CICS PA fields and how to correlate these fields with the CICS SMF 110 performance class data, see IBM Knowledge Center.
8.2.4 CICS Explorer
CICS Explorer is a management tool for developers and systems administrators. It offers a graphical interface for managing one or more CICS environments. It can be used for most systems management tasks, including viewing CICS system initialization (SIT) parameters, managing resource definitions, and building and deploying CICS bundle projects.
CICS Explorer is required if you deploy applications by using CICS bundles, including Java applications that are deployed into Liberty.
In our scenario, CICS Explorer is used to monitor the JVM servers, bundles and bundleparts, URI maps, and other related resources. A sample CICS Explorer JVM server operations view, which includes the JVM server statistics records, is shown in Figure 8-19 on page 237.
Figure 8-19 CICS Explorer JVM Server view
8.2.5 IBM Health Center
IBM Health Center is a diagnostic tool for monitoring the status of a Java virtual machine (JVM) and can be used with any IBM JVM, including CICS JVM servers. It features low runtime overhead and provides a highly customizable client as part of the IBM Support Assistant (ISA) workbench. Because the Health Center client is Eclipse based, it can also be installed into other Eclipse environments, such as the IBM CICS Explorer or z/OS Explorer.
The first step is to install the Health Center plug-in into your Eclipse client. Complete the following steps to add a software repository to your Eclipse:
1. In the CICS Explorer, click Help → Install New Software.
2. Click Add to add a software repository and provide the following location, as shown in Figure 8-20:
– Information Name: IBM Health Center site
– Location: http://public.dhe.ibm.com/software/websphere/runtimes/tools/healthcenter/
Figure 8-20 Adding the Health Center repository
Figure 8-21 Health Center Installation window
4. Review the license information and click Next.
5. Click OK to confirm the installation of unsigned content, and then, click Yes to restart the Eclipse when prompted.
Enabling the JVM server for monitoring
To enable a JVM for monitoring, it is necessary to add a few JVM arguments to enable the Health Center agent to listen on the required TCP/IP port. When a CICS JVM server is used, the JVM arguments are set in the JVM profile, which is in the UNIX System Services directory that is specified by the JVMPROFILEDIR SIT parameter.
Add the following –Xhealthcenter JVM argument in your JVM profile. This argument enables the Health Center agent to collect monitoring data and specifies the listening port. We specified the port number 57108, as shown in the following example:
-Xhealthcenter:port=57108
The JVM must be disabled or enabled to implement this change. After the JVM server is started, the Health Center agent initializes and immediately starts buffering monitoring data ready for collection by the Health Center client. At this stage, you can check that the Health Center agent is listening on the TCP/IP port by using the following MVS TSO command:
NETSTAT (PORT 57108)
|
Note: Only one Eclipse client session can be connected to an IBM Health Center agent at any time.
|
Using the Health Center
Now that the JVM server is running, you can use the Health Center views to monitor the JVM server. The Health Center Environment perspective provides a convenient way to access all the Health Center function. It can be accessed by clicking Windows → Open Perspective → Other → Health Center Environment.
To connect to the Health Center agent that is running in the JVM, you must run the connection wizard. This wizard can be accessed from any of the Health Center perspectives by clicking File → New Connection menu.
The Health Center connection wizard starts and you see the window that is shown in Figure 8-22. Click Next and enter the information for the listening port (57108) as specified previously in the CICS JVM profile.
Figure 8-22 Health Center connection wizard
Click Next and the window in which you confirm the port that is being used by a Health Center agent on the host is shown (see Figure 8-23). Click Finish.
Figure 8-23 Connection confirmation
Figure 8-24 IBM Health Center main window
In the window that is shown in Figure 8-24, the general information regarding runtime environment, runtime properties, environment variables, security properties is shown. On the left side of the window, several links to information about the status of the following items are available:
•Environment information
•Classes
•Garbage collection
•I/O
•Locking
•Threads
•Native memory
•Method Profiling
Figure 8-25 on page 241 shows the Garbage Collection perspective. This perspective provides a set of views to assist in analyzing the collection (GC) process that is used by the JVM to manage memory in the JVM heap.
By using the default gencon GC policy the Java heap is split into two areas: the new or nursery area, and the old or tenured area. CICS JVM server statistics call new or nursery activity minor GC and call old or tenured activity major GC.
Figure 8-25 IBM Health Center garbage collection window
Much of this information is also displayed in the CICS Explorer JVM Servers view, which can be added to the Garbage Collection perspective by clicking Window → Show view → Other → JVM Servers. In addition, individual Health Center graphs for Used heap and Pause time can be added by using the same Show view menu. A customized perspective is created with a combination of Health Center and CICS views.
Custom perspectives such as these can be saved for future use by right-clicking the current perspective and selecting Save As, which adds the perspective to the list of available perspectives on the top row.
The Threads perspective provides detailed analysis of all the threads that are running within the JVM. A graphic view shows the thread count utilization against time.
The Current Threads view supplies a useful thread name filter, which allows the display to be filtered. For each thread that is displayed, the call stack can be displayed, which shows where each thread is suspended.
In our example that is shown in Figure 8-26 on page 242, we can see that CICS task 53525 is running under transaction ID CSMI and is suspended in a Thread.sleep() method that is named from the main method of the JavaLinker() class.
Figure 8-26 IBM Health Center threads window
8.2.6 IBM Application Metrics for Java
Application Metrics for Java (Javametrics) instruments the JVM in a Liberty server for performance monitoring, which provides data visually in a web-based dashboard.
Javametrics can be downloaded from the GitHub website.
The following data collection sources are included:
•Environment: Machine and runtime environment information
•CPU: Process and system CPU
•GC: Percentage time that is spent in garbage collection
•Memory: Java native and non-native memory usage
•HTTP: HTTP request information
For more information, see the GitHub repository.
To install Javametrics in our environment, we completed the following steps:
1. Downloaded the latest Javametrics release from GitHub, which includes the following components:
– javametrics-dash-x.x.x.war (Javametrics Web Application)
– javametrics-agent-x.x.x.jar (Javametrics agent)
– ASM directory with related components
2. Copied the web application javametrics-dash-1.0.1.war file into our application deployment directory /var/cicsts/SC8CICS7/ITSOJVM1/apps.
3. Created a Javametrics directory in /var/cicsts/SC8CICS7/ITSOJVM1/app and copied the javametrics-agent-1.0.1.jar and asm folder into this directory (see Figure 8-27).
.
Figure 8-27 Javametrics application deployment directories
Example 8-5 Javametrics deployed as an application in server.xml
<webApplication id="javametrics-dash-1.0.1"
location="/var/cicsts/SC8CICS7/ITSOJVM1/apps/javametrics-dash-1.0.1.war"
name="javametrics.dashboard">
</webApplication>
5. To start data collection, Javametrics requires a Java option to be set to load the agent JAR. In our scenario, the following line was added to the JVM profile for SC8CICS7 region:
-javaagent:/var/cicsts/SC8CICS7/ITSOJVM1/apps/Javametrics/
javametrics-agent-1.0.1.jar
After the application is installed, you should see the messages in the Liberty messages.log file that are shown in Example 8-6.
Example 8-6 Javametrics deployed as an application
CWWKZ0018I: Starting application javametrics-dash-1.0.1.
SRVE0169I: Loading Web Module: javametrics.web.
SRVE0250I: Web Module javametrics.web has been bound to default_host.
CWWKT0016I: Web application available (default_host): http://wtsc80.cpolab.ibm.com:57080/javametrics-dash/
CWWKZ0001I: Application javametrics-dash-1.0.1 started in 0.307 seconds.
SESN0176I: A new session context will be created for application key default_host/javametrics-dash
CWWKH0046I: Adding a WebSocket ServerEndpoint with the following URI: /javametrics-socket
The URL for the dashboard consists of the server’s default HTTP endpoint and javametrics-dash, as shown in the following example:
http://wtsc80.cpolab.ibm.com:57080/javametrics-dash/
Our Javametrics dashboard is shown in Figure 8-28 on page 244, in which you can see the following information:
•HTTP Incoming Requests: Response time measured in milliseconds.
•HTTP Throughput: Measured as requests per second (RPS).
•Top five average Response Times.
•Garbage Collection Time: Expressed in percentage over total along time.
•Heap size: Used memory, used native memory, total used memory, and used heap after GC along time.
•Environment information: Host name, number of processors, OS architecture.
Figure 8-28 Javametrics Dashboard
Most of the data is plotted as the following line graphs:
•HTTP Incoming Requests, HTTP Outgoing Requests, and Other Requests show event duration against time.
•HTTP Throughput shows requests per second.
•Average Response Times shows the top five HTTP endpoints that on average take the longest to respond.
•CPU and Memory graphs show system and process usage over time.
•Heap shows the maximum heap size and used heap size over time.
A maximum of 15 minutes of data is shown across all graphs. If substantial data is being produced by the application that is monitored, the dashboard automatically aggregates the data.
If a graph includes points, hovering over one of these points provides more information. For example, HTTP Incoming Requests show the response time and the requested URL, as shown in Figure 8-29 on page 245.
Figure 8-29 Hovering over a points to get more information
The Javametrics dashboard can help you to identify the following common performance problems:
•Slow HTTP response times on some or all routes
•Lower than expected throughput in the application
•Spikes in demand causing slowdown
•Higher than expected CPU usage for the level of throughput/load
•High or growing memory usage (potential memory leak)
8.2.7 Runaway tasks
To provide a system level mechanism to stop looping applications, the JVM server in CICS TS V5.4 now integrates CICS task runaway detection with the JVM server phase-out function.
Unlike traditional CICS tasks, a task that is running Java on a T8 TCB cannot be ended without consequences to other workload in the same JVM. Language Environment and the JVM server run in a POSIX-compliant environment, which mandates that if a thread is ended, the parent process is also ended. In turn, all child processes are ended abruptly, which causes all tasks in the JVM to fail immediately.
A task that is running in a JVM server that exceeds the RUNAWAY interval that is defined for the transaction experiences a more controlled stop process. This process differs from the traditional CICS behavior and you evaluate whether you want runaway intervals to apply to your Java tasks, or what value to set.
|
Note: This new runaway function is also included in CICS TS V5.3 with APAR PI77502.
|
JVM server runaway processing
When a task that is running Java experiences a runaway interval condition, CICS intercepts the condition and triggers a phase-out of the JVM server. New transactions are prevented from entering the JVM server and work is left to drain.
Then, if the task completes its processing, the JVM server enables again and becomes available for new requests. In many cases, if a task that is running in Java exceeds the runaway interval value, it is likely to be a bad application, such as a tightly looping application that prevents successful phase-out recycling of the JVM server.
When such a looping application is detected, the runaway timer triggers again after another interval and the JVM server phase out is escalated to a JVM server purge. Remaining tasks are subject to task purge processing and in most cases, are ended.
If further runaway intervals are exceeded, the JVM server purge escalates to a forcepurge and ultimately a kill until all running tasks are forcefully ended. The JVM server recycles back to the enabled state ready for new requests. If the JVM server must escalate as far as a kill request, it is prudent to recycle CICS at the earliest opportunity.
|
Tip: A runaway condition for a task that is running in a JVM server can cause temporary availability problems for the entire JVM server. For this reason, CICS modifies the runaway interval value that is specified for the transaction by multiplying it by a factor of 10 (up to a maximum value of 45 minutes). For example, if the TRANSACTION definition specifies RUNAWAY=5000, the effective runaway interval for that task when it runs in a JVM server is 50,000 milliseconds.
|
Setting the runaway interval value
By default, the CJSA transaction definition that is used for Liberty JVM servers has runaway detection active and set to the system interval. If you do not want runaway intervals to apply to these tasks, you can run work under your own transaction definitions set by using URIMAP definitions with the runaway interval set to 0, or another value of your choice.
8.2.8 CICS policies
In this section, we describe how a CICS policy can be defined and then deployed into a CICS region such that the policy rules are applied to all tasks in that region.
CICS TS V5.1 introduced the capability to define policies to monitor the resource utilization of a user task, and to automatically respond when resource usage exceeds the thresholds you define. In this way, excessive resource usage and certain types of looping and runaway transactions can be detected and managed.
A CICS policy is defined in a CICS bundle project and each bundle can define one or more policies. After the bundle is defined, it can then be deployed to a CICS region by using one of the following methods:
•At the region level.
•For a CICS Cloud application, at the platform level.
•For a CICS Cloud application at the application level.
•For a specific resource entry point, such as a URI map, transaction, or program to restrict all request that use that entry point.
We use the first method and deploy a policy at the region level to issue a message if a transaction runs for longer than 5 seconds. When this issue occurs, a message is written to the CICS log, which can then drive further automation, if required. Messages are sent to the CMPO destination, which by default is redirected to CSSL but can be directed to a separate log destination, if required.
|
Tip: When CICS policies are defined, we recommend the use of a minimum of CICS Explorer V5.4 because it supplies a new policy editor. This editor makes it able to view and edit policies after they are created.
|
Creating the policy
Complete the following steps to create a policy:
1. In CICS Explorer, open the Resource perspective. Click File → New → CICS Bundle Project and create a project with the name elapsedtime.policy.bundle, as shown in Figure 8-30.
Figure 8-30 New CICS Bundle Project
2. Right-click the new bundle project and select New → Policy Definition.
3. In the Create Policy Definition window, enter JVM_Elapsed.policy as the file name and click Finish (see Figure 8-31).
Figure 8-31 Creating a policy definition
4. When the Policy Overview window appears, enter the description JVM elapsed >5, as shown in Figure 8-32.
Figure 8-32 Policy definition tab
5. Click New to define a new rule. A pop-up window opens (see Figure 8-33) in which you are prompted to choose the type of rule. In our scenario, we chose Task Rules → Time to create a rule that is based on the elapsed time in a transaction (time rules can be based on elapsed time or CPU time).
Figure 8-33 New policy rule
6. Click OK. The Rules editor opens is which you set the following conditions for the rule, as shown in Figure 8-34 on page 249.
a. Select Elapsed time in the condition drop-down list and enter 5 seconds for the greater than condition trigger.
b. Under Action, select the system default to Issue a message.
c. Close the editor and select Yes to save the rule.
Figure 8-34 Defining policy rules
7. Export the bundle project to a zFS location. We exported our bundle project elapsedtime.policy.bundle to the following CICS bundle deployment directory in zFS (see Figure 8-35):
/var/cicsts/SC8CICS7/ITSOJVM1/bundles
Figure 8-35 Export to zFS window
8. This bundle must be installed into CICS. By using CEDA, create a CICS BUNDLE resource that is named JVMELA and set the bundledir attribute as the exported zFS directory elapsedtime.policy.bundle_1.0.0 in our JVM server bundle deployment directory, as shown in Figure 8-36.
Figure 8-36 CEDA define BUNDLE
9. Install the BUNDLE JVMELA. It should enter the Enabled state.
If you enter the transaction CEMT INQ BUNDLE (see Figure 8-37), you see that the bundle JVMELA marked as Enabled and includes a Part count, Target count and Enabled count of 00001 indicates that the policy that is defined in the bundle part is active.
Figure 8-37 CEMT INQ BUNDLE
To test our policy, we ran a modified version of our restapp application taskInformation service. This application is modified to delay for 10 seconds within the Java code before running any JCICS commands. The JVM elapsed time policy then starts when the first JCICS command runs after the elapsed time for the transaction exceeds 5 seconds, as shown in Example 8-7.
Example 8-7 Message issued when policy triggered
DFHMP3001 11/30/2017 04:14:39 SC8CICS2 Task 04417(CJSA) exceeded a policy threshold. BundleId=elapsedtime.policy.bundle, PolicyName=JVM_Elapsed, RuleName=Elapsed_5, RuleType=time, Category=elapsedlimit, Threshold=5000000 (Value=5,Unit=S),
CurrentCount=10002168.
CurrentCount=10002168.
In this way, a policy can be used to monitor the activity of CICS transactions and use message automation to trigger other automated procedures.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.