Performance and throughput considerations
Performance, like beauty, is in the eye of the beholder. It can be very subjective but at the same time, can be precisely measured. This chapter discusses the performance attributes of JES2 and JES3. Though it will not be able to answer the question of how JES2 will perform in your environment, it will improve your understanding of the performance differences.
13.1 Performance of JES2 compared to JES3
Measuring performance involves looking at factors such as CPU usage, I/O rates, and memory usage. Comparing one version of a product to another generally involves keeping the workload constant, updating the products being compared, rerunning the workload, and comparing the measurements. But when comparing JES2 to JES3, the problem is complicated by the differences in where each JES does its processing.
In JES3, there is a JES3 address space on each z/OS image. One of those address spaces is designated the JES3 global. The other JES3 address spaces are designated JES3 locals. The JES3 global maintains the job queues, selects work to be processed on each z/OS image in the JESplex, and processes many of the requests made to JES3. The JES3 locals normally do relatively little processing. The local’s primary purpose is to act as a hot standby in case the global goes down, and to process requests that must be run on that particular z/OS image. For example, data set locate processing using a catalog or volume that is only accessible on that image, and commands that need processing on the local’s image.
In JES2, similar to JES3, each z/OS image has a JES2 address space. However in JES2, all instances of JES2 are the same. There is no centralized global processor responsible for distributing work to each member of the JESplex. Instead, each JES2 address space is responsible for providing work to that z/OS system and for servicing requests from that system. Each address space maintains a copy of the work queues, selects work to process on the system, and handles all requests made of JES on that member.
There is no straightforward comparison that can be made of a JES3 address space and a JES2 address space. Perhaps in a single member JESplex, some comparisons can be made. But there are few of these environments in the real world. You could aggregate the consumption of all the JES3 address space and all the JES2 address spaces, but that would miss another important component, communications used by JES to accomplish the needed tasks. JES3 uses XCF (through a component called JESXCF) to perform the required JESplex communications. JES2 also uses XCF (through JESXCF) but to a lesser degree. However, JES2 uses the checkpoint to share the work queues among the member.
Processing for the XCF messaging done by JES is part of the JESXCF and XCF address spaces. Since JES3 does more of its communications using JESXCF, that also has to be considered when trying to measure performance. JES2 checkpoint processing occurs in the JES2 address space. If the JES2 checkpoint is in a Coupling Facility structure, there is the cost of XES processing and the Coupling Facility overhead that is also part of the performance picture.
The comparison is further complicated by the differences in where each JES performs different parts of their processing. Depending on the function (and sometimes the options), processing might be done in the JES address space or in another address space. Here are some examples of processing differences:
C/I Conversion and interpretation processing converts the JCL that was submitted into an internal representation that can be used by z/OS to run a job. In JES3, C/I processing can be done on the JES3 global or it can be offloaded to a JES3 C/I FSS. In JES2 V2R1, it works the same way. Before that release, conversion was always done in the JES2 address space and interpretation was done when work was selected for execution. That processing might occur on the system that the job was submitted on, but it does not necessarily have to occur there.
Input Input processing creates the basic structure for a new job that is being submitted. During the input phase, a first pass through the submitted JCL is performed, processing any JECL cards and building the required instream data sets. In JES3, input processing is performed in the JES3 address space on the global. In JES2, internal reader and NJE over TCP/IP processing is primarily done in the submitting (application or NETSERV) address space. For other input sources, processing is done in the JES2 address space.
Extended Status Extended status is an interface into JES to query the status of jobs and output in the system. JES3 does most of the processing in the JES3 address space on the global. All the results are sent from the global to the requesting address space via JESXCF. In JES2, all the processing for extended status is performed in the requesting address space.
Processing for the JES Property SSI and the JES Device SSI also is done by the global address space in JES3 and in the requesting address space in JES2.
These interfaces are used by products like SDSF, FTP, z/OSMF, and others that display information about jobs and devices associated with JES.
SNA NJE NJE processing involves CPU intensive processing to build the data records required to transmit data across NJE. Since JES3 uses BDT to perform SNA NJE processing, that CPU intensive processing occurs in the BDT address space. JES2 does all the SNA processing in the JES2 address space.
Application SPOOL I/O
When applications write SYSOUT to SPOOL, they use a JES-specific access method under the covers to perform the SPOOL I/O. In JES3, SPOOL write I/Os from all address spaces on a system are queued to a single write queue (per SPOOL extent) to be written. The actual STARTIO request will be issued in whatever address space is in control when the current I/O ends or when the first element is added to the queue. JES3 maintains one active I/O per SPOOL extent per system.
In JES2, each address space performs its own SPOOL I/O. Each address space can start up to 2 I/Os per SPOOL extent. Each I/O is recorded (charged) to the address space that issued the I/O. I/O queuing priority also applies to SPOOL I/Os. JES2 can also take advantage of multiple paths (PAV) to the SPOOL volumes.
Monitor Each JES has a real-time monitor that looks at what the JES is doing and reports anomalous situations. These monitors tend to poll JES looking for processes that are looping or waiting for extended periods of time. By their nature, they tend to use noticeable amount of CPU. In JES3 the monitor runs in the JES3 address space. In JES2, there is a separate address space (JES2MON) that performs the monitoring.
There are other more subtle examples of these differences. Because of the complexity of where processing is done and trying to identify what processing is JES-related versus application-related, we cannot make simple statements about how much CPU one JES uses compared to the other. What we can do is look at factors like overall system overhead to understand the requirements of each JES. We can also look at wall clock time to complete specific activities as a way to compare the performance of the two JESs.
13.1.1 Steady state processing
When trying to compare the performance of JES3 and JES2, one easy environment to measure is steady state processing. In an idle system how much CPU is being used by the JESs? In JES3’s case, there is nothing for it to do when there are no requests to process and thus the global is mostly idle. It is awakened by requests to perform processing as they occur. However, each JES2 acts independently and primarily discovers work by reading the JES2 checkpoint data set. In effect, it is polling the checkpoint on a periodic basis to see whether any other member has added work that needs to be processed.
As a result, even in an idle system, JES2 is constantly reading and writing the JES2 checkpoint data set and consuming resources. How often this is done can be controlled by JES2 tuning parameters on the JES2 MASDEF statement. For more information about tuning JES2, see z/OS JES2 Initialization and Tuning Guide, SA22-7532.
Another thing that occurs on an idle JES2 system is the maintenance of data areas used by various interfaces. Even though JES2 processes extended status requests in the user address space, the data that is used to satisfy the request comes from a copy of the JES2 work queues that is maintained in a data space. The data space copy is kept up to date every time the JES2 checkpoint is written. Similarly, device information is stored in 64-bit common storage for use by the JES device information requests.
13.2 Throughput considerations
Where performance is often looked at as how much resource is needed perform a task, throughput is the amount of work that can get done in a given time. In some cases, throwing more resources at a problem can improve throughput. However, in some cases, throwing more resources at a problem will not get it done any faster.
Often the solution to a throughput problem is not more resources but a better way of utilizing the existing resources to get more stuff done in a unit of time.
13.2.1 MAS considerations
Both JES3 and JES2 have a single main task that is running in the JES address space. This task services the requests that are made of each JES.
In JES3 though, each JES3 address space has a main task, and most of the work occurs in the main task on the JES3 global. This task must handle most of the requests made of JES3 throughout the JESplex.
In JES2, the main task in each JES2 address space handles JES2 requests that originate on that member (they are not sent to other members to process). However, the JES2 main task must access the JES2 checkpoint to respond to many requests. Since only one member of a MAS can access the checkpoint at a time, there can be only one main task actively processing these requests in the JES2 MAS. However, not all processing done by the JES2 main task requires access to the checkpoint, meaning that some progress to be made on requests when the checkpoint is not owned.
If JES2 is running as a single member MAS, it can constantly own the checkpoint and there are no delays introduced by not owning the checkpoint. However, when a second member joins a MAS, the checkpoint access introduces a delay processing many requests (this is known as a MAS penalty).
Tuning of the JES2 checkpoint can greatly reduce the impact of checkpoint serialization. This can make the checkpoint available in a more timely manner. But there is only so much tuning that can be done for the checkpoint. Eventually, you need to start thinking about parallelizing processes to improve throughput.
Submitting jobs
Submitting jobs to JES involves allocating an internal reader, writing a JCL stream to the internal reader, then closing and freeing the internal reader. As part of this process, JES must examine every card that is submitted, process cards that have information needed at JES input processing time (JECL, JOB card information, and so on), and separate instream data from JCL cards.
In JES3, this processing is done by writing to SPOOL all the cards passed to the internal reader for the job. Once finished, the application must close the internal reader or issue an ENDREQ macro to get the SPOOL data set queued to the JES3 global. The JES3 local does not get involved in the submitting of jobs. So when the address space completes writing the job to SPOOL, a request is queued to the global to process the job. The task in the submitting address space will wait for input processing to complete before returning from the ENDREQ or CLOSE that was issued. To perform input processing for the job, the global reads the cards written to the internal reader data set and process the JCL stream. Each stream (internal reader submission) is processed sequentially, however, multiple streams can be processed by separate threads (DSPs) in the JES3 main task. In this scheme, adding more DSPs that can process internal reader data streams (via the OPTIONS INTRDR= keyword) can improve throughput but only to the point that the single main task TCB can interleave I/Os that read and write various data. Parallelizing internal reader submission (using 20 internal readers each submitting one job versus one internal reader submitting 20 jobs) only helps if there are enough DSPs to process the work.
JES2 performs input processing in a completely different way. In JES2, the bulk of the input processing work occurs in the address space that allocated the internal reader. However, the JES2 main task must place the job in the job queue at the start of the job submission (when the JOB card is detected) and move the job to its next queue at the end the job. These processes must access the checkpoint and are subject to checkpoint delays. Due to the nature of the main task processing, a single checkpoint update (delay) can process one or 100 or more jobs. So by submitting jobs using multiple internal readers, you can greatly increase the number of jobs per minute that can be submitted. One hundred internal readers can submit 100 jobs (one job per internal reader) in the same time one internal reader takes to submit one job. In JES2’s case, there are no resources that can make input processing for a single internal reader go faster, but by parallelize the processing (using multiple internal readers), you can get a significant throughput improvement.
Many job scheduler products have options to increase the number of internal readers they use to submit jobs. Though these options might not have been needed in JES3, use them when you run JES2.
13.2.2 Setting limits and counts
Limits are a way to manage resources in a system. Limits can be used to manage a constrained resource like memory, or they can be used to limit the effect of runaway processes. Counts control the number of processors available to accomplish a function. These processes are consumers of resources. Some processes can be enhanced just by adding resources. Other processes can be enhanced by adding more processors. Still others need both more resources and more processors.
There are a number of ways to manage constrained resources. Applications could choose to not have any external limit and react when a resource is no longer available. Alternatively, an application could set a hard limit that causes processing to wait or fail when the limit is reached. Or the application could include code that cleverly manages limits.
Understanding what limits exist and how to properly manage those limits can improve throughput and prevent outages.
JES2 processing generally either has no limits or a hard limit. No limit is something that is only limited by the architecture (such as a data space can only be 2 gigabytes in size). A hard limit is something specified via a JES2 parameter (such as the number of data buffers in JES2). Hard limits can be changed but require someone to take an action to do so.
As we will see, most limits in JES2 can be displayed using commands and products such as SDSF. However, no matter how many ways an application notifies users when limits are being reached, it is no good if no one is looking or they are looking in the wrong place. JES2 has a number of ways to assist installations in manage resource limits. Understanding this will ensure that JES2 continues to operate smoothly.
Monitoring JES2 resources
The resource limits in JES2 are monitored by a process in JES2 called the resource monitor. The monitor is looking for resources whose use has reached an installation-specified limit. Every resource has an installation-specifiable warning level used by the resource monitor to determine at what point a HASP050 message will be issued. The warning level is specified as a percentage on WARN= keywords on various JES2 statements. The default is always WARN=80 (start issuing warning messages when the resource usage reaches 80%).
When a resource usage reaches the warning level, a highlighted HASP050 message is issued. The HASP050 message contains the current utilization percentage. For example, a shortage of SPOOL space (Track GroupS – TGS) message would look like this:
$HASP050 JES2 RESOURCE SHORTAGE OF TGS - 93% UTILIZATION REACHED
Every 30 seconds the current utilization of any resource that whose usage has exceeded the threshold is examined. If the utilization has dropped below the warning level, the message is deleted (DOMed). If the percentage has changed but is still above the threshold, the old message is deleted (DOMed) and a new message issued with the new percentage. If the utilization is nearly 100%, the old message is deleted and a new message will be issued every 30 seconds. This ensures that the installation is reminded that the resource has been exhausted.
If this is a MAS-scope resource (such as SPOOL space), the first member that discovers the shortage issues the message. Other members defer messaging to that member. This prevents a flood of HASP050 message from each member. However, if resource utilization reaches near 100%, the HASP050 message is issued on every member.
The HASP050 message for some resources includes additional information to help determine a more appropriate resource limit. This occurs when a process requires a resource that has been exhausted. In this case, a longer form of the HASP050 message is used as shown in Example 13-1.
Example 13-1 Example of a long form HASP050 message
*$HASP050 JES2 RESOURCE SHORTAGE OF BUFX - 100% UTILIZATION REACHED
A TOTAL OF 292 BUFX ARE CURRENTLY DEFINED, OF WHICH:
292 (100%) ARE IN USE
3 (1%) ARE BEING WAITED FOR
0 PROCESSORS REQUESTED BUFX BUT DID NOT WAIT
THE LARGEST UNFULFILLED REQUEST WAS FOR 0 BUFX
A MINIMUM OF 295 BUFX IS REQUIRED TO SATISFY CURRENT DEMAND
In this example, the BUFX resource (specified on BUFDEF EXTBUF) has been completely exhausted and at least 295 buffers are needed to satisfy current demand. Use an operator command to increase the limit (to 295 plus at least another 20% so that you will not be above the warning threshold again). The long form of the message is used primarily for buffers used by multiple processes in JES2 (such as data, control block, VTAM, BSC, and header buffers).
Many installations automate on the HASP050 message and based on the resource, trigger a text message or send an email to someone responsible for the system. However, in many installations, the HASP050 message has become so frequent that it is ignored. This typically happens when the warning level for that resource is set too low and the system is normally operating above the warning level for some resources. As a result, the operations staff learns to ignore that HASP050 message because of the frequency of it, and end up ignoring all HASP050 messages.
Because of this, you must scan SYSLOGs for instances of HASP050 messages and take the appropriate action to address the cause of the message. You can either raise the resource limit, raise the warning level, or remove monitoring of the resource (by setting the warning level to 0). This ensures that you only see the HASP050 message when there is a problem and operations can be told to take action whenever they see the HASP050 message.
JES2 resources usage history
One of the functions of the JES2 health monitor is to maintain a history of the utilization of the major JES2 resources. The history is maintained in memory and encompasses the life of this JES2 subsystem. There are two ways to retrieve this information. The JES2 command $JDHISTORY will display this on the console as shown in Example 13-2.
Example 13-2 Display of the history of JES2 resource usage
$HASP9130 D HISTORY 866
$HASP9131 JES2 BERT USAGE HISTORY
DATE TIME LIMIT USAGE LOW HIGH AVERAGE
-------- -------- --------- --------- --------- --------- ---------
2012.250 11:00:01 650 539 389 539 435
2012.250 10:27:14 650 389 129 490 310
$HASP9131 JES2 BSCB USAGE HISTORY
DATE TIME LIMIT USAGE LOW HIGH AVERAGE
-------- -------- --------- --------- --------- --------- ---------
2012.250 11:00:01 42 0 0 0 0
2012.250 10:27:14 42 0 0 0 0
$HASP9131 JES2 BUFX USAGE HISTORY
DATE TIME LIMIT USAGE LOW HIGH AVERAGE
-------- -------- --------- --------- --------- --------- ---------
2012.250 11:00:01 300 290 0 298 268
2012.250 10:27:14 292 0 0 22 2
$HASP9131 JES2 CKVR USAGE HISTORY
DATE TIME LIMIT USAGE LOW HIGH AVERAGE
-------- -------- --------- --------- --------- --------- ---------
2012.250 11:00:01 2 0 0 0 0
2012.250 10:27:14 2 0 0 0 0
$HASP9131 JES2 CMBS USAGE HISTORY
DATE TIME LIMIT USAGE LOW HIGH AVERAGE
-------- -------- --------- --------- --------- --------- ---------
2012.250 11:00:01 201 0 0 0 0
2012.250 10:27:14 201 0 0 0 0
$HASP9131 JES2 CMDS USAGE HISTORY
DATE TIME LIMIT USAGE LOW HIGH AVERAGE
-------- -------- --------- --------- --------- --------- ---------
2012.250 11:00:01 200 0 0 0 0
2012.250 10:27:14 200 0 0 0 0
$HASP9131 JES2 ICES USAGE HISTORY
DATE TIME LIMIT USAGE LOW HIGH AVERAGE
-------- -------- --------- --------- --------- --------- ---------
2012.250 11:00:01 100 0 0 0 0
2012.250 10:27:14 100 0 0 0 0
$HASP9131 JES2 JNUM USAGE HISTORY
DATE TIME LIMIT USAGE LOW HIGH AVERAGE
-------- -------- --------- --------- --------- --------- ---------
2012.250 11:00:01 9999 212 111 212 142
2012.250 10:27:14 9999 111 0 211 104
$HASP9131 JES2 JOES USAGE HISTORY
DATE TIME LIMIT USAGE LOW HIGH AVERAGE
-------- -------- --------- --------- --------- --------- ---------
2012.250 11:00:01 200 57 6 57 21
2012.250 10:27:14 200 6 0 104 46
$HASP9131 JES2 JQES USAGE HISTORY
DATE TIME LIMIT USAGE LOW HIGH AVERAGE
-------- -------- --------- --------- --------- --------- ---------
2012.250 11:00:01 500 212 111 212 142
2012.250 10:27:14 500 111 0 211 104
$HASP9131 JES2 LBUF USAGE HISTORY
DATE TIME LIMIT USAGE LOW HIGH AVERAGE
-------- -------- --------- --------- --------- --------- ---------
2012.250 11:00:01 166 0 0 1 1
2012.250 10:27:14 166 0 0 1 0
$HASP9131 JES2 NHBS USAGE HISTORY
DATE TIME LIMIT USAGE LOW HIGH AVERAGE
-------- -------- --------- --------- --------- --------- ---------
2012.250 11:00:01 23 0 0 0 0
2012.250 10:27:14 23 0 0 0 0
$HASP9131 JES2 SMFB USAGE HISTORY
DATE TIME LIMIT USAGE LOW HIGH AVERAGE
-------- -------- --------- --------- --------- --------- ---------
2012.250 11:00:01 52 0 0 0 0
2012.250 10:27:14 52 0 0 0 0
$HASP9131 JES2 TBUF USAGE HISTORY
DATE TIME LIMIT USAGE LOW HIGH AVERAGE
-------- -------- --------- --------- --------- --------- ---------
2012.250 11:00:01 104 0 0 0 0
2012.250 10:27:14 104 0 0 0 0
$HASP9131 JES2 TGS USAGE HISTORY
DATE TIME LIMIT USAGE LOW HIGH AVERAGE
-------- -------- --------- --------- --------- --------- ---------
2012.250 11:00:01 525 484 230 484 309
2012.250 10:27:14 525 230 21 525 254
$HASP9131 JES2 TTAB USAGE HISTORY
DATE TIME LIMIT USAGE LOW HIGH AVERAGE
-------- -------- --------- --------- --------- --------- ---------
2012.250 11:00:01 3 0 0 0 0
2012.250 10:27:14 3 0 0 0 0
$HASP9131 JES2 VTMB USAGE HISTORY
DATE TIME LIMIT USAGE LOW HIGH AVERAGE
-------- -------- --------- --------- --------- --------- ---------
2012.250 11:00:01 24 0 0 0 0
2012.250 10:27:14 24 0 0 0 0
$HASP9132 MAIN TASK SAMPLING PERCENT HISTORY
DATE TIME COUNT ACTIVE W-DISP IDLE WAIT L-LOCK N-DISP PAGING
-------- ----- ------ ------ ------ ------ ------ ------ ------ ------
2012.250 11:00 46885 0.06 0.00 99.93 0.00 0.00 0.00 0.00
2012.250 10:27 36608 1.84 0.00 97.91 0.24 0.00 0.00 0.00
$HASP9133 JES2 ERROR HISTORY
DATE TIME MAIN DISTERR CBIO SUBTASK NODUMP OTHER
-------- -------- -------- -------- -------- -------- -------- --------
2012.250 11:00:01 0 0 0 0 0 0
2012.250 10:27:14 0 0 0 0 0 0
$HASP9134 JES2 <16M USER STORAGE USE HISTORY (PAGES)
DATE TIME REGION USAGE LOW HIGH AVERAGE
-------- -------- ------- ------- ------- ------- -------
2012.250 11:00:01 1,274 357 341 357 356
2012.250 10:27:14 1,274 341 299 341 340
$HASP9134 JES2 <16M SYSTEM STORAGE USE HISTORY (PAGES)
DATE TIME REGION USAGE LOW HIGH AVERAGE
-------- -------- ------- ------- ------- ------- -------
2012.250 11:00:01 1,274 88 88 90 88
2012.250 10:27:14 1,274 88 61 90 86
$HASP9134 JES2 >16M USER STORAGE USE HISTORY (PAGES)
DATE TIME REGION USAGE LOW HIGH AVERAGE
-------- -------- ------- ------- ------- ------- -------
2012.250 11:00:01 476,928 163,550 163,450 163,551 163,541
2012.250 10:27:14 476,928 163,450 162,984 163,452 163,432
$HASP9134 JES2 >16M SYSTEM STORAGE USE HISTORY (PAGES)
DATE TIME REGION USAGE LOW HIGH AVERAGE
-------- -------- ------- ------- ------- ------- -------
2012.250 11:00:01 476,928 3,422 3,422 3,464 3,422
2012.250 10:27:14 476,928 3,422 3,339 3,474 3,426
Entries in this display include one line for every time interval. Intervals start at the top of every hour and when the monitor is started (this system was IPLed at about 10:20, thus the first entry). The $JDHISTORY command by default displays all resources and up to 24 hours of history. The operands on the command can alter what is displayed. See z/OS JES2 Commands, SA22-7526 for more information about the command syntax.
The response includes information about other resources that JES2 uses. The sampling history contains information about what the main task is doing at each sample (active, idle, waiting, and so on) and there is also a history of the memory usage in the JES2 address space.
Another way to view the resource usage history is to use the SDSF RM display. By default, it displays only the current interval, but additional history is available. One advantage to the SDSF display is that it allows you to set a new limit for a resource or adjust the warning level via an overtype on the window. Example 13-3 on page 164 contains a sample display.
Example 13-3 Sample SDSF RM display
Display Filter View Print Options Search Help
------------------------------------------------------------------------------------------
SDSF RESOURCE MONITOR DISPLAY SY1 LINE 1-17 (17)
COMMAND INPUT ===> SCROLL ===> CSR
PREFIX=* DEST=(ALL) OWNER=* SYSNAME=SY1
NP RESOURCE SysId Status Limit InUse InUse% Warn% IntAvg IntHigh IntLow OverWarn%
BERT IBM1 WARNING 650 541 83.23 80 464 541 389 49.22
BSCB IBM1 42 0 0.00 80 0 0 0 0.00
BUFX IBM1 WARNING 300 290 96.66 90 274 298 0 93.46
CKVR IBM1 2 0 0.00 80 0 1 0 0.00
CMBS IBM1 201 0 0.00 80 0 0 0 0.00
CMDS IBM1 200 0 0.00 80 0 0 0 0.00
ICES IBM1 100 0 0.00 80 0 0 0 0.00
JNUM IBM1 9999 213 2.13 80 162 213 111 0.00
JOES IBM1 200 57 28.50 80 31 57 6 0.00
JQES IBM1 500 213 42.60 80 162 213 111 0.00
LBUF IBM1 166 0 0.00 80 0 1 0 0.00
NHBS IBM1 23 0 0.00 80 0 0 0 0.00
SMFB IBM1 52 0 0.00 80 0 0 0 0.00
TBUF IBM1 104 0 0.00 0 0 0 0 0.00
TGS IBM1 WARNING 525 487 92.76 80 358 487 230 49.28
TTAB IBM1 3 0 0.00 80 0 0 0 0.00
VTMB IBM1 24 0 0.00 80 0 0 0 0.00
The information displayed contains some additional columns that the $JDHISTORY does not include. The Status column tells you if the current utilization is above the warning level and the OverWarn% column tells you the percent of time in the interval the resource was over the warning level. This percentage is very useful to determine if the shortage is due to a short-term spike in usage or indicative of a longer term trend.
 
Note: The list of resource short names and the commands that control them can be found in the description of the $HASP050 message in z/OS JES2 Messages, SA22-7537.
JES2 resource limits
Limits can be grouped into checkpoint limits (MAS scope limits) and local limits (single-member-scope limits). The key difference is the impact of a shortage and the effort needed to increase or manage the limit.
Checkpoint limits
The JES2 checkpoint, whether stored on a DASD data set or in a CF structure is limited in size. This limit is imposed by the media that the checkpoint is stored in and can only be altered by installation actions. As a result, data structures placed in the checkpoint are also limited in number. These structures are also limited by the internal structure of the data.
Because these resources exist in the checkpoint, any shortage impacts all members of the MAS. To prevent a MAS-wide impact, these resources must be monitored and appropriate notifications made when a shortage exists.
The following is the list of checkpoint-related limits:
JOBDEF JOBNUM This defines the number of jobs that JES2 can store in the checkpoint at any one time. This is similar to the joblim value on the OPTIONS JOBNO JES3 initialization statement. When this value is reached, no new jobs (started tasks, time sharing users, or batch jobs) might enter the system. New jobs will wait in their appropriate input processing routine for existing jobs to be purged.
JOBDEF RANGE This defines the valid range of job numbers that can be assigned on this system. This is similar to the lowest and highest values on the OPTIONS JOBNO JES3 initialization statement. Jobs entering from NJE (including SPOOL reload processing) can be assigned numbers outside this range if JOBDEF RASIGN is set to YES. If all the job numbers in the specified range are in use, new jobs entering the system that require a job number within the range (non-NJE jobs for example) will wait in their appropriate input processing routine for an appropriate job number to be freed. The number of jobs in the range must be at least as large as the number of jobs defined on JOBDEF JOBNUM.
OUTDEF JOENUM Unlike JES3, JES2 groups SYSOUT data sets for processing when a spin data set is created or when a job goes through output processing. When this is done, the group of SYSOUT data sets that will be processed together is said to be placed into a Job Output Element (JOE). For JES2 to process a group of SYSOUT data sets it must be grouped into a JOE. You need to define enough JOEs to hold all the spin SYSOUT data sets and non-spin output data sets that will be ready to process on SPOOL. If there are no JOEs available, spin data sets will be placed in an unspun state and processed when JOEs become available. Jobs that complete will wait, queued for output phase processing. However, jobs will continue to run even though their output cannot be processed. In addition to output not being processed, SPOOL utilization will increase since data sets and jobs are not being purged. A general rule of thumb is to define JOENUM to be at least 2.5 times JOBNUM.
CKPTSPACE BERTNUM Another JES2 unique concept is BERTs. BERTs address the need to both extend the contents of data structures and add new data structures to the JES2 checkpoint. BERTs are generic 64-byte blocks of data that are stored in the JES2 checkpoint. They are used to extend the job and output group control blocks with optional data and they also store information on job class properties, duplicate job names, and WLM service class queues. The number of BERTs associated with an object depends on the data being stored in them. So their usage can change based on the property of the jobs and SYSOUT groups represented in the checkpoint. Because of their broad usage, a shortage of BERTs can create extreme problems for a JES2 system. JES2 takes a number of actions as the number of free BERTs falls including stopping SYSOUT creation (similar to a JOE shortage), stopping batch job input processing, and eventually stopping all job input processing. However, running out of BERTs has caused systems to unexpectedly fail in ways that forced a JES2 cold start.
JES2 will automatically increase the limit for BERTs if JOBNUM or JOENUM is increased and BERTNUM is outside what is considered normal. The checkpoint data set must be large enough for the increased number of elements being requested and the potential increase in the number of BERTs.
 
Warning: Do not allow your MAS to run out of BERTs. If you do encounter a severe BERT shortage, do not shut down JES2 or IPL the system. You must resolve the BERT shortage via operator commands such as increasing the number of BERTs or purging jobs that use BERTs before restarting a JES2 member. Failure to do so might force a JES2 cold start.
To increase the number of BERTs, you might need to decrease some other resource in the checkpoint to free up space in the checkpoint before increasing the number of BERTs.
For more information about BERTs, see “BERT shortage special processing” on page 169.
SPOOLDEF TGSPACE JES2 keeps track of what SPOOL space is available using a bitmap in the JES2 checkpoint. Each bit represents a track group of SPOOL space. The size of the track group bitmap is controlled by the MAX parameter on SPOOLDEF TGSPACE. This is not the amount of total SPOOL space that is available for use by JES2. Rather it is the amount that can be defined to JES2. If this value is too low, JES2 will not be able to start new SPOOL volumes. However, if the amount of defined SPOOL space is exhausted, jobs will stop processing data. Also, since job termination uses a small amount of SPOOL space, jobs often cannot come out of execution if they have run out of SPOOL space and no additional space is available. Unlike JES3, JES2 does not reserve SPOOL space for use by a certain class of jobs.
Non-checkpointed limits
There are a number of limits that deal with how storage is used. These limits prevent a process from consuming all available storage, thus ensuring that some storage will be available for general short term use. If the limit of any of these resources is reached, the process requesting the storage normally waits for elements to be freed. However, in some cases, the limit will cause additional processing that needs the resource to fail.
The following is the list of local (non-checkpoint-related) limits:
TPDEF BSCBUF These are buffers used in BSC communication. They are used for NJE (including NJE over CTCs) and RJE. These buffers are in 24-bit storage (below the 16-megabyte line). If you do not have BSC lines (LINExxxx TYPE=unit_addr), set the maximum for these to 0. If you do have BSC lines, JES2 calculates a default based on the number of lines defined. The default must be used for this parameter unless buffer shortages are reported by the HASP050 message. A shortage of these buffers will prevent BSC connections from being established. Because these buffers are in 24-bit virtual and real storage, it is recommended if you avoid using BSC for NJE or RJE, and that BSC lines are deleted and the number of BSC buffers defaulted (or be set to) 0.
If you dynamically add BSC lines using an operator command, this value must also be increased (by command) to prevent any shortages.
TPDEF SNABUF These are buffers used in SNA communication. They are used for both NJE and RJE. These buffers are in 31-bit storage (above the 16-megabyte line). If you do not have SNA lines (LINExxxx TYPE=SNA), you must set the maximum for these at 0. If you do have SNA lines, JES2 will calculate a default based on the number of lines defined. The default must be used for this parameter unless buffer shortages are reported by the HASP050 message. A shortage of these buffers will prevent SNA connections from being established. If you are not using SNA for NJE or RJE, you can delete the SNA lines and set the number of SNA buffers to zero (or allow it to default to zero). If you dynamically add SNA lines using an operator command, this value must also be increased (by command) to prevent any shortages.
BUFDEF BELOWBUF These are buffers in the JES2 address space used to perform I/O to JES2 data blocks (SYSOUT data sets). There are a number of older functions in JES2 (in particular JES2-managed printers, including RJE) that use 24-bit buffers (below the 16-megabyte line). JES2 will calculate the default number of buffers based on a number of initialization parameters. This is normally sufficient for most installations. However, some installations can benefit from an increased number of these buffers. The HASP050 message provides extended information on what to set this value to if a shortage is encountered. A shortage of these buffers will delay processing of SYSOUT. Over-specifying this parameter can result in storage shortages for other processes, especially those that require 24-bit storage.
BUFDEF EXTBUF These are buffers in the JES2 address space used to perform I/O to various control blocks. Most processes in the JES2 address space use these buffers for I/O. They are in 31-bit virtual and real storage. They also have a small associated data area in 24-bit storage that is used for I/O. JES2 will calculate the default number of buffers based on a number of initialization parameters. This is normally sufficient for most installations. However, some installation can benefit from an increased number of these buffers. The HASP050 message provides extended information on what to set this value to if a shortage is encountered. A shortage of these buffers will delay processing of jobs and SYSOUT. Over-specifying this parameter can result in storage shortages for other processes (especially with the 24-bit area that is associated with these buffers).
 
Note: JES2 uses both 24-bit (BELOWBUF) and 31-bit (EXTBUF) buffers to perform I/O. With any new releases, the distribution of buffers used can change greatly as functions are reworked to use more 31-bit buffers and as work is shifted out of the JES2 main task. If a non-default number of buffers is specified in the JES2 initialization deck, when a new release of JES2 is being implemented, a test must be done to determine the new default JES2 is using, and the buffer counts adjusted accordingly.
CONDEF BUFNUM When JES2 issues messages, it does not issue a WTO out of the JES2 main task since a WTO could do an MVS WAIT. To prevent the wait, WTOs are issued out of a subtask in the JES2 address space. This is similar to what JES3 does with its WTOs. The WTOs are queued to the subtask using the CMB data area. The CMBs are in 31-bit virtual storage. To prevent a runaway process from flooding the system with messages, the number of CMBs is limited by this parameter. JES2 defaults this value to 100. If there is a shortage of CMBs, the requester of the WTO will be suspended ($WAITed) until a CMB is available. This can slow command response processing. Over specifying this value would impact available 31-bit storage. If the HASP050 message reports a shortage, increase this value as needed.
There is the concept of a single emergency CMB for use by the command processor when the pool of CMBs is exhausted. This will allow commands to be processed, though very slowly.
CONDEF CMDNUM When a JES2 command is issued by the operator or received over NJE, a command buffer in ECSA is used to hold the command (31-bit common storage). These commands are queued to the JES2 command processor for processing. Unlike JES3, JES2 has one main processor of all commands (they are not distributed to other processes within JES2 for processing). The exception is JES2 monitor commands, which are processed by the monitor in the monitor address space. There are two instances of the JES2 command processor, but most commands are processed by the main command processor.
If the command processor becomes backed up, new commands can consume command buffers until they are exhausted. At that point, no new JES2 commands can be entered. Increasing the number of command buffers will use additional ECSA. If the HASP050 message indicates a shortage of command buffers, investigate the source and nature of the command to determine why they are not being processed in a timely manner. If necessary, increase the number of command buffers.
CKPTDEF VERSIONS Checkpoint versions allow an address space to access checkpoint data (job and output queues) without having to send a request to the JES2 address space. They provide a stable copy of the checkpoint that the application (either directly or through some other API) can use to satisfy query requests. JES maintains these versions as a background process that runs whether there are requester for versions or not. There is always an available current version and one that is ready to be updated. When an application is assigned a version, JES2 is locked out of updating that particular version (note that one version is assigned to multiple applications if it was current when they requested a version). If the application takes a long time to process the version (and thus keeps it locked), JES2 will create an additional version to satisfy the requirement of a current version and one waiting to be updated. The more applications that request and hold versions, the more concurrent versions are needed. CKPTDEF VERSIONS puts an upper limit on the number of versions that can exist at any time. Allowing more versions uses additional real storage to manage the copies of the checkpoint. The default of 2 is low for most systems that have processes that use tools to query or manage the JES2 work queues. A value of 10 is reasonable for most systems. If there is an HASP050 message warning of a shortage of versions, investigate the users of versions to determine why the applications using them are holding the version so long.
TPDEF SESSIONS Every JES2 SNA session (NJE or RJE) that is established uses a data area called an ICE to represent the LU-LU session. These are in 31-bit storage in the JES2 address space. If there are no data areas available, JES2 will be unable to complete any additional SNA session. In general, you must define at least as many sessions as you have SNA lines (this is the default). However, certain RJE connections can have multiple LUs for a single RJE (a single line). If that is the case in your environment, you must define additional sessions for those RJEs.
NJEDEF HDRBUF SNA and BSC NJE need to process NJE data headers that can be up to 32 KB. Header buffers are used to hold these buffers. They are also used in some cases outside of NJE to hold NJE headers. The header buffers reside in 31 bit-storage in the JES2 address space. JES2 calculates a default for the number of header buffers based on various initialization statements. Unless the HASP050 message indicates a shortage, the default value is normally sufficient.
SMFDEF BUFNUM Like many things in JES2, SMF records cannot be written in the JES2 main task because of concerns for MVS waits. SMF records created by the main task are placed in SMF buffers and queued to a subtask for processing. These buffers are used to queue the SMF records to the subtask. The buffers are in 31-bit storage in the JES2 address space. If the number of buffers is exhausted, processes needing buffers will end up being suspended until there are free buffers. The default number of SMF buffers is a fixed value. Depending on your workload and the ability of the SMF process to record the SMF records, you might need to increase the number of SMF records you have defined. Monitor the HASP050 message to see if the number you have defined is sufficient. Note this is one of the few buffers whose limit cannot be increased via operator command.
TRACEDEF TABLES Trace tables are used when JES2 tracing is active (using the TRACEDEF statement). Tracing can be done in memory only or they can be records to a SYSOUT data set. The more tables that you define, the more data that can be buffered waiting for tables to be written. In general, more smaller tables are better than fewer large tables. If there are no trace tables available when a trace record needs to be written, the trace record is discarded. Generally, IBM service recommends a trace table size and number for traces they request. Trace tables are in 31-bit common storage (ECSA). The size of a table and the number are controlled by the TRACEDEF statement.
BERT shortage special processing
The block extension reuse table (BERT) resource, because of the nature of how it is used, requires some special processing and careful monitoring. BERTs are extensions to multiple control blocks. They are variable in size and have the property that if the extension is not needed for a particular control block, no storage (that is BERT) is needed to back it. But at any time if the data associated with a control block changes and needs to be placed in a BERT, the number of BERTs required increases. BERTs also provide a serialization mechanism known as the BERT lock.
The difficulty occurs when a process needs to acquire a BERT either for additional data or for a lock and there are no BERTs available. That process must wait for a BERT to become available. Unfortunately, this can occur with processes that do not normally wait and often results in major processing becoming unavailable. For example, jobs often cannot be purged if there is a BERT shortage.
A further complication can occur at warm start time. There are processes that require the BERT lock or additional BERTs to complete warm start processing. If they are not available, some aspect of warm start processing might have to be delayed. However, there have been unfortunate cases where the shortage of BERTs at warm start has resulted in warm start failing. Usually these problems have been addressed, but there might still be situations where a warm start cannot complete. Because of problems in the past, customers have had to cold start to resolve cases where a BERT shortage prevented a warm start.
The best practice is to avoid BERT shortages. This can be done by paying particular attention to BERT usage at your installation and monitoring for HASP050 messages that indicate BERT shortages. JES2 also has added two messages specifically for when BERTs are about to run out. The first message, shown in Example 13-4, indicates that the number of BERTs is getting critically low.
Example 13-4 Critical BERT shortage message
$HASP052 JES2 BERT resource shortage is critical --
IMMEDIATE action required
 
DO NOT TERMINATE JES2 OR IPL. Doing so might result in a COLD
start.
 
CURRENT BERTNUM=650, Free BERTs=10
 
Correct BERT shortage by --
- $T CKPTSPACE,BERTNUM=nnnn (increase BERTs)
- $P Jnnnn (purge pre-execution jobs)
Depending on the number of BERTs remaining, certain processes will be suspended to prevent all BERTs being used. The following are additional lines that can be added to the message to indicate what processes have been suspended:
•No batch jobs can be submitted
•JOEs requiring BERTs cannot be added
•No batch jobs, STCs, or TSUs can be submitted
If the number of BERTs drops to the point where certain processes were failed because of the inability to obtain the needed BERTs, the message shown in Example 13-5 is issued.
Example 13-5 Extreme BERT shortage message
$HASP051 EXTREME BERT SHORTAGE DETECTED.
JES2 PROCESSING IS IN A NON-STABLE STATE. RESTART JES2 AS
SOON AS MORE BERTS HAVE BEEN MADE AVAILABLE.
In this case, there might be errors in the data structures on this member or errors in the one of the JES2 work queues.
In the event of a critical BERT shortage, an action must be taken immediately to relieve the shortage. Increasing the number of BERTs via the $T CKPTSPACE command is the simplest action (assuming the checkpoint is large enough to contain additional BERTs). If the checkpoints are not large enough, you can either move to larger checkpoints using the checkpoint reconfiguration dialog or you can reduce the limit on some other checkpointed resource to make room in the checkpoint for additional BERTs.
Another way to address a critical shortage is to reduce the number of BERTs in use. You can use the $D CKPTSPACE,BERTUSE command to determine what processes are using the most BERTs. Example 13-6 shows a sample of the output from that command.
Example 13-6 Output of the $D CKPTSPACE,BERTUSE command
$HASP852 CKPTSPACE
$HASP852 CKPTSPACE CURRENT BERT UTILIZATION
$HASP852 TYPE COUNT CB COUNT
$HASP852 -------- --------- ---------
$HASP852 INTERNAL 14 1
$HASP852 JQE 1,067 827
$HASP852 CAT 114 38
$HASP852 WSCQ 12 3
$HASP852 DJBQ 6 3
$HASP852 JOE 151 151
$HASP852 DAS 0 0
$HASP852 FREE 62,736 0
The command output helps you understand what data areas are using BERTs and if you need to purge jobs (JQEs) or output (JOEs) to free up BERTs. This also can help you understand if there is a runaway process that is consuming BERTs and what it is that it is using BERTs for.
An extreme BERT shortage must be dealt with in the same way. BERTs must be freed to allow the system to return to normal processing. But in the case of an extreme shortage, any member that is issuing the HASP051 message must be hot-started to clean up any residual affect of the shortage.
JES2 processor counts
Just as JES3 has DSPs and FCTs whose counts can be specified, JES2 has PCEs whose counts can be adjusted. Adjusting some of these counts alters the amount of work a particular JES2 address space can perform. However, unlike JES3, which does all its work on the global, JES2 performs work in any JES2 address space that has available processors to do the work. Decide if JES2 work will be spread across all members of the MAS or if some members will do less JES2 work than others (a purely symmetrical arrangement or an asymmetrical arrangement). When you have determined that, setting processor counts can influence what JES2 processing a member does.
The PCEs that an installation can influence the counts of are specified on the PCEDEF initialization statement. They can be set at JES2 initialization and cannot be altered by an operator command. The following counts can be specified on PCEDEF:
CNVTNUM JES3 specifies the number of converters for demand select and batch when defining converter DSPs on the global or in a CI FSS. JES2 does not make any such distinction. Any JES2 converters can process any job type. Converters in JES2 run on all members of a MAS. Which system a job gets converted on is controlled by the system affinity (SYSAFF) of a job. Demand select jobs always have an affinity for the member they were started on or logged in to. Therefore, all JES2 members must have at least one converter.
JES2 has code to deal with waits that might be introduced by JCLLIB data sets. If a data set specified in a JCLLIB cannot be allocated due to ENQ contention (data set is allocated exclusively and cannot be allocated shared by conversion) or if a JCLLIB data set is archived, the converter could potentially wait for the data set. If a number of jobs that need archived or ENQed data sets are submitted, they could tie up most if not all of the JES2 converters. Since JES2 does not distinguish between demand select jobs and batch jobs, if the converters were all tied up, no one could log on to that member and no started tasks could be started.
To prevent this, JES2 does make a distinction between converters. The odd-numbered converters are defined as ones that can wait and the even-numbered ones cannot. This needs to be considered when defining the number of converters on a member. In general, define at least two converters, even if the member is not doing a significant amount of JES2 work. This prevents a single job waiting in converter from preventing any starts or logons on the member.
The main resource used by converters is buffers (EXTBUFs) and TCBs (that consume below the line storage). The default number of converters is 10 and the maximum is 25. Each converter uses at least three buffers (more for jobs with more instream data). If a member is to do minimal JES2 work, define the number of converters down to two. If the member will be processing many jobs, increase the value to the maximum and potentially adjust the number of EXTBUFs up.
PURGENUM JES2 purge processing handles not only the purging of jobs but also the purging of spin data set and output groups that require ENF 58 processing. All these operations tend to be I/O-intensive because JES2 used record 0 at the start of each track group to track ownership of the track group. When a track group is freed, record 0 must be updated to indicate the track is free. Because purge processing is so I/O intensive, it can benefit from additional PCEs. It primarily uses its own buffers to read and write the record 0 buffers but it does use a couple of EXTBUFs to hold various control blocks (two or three per purge processor).
Purge processing is not affected by any affinity rules and can be performed on any member. It is generally looked at as a background task. The exception is when there is a resource shortage, where purge processing becomes the bottleneck to freeing many checkpoint resources. The default number of purge processors is 10 and the maximum is 25. A JES2 member that does little JES2 work can have fewer purge processors. Other members must have the default of more purge processors defined. If jobs or output elements (JOEs) are not being purged in a timely manner, the number of purge processors must be increased.
PSONUM The PSO interface is an older interface used to process requests from the TSO OUTPUT and RECEIVE command and requests from processes that use the SYSOUT external writer interface. These are lightly used interfaces in today’s world where the SYSOUT API (SAPI) is one of the primary ways to access SYSOUT. PSO processors handle requests made from the local member (not from other MAS members).
The default number of purge processors is two and the maximum is 10. Unless you have a particularly old application that uses the PSO interface, the default on all members is generally sufficient. If you do have a large PSO user, increase this number on the member that the application uses and any alternate member the application could use. Additional PSO processors use additional EXTBUFs.
OUTNUM Much like the output services phase in JES3, the JES2 output phase makes SYSOUT created by a job available for processing by printers and application interfaces such as SAPI. It is during the output phase that SYSOUT data sets are grouped into output groups (JOEs). All non-spin output is processed during the output phase. Output processing can occur on any member of the MAS. When a job is added to the output queue, its affinity is set to any member (thus system affinity does not apply to the output phase).
Output processing can be a heavy user of EXTBUFs. All the jobs non-spin output control blocks (PDDB IOTs) must be read into buffers before output processing starts processing a job. When a job completes output processing, the buffers are all written back out.
The default number of output processors is 10 and the maximum is 25. A JES2 member that does little JES2 work can have less output processors defined. Other members must have the default of more output processors defined. If jobs are spending a long time awaiting output, consider raising the number of output processors. However, first ensure that there are no shortages of EXTBUFs that are slowing down output processing. Remember when increasing the number of output processors, you need to watch the EXTBUF usage to ensure that you do not create a shortage.
STACNUM Another, older, interface to access JES data is the TSO STATUS and CANCEL commands. These send requests to JES to inquire about the status of a job or to cancel (and potentially purge) a job. Though the CANCEL interface is still used today by some applications that want to cancel jobs without resorting to an operator command, the STATUS interface has been largely replaced by the extended status interface (except for the actual TSO STATUS command).
These commands both get processed in the JES2 address space by the STAC processors (STAtus/Cancel processors). These processes use very little resources but must make a SAF call to authorize any cancel request. They process all requests for the status and cancel interface issued on the local member. The default number of STAC processors is two and the maximum is 10. Unless you have an old application that uses the status or cancel interface, the default on all members is generally sufficient. If you do have a large user of the status or cancel interface, increase this number on the member that the application uses and any alternate member the application could use.
SPINNUM There are two basic ways SYSOUT data sets are created. The traditional type of data set is processed when a job or step completes execution and is created through output processing. However, applications can choose to create output that is available for processing immediately upon deallocating the SYSOUT data set. These data sets are called spin data sets by JES.
JES2 queues these data sets to the JES2 address space by using an ECSA buffer. The spin process in the JES2 address space will build the appropriate output element (JOE) for the data set and queue it for processing. This process does a number of I/Os for each data set as part of the process of building the output element. All processing is done in the ECSA buffer that was obtained.
Additionally there is processing done by this process when there is an ECSA shortage, a JOE shortage, when SPIN data sets arrive via NJE, or when a system restarts. This is called unspun processing. It is an I/O intensive process that reads in the SPOOL control blocks for spin data sets and looks for data sets that could not be processed. Certain spin processors can do the unspun processing, while others are reserved to just handle ECSA requests.
Spin processors handle ECSA requests from the local member and unspun requests from any member. In general, ECSA requests can be processed by one or two spin processors. Unspun requests can benefit from additional spin processors. The default number of spin processors is three and the maximum is 10. Increasing the number of spin processors can help unspun processing, but that is not a common process. Because of this, consider increasing the number of spin processors on the members of the MAS that do most of the JES2 intensive work and use the default on members that do little other JES2 work.
JES2 subtask counts
Like JES3, JES2 has general-purpose subtasks that can be used to perform functions that must MVS wait. Originally these were developed to deal with SAF security requests, but now they are also used by other processes in JES2. In JES3, the number of these subtasks is managed by JES3. However, JES2 has a parameter to control the number defined. The JES2 SUBTDEF GSUBNUM parameter controls the number of subtasks JES2 creates during initialization. The default is 10 with a maximum of 50. Each subtask creates a TCB in the JES2 address space.
The number of TCBs that can be defined in an address space is limited by the amount of 24-bit storage that is available for z/OS control blocks. This is also influenced by the number of 24-bit buffers that you define. The default is acceptable for most installations. However, if you are a heavy user of security profiles that could cause delays in processing SAF requests, you might need to increase the number of subtasks defined. There is an internal command that tracks the usage of general-purpose subtasks, $D PERFDATA(SUBTSTAT) that, among other things, displays the queue time of general subtask requests (AVG_QUEUE_TIME=). This is displayed in seconds. If the time is more than 0.0001 seconds for the most common requests (for example, ENFISSUE or $RACROUT), consider increasing the number of subtasks.
13.2.3 Configuring work queues
To maximize throughput, one area you need to understand is how work is queued within JES and how JES accesses those work queues. One thing to avoid is long queues that must be run looking for work to process. This causes delays both because of the CPU needed to access the queue and the cache misses if the queue is particularly long. There are two queues that are of most concern, the job queue and the SYSOUT (or JOE) queue.
Job queue considerations
The job queue is divided by phase of processing that the job is in. Installations can only influence the execution queue. All other queues are managed totally by JES2.
The execution queue is processed by initiators selecting jobs to run. There are two types of initiators: JES-managed and WLM-managed. Jobs are placed on the WLM work queue if the jobclass they are in is set to MODE=WLM. The WLM service class queues are ordered by arrival time, with the oldest jobs earlier in the queue. Since this is the intended order of selection, there is little consideration needed when setting up the WLM service queue. Jobs on a WLM work queue are also located on a JES2 jobclass queue. But this jobclass queue only exists to organize the jobs and is not used for selection purposes.
Jobs that are not in WLM-managed job classes are selected by JES-mode initiators. These initiators have a list of job classes that they select from. This is different from JES3 where job classes are associated with job class groups and initiators are then assigned to a job class group. A JES-mode initiator in JES3 will only select jobs from the job class group that it is assigned to.
Since in JES2 initiators can be associated with one or more job classes, one question is what classes will be assigned to the initiator? The list of job classes is an ordered list and selection is done from the first class in the list down to the last looking for eligible work. The more classes that are associated with a job initiator, the more classes it must search looking for work. So is it better to have 10 initiators with 10 classes each, or to have the 10 initiators with one class each?
It depends on how you want to spend your resources. Idle initiators consume an address space and associated storage and must be examined whenever work is added to the awaiting-execution queue. When work becomes available on any execution queue, all idle initiators search for work that might have been added. There is no communication of what queue the work became available on. This means that all idle initiators will scan for pending work. Many idle initiators result in all their related job queues being scanned for available work.
There is some performance code for this case. If a class is found to not have any work for one initiator, it is assumed to not have work for any initiator on this member. So only one initiator needs to scan a class looking for work each time initiators are scanned. This implies that one or 100 idle class A initiators will consume similar amounts of CPU when looking for work.
Any exit 14 or 49 code that makes a distinction between which initiators can select a job based on some property other than class negates this performance benefit of this feature. For example, if exit 49 were to prevent odd-numbered jobs from running on even-numbered initiators (for whatever reason), a job that cannot be selected on one class A initiator might be selectable on another. This requires each initiator to scan every job queue associated with it every time.
The advantage of having just one class per initiator is that though there are fewer per class, they are reserved and waiting for that class.
Alternatively, if all 10 classes are associated with all 10 initiators, the only time an initiator will go idle is when all 10 classes have no work. But every time an initiator finishes its work, it needs to scan all the associated class queues looking for another job. This scan does not optimize the processing since the scan occurs when the initiator is idle and it cannot affect other initiators. However, in this case, there will be fewer idle initiators and less scans when work is added to the execution work queue. It also might be possible to get away with fewer initiators.
In general, even with the performance benefits that occur when idle initiators use the same classes, the processing needed to scan all the class queues when any job is added to an execution queue is generally greater than the effect of scanning all classes when an initiator goes idle. So loading initiators with job classes will perform better than having idle initiators waiting for a particular job class. There is a risk with this that an initiator will not be available when new work starts arriving for a job class.
But assuming one of the criteria you use to assign job classes is by expected execution time, you can minimize the wait time by limiting the number of long running job class jobs that can be active at one time. This can be set at the job class level with the XEQCOUNT and XEQMEMBER keywords. These work similar to the TDEPTH and MDEPTH parameter on the CLASS statement in JES3.
SYSOUT queue considerations
Unlike job queues, it is much more important to configure your SYSOUT queues properly as opposed to your job classes. This is because there are generally many different types of processes and devices selected during SYSOUT and far more characteristics to choose from. Also, with jobs, most jobs in an installation are awaiting hardcopy (output in JES3) processing; there are relatively few jobs in the execution queues. However, all SYSOUT is eligible to be processed on one queue or another.
In JES2, there are 36 SYSOUT classes (A-Z and 0-9). Each SYSOUT class has three queue heads that represent the type of destination that the SYSOUT is routed to (these are described in “SYSOUT destination types” on page 176). There is also a separate queue for held SYSOUT and one for SYSOUT destined for another NJE node.
Within each of the class queues (A-Z and 0-9), output elements (JOEs) are queued by route code (destination) and within destination, by priority. JOEs with the same destination and priority are queued first-in first-out (FIFO). The network queue is arranged in priority order with JOEs that have the same priority arranged FIFO. The held JOE queue is simply a last-in first-out (LIFO) queue (stack) of JOEs that are held.
Remember in JES2, the held queue is used for all output groups that are “not ready” to be processed. Output groups on the hold queue can be associated with TSO or external writers as is the case for JES3.
Properly balancing the use of these queues can have a large impact on SYSOUT selection performance. Additionally, using selection criteria that line up with the queuing methods can also lead to better throughput.
SYSOUT destination types
JES2 has three types of non-NJE SYSOUT destination: local, remote, and user. These represent the three queue heads associated with each SYSOUT class. JES2 stores the destination (or route code) of a SYSOUT data set as a 2-byte node number, a 2-byte binary destination, and an 8-byte user route code.
Remote destinations traditionally routed output to an RJE workstation. Internally they are stored as the 2-byte local node number, a 2-byte remote node number, and a blank user route code. These destinations are normally set to Rnnnnn where nnnnn is the remote number.
There are two approaches of local destinations. There is the standard LOCAL or ANYLOCAL destination where the 2-byte node number is set to the local node, the binary destination is 0, and the 8-character user route code is blank. The second approach of local destination is called special local. This is where the 2-byte node number is set to 0, the 2-byte binary destination is set to a special local route code, and the user portion is blank. This is generally specified as Unnnnn where nnnnn is an arbitrary number assigned to the destination.
The user destination is any output that is routed to an 8-character destination that is not represented by a local or remote destination. It has the 2-byte local node number, a zero binary destination, and an 8-character user destination.
Any printer or writer can be set up to select from any of these destination types. A local printer can be set to select output with a remote destination. SAPI can select output routed to a special local route code.
By understanding how JES2 separates output on queues, you can spread SYSOUT across multiple queues and avoid performance problems associated with long SYSOUT work queues.
Using DESTIDs to route output
DESTIDs in JES2 are a way to assign a name to a destination. They are in effect an aliasing function that allows installations to rearrange how SYSOUT is managed without having to alter JCL.
Using DESTIDs, JCL and application writers do not need to know what the underlying route code is when directing output. For example, assume you want a printer to select route code U12 (special local 12) output. Applications can code DEST=U12. There are two problems with this approach. The first is that applications need to be aware of the U12 and associate it with a particular printer. Second, if you decide to consolidate the printers selecting U11 and U12, you need to set up the new printer to select multiple route codes. This can be done but is less efficient. Also, a printer can only select up to four route codes.
Using DESTIDs, you can assign a name to a route code. Assume you want to assign the name FRANK to route code U12. This is for the printer in Frank’s office. You would then define a DESTID of:
DESTID(FRANK) DEST=U12
This statement works in two directions. Any JCL or operator command (or other external) that references FRANK will internally be converted to U12. It will also change all displays of U12 to instead display FRANK. In effect, the U12 internal route code has been replaced with the destination named FRANK.
Now assume that after looking at how output is being processed, you decide to move output for Frank’s printer to a remote queue instead of the local queue (to improve some throughput issues). You can alter the DESTID for FRANK to a remote destination:
DESTID(FRANK) DEST=R12
This defines the FRANK DESTID to now be a remote route code and places it on the remote queue instead of the local queue. Note that no JCL needs to be changed; all commands that used FRANK are not affected, printer applications are not impacted, and so on. If there is existing output that might have been routed to the old U12 destination, you would need to issue the command:
$TOJ(*),/DEST=U12,DEST=FRANK
This might need to be issued a couple of times in case there are any jobs actively creating output for the old designation for FRANK.
You can also have two DESTIDs resolve to the same route code. So if Frank shares an office with John, you might want to add a DESTID for JOHN that goes to the same route code as the updated FRANK:
DESTID(JOHN) DEST=R12
But this introduces a problem. If you display something with a route code of R12, should it display FRANK or JOHN? The default processing is to display neither and instead display R12. This is not ideal. To address this issue, there is another keyword on DESTID to indicate which is the primary destination (and therefore, which are the alternate). Assuming we want the printer to display as FRANK even though it can also be referred to as JOHN you would specify:
DESTID(FRANK) DEST=R12,PRIMARY=YES
Now anything that references a destination of JOHN or FRANK or R12 will display as FRANK.
DESTIDS can also be used with the 8-character user route code. So you can set a device to select a route code of POK that has no DESTID. If you want NY to also be routed to POK, you can define a DESTID:
DESTID(NY) DEST=POK
In this case, assuming there is no DESTID for POK, all output with a route code of POK will display as NY (since the DESTID takes precedence for displays). If you want the destination to display as POK, define the DESTID:
DESTID(POK) DEST=POK,PRIMARY=YES
Effectively, this just changes how output routed to the 8-character user route code is displayed.
There are many other features and capabilities using DESTIDs including the ability to alter processing that occurs with NJE routed output and output received via NJE. There is also a DESTDEF statement that allows a more global control over how destinations are handled. A complete description of how DESTIDs work can be found in “Output Routing” section of the z/OS JES2 Initialization and Tuning Guide, SA22-7532.
Configuring SYSOUT queues for performance
Now that we understand what SYSOUT queues exist (class queues, hold queue and network queue), and how we can get output placed on various queues, we can begin to look at how to configure the MAS to get the best performance.
The first step is to understand the producers and consumers of SYSOUT. Is the consumer a series of real printers? How many devices do we have? Can they all be thought of in the same way but in separate locations? Do you have an archive product that manages SYSOUT groups? How does it select what to archive? Is most output sent over NJE to other nodes for processing?
On the producer side, does output get created in bursts or is it a steady pace? Does it produce output faster than the consumers can process it and you catch up during a quiet time? What kind of controls are there to alter the SYSOUT class or destination created by applications (what is possible to change easily)? Are the producers associated with some property of the SYSOUT class when they create their output, or is the class just there to help manage the output?
Looking at the consumers, let us look at how they select output. Both JESes have a rich set of criteria to select output. There are logical factors like class, external writer name, PRMODE, limits, and jobname. There are physical factors like destination (where is it printing), forms, and burst. But when defining how your system manages SYSOUT, you need to consider how the output is organized.
Let us look at a simple example. Assume that you have a printer that will process all output associated with an external writer. In JES2, output routed to an external writer can be processed by any printer, by SAPI, or by an actual external writer. To simplify the example, assume this is a printer selecting the output (since we can then look at the printer parameters being used). So in setting this up, the simplest thing to do would be to place only the external writer specification in the work selection list and set the writer value to what you want to process. Here is how you would specify that:
PRT1 WS=(W),WRITER=TEST
From a configuration standpoint, this is easy to do. But remember how JES2 arranges output. Based on this definition, you are not selecting on a class or a destination. So to find an element to process, the printer needs to look at all the class queues and the destination queues off the class queues. The result is that this simple selection of criteria will scan all the output elements on the system looking for something to print (except output elements queued for NJE or on the hold queue).
But it is worse than that. JES2 has code that can do a quick scan of output queues looking for where on the existing chains an output group might be found. But these are also dependent on the device selecting on class and destination. Since our simple setup did not select on class or destination, there was no optimization possible. So this simple setup is about as bad as it can get from an overhead perspective.
The HELD queue is not scanned in this case since real printers can only select ready output. But had this been a SAPI printer, there would be the capability of scanning the held queue too.
So let us try to improve this a bit. We probably do not want to select output routed to a specific destination. So we can restrict this to locally destined output. So our improved printer specification is:
PRT1 WS=(W,R),WRITER=TEST,ROUTECDE=LOCAL
Now you have reduced the number of output groups scanned significantly. JES2 will only examine the local queues of each output class and stop scanning that queue when it finds an output group that is not LOCAL. This simple but logical change can significantly reduce the overhead of scanning for work.
However, this points out a potential conflict that can occur when creating output. If you create a SYSOUT data set that has a writer of TEST specified and a routecode of R15 (remote 15), where should it be processed? Printers that print route code R15 output generally specify that the writer value be blank. So output with conflicting routing information might never be picked up for processing.
We can further improve this processing if we decide that output for writers will be sent to a specific class. We could then add the class to the selection criteria and further reduce the number of output groups scanned. Your printer definitions would then look something like this:
PRT1 WS=(W,Q,R),WRITER=TEST,Q=T,ROUTECDE=LOCAL
Now the printer will only look at class T output routed to destination LOCAL for output groups. Since class and destination are how JES2 arranges SYSOUT, this greatly reduces the overhead in searching for work for a printer. Since JES3 SYSOUT destined for an external writer already has to go to an external writer SYSOUT class, using a specific class for this purpose will be simple when moving from JES3 to JES2.
Routing using writer name is one example, but let us look at sending output to a printer using destination. In JES3, printers have names associated with them and SYSOUT is sent to the printer using the printer name. JES2 is similar in that output is sent to a destination, but the destination is not a printer name. It is one of the three types of destinations described in “SYSOUT destination types” on page 176. Generally, applications will not use the Uxxxxx or Rxxxxx formats of a destination to route output. Instead, the installation will define a DESTID that maps a meaningful name to a Uxxxxx or Rxxxxx route code.
The simplest destination type to use is the 8-character user route code. This is an arbitrary 8-character value that the creator and consumer of the output agree to use to identify the output to process. For example, an application can be created that runs a series of jobs and then runs a final job that uses SAPI to collect all the output. This could use the application name as the destination for the output and the selection criteria for the SAPI application.
This is fine for a simple application, but not for a major print processing application. For this type of application, we need to understand the nature of the output being created and the capability of the application to control what is selected.
If the application can only access a single SYSOUT class, assign it to a dedicated class (do not use that class for anything other than that application). If the application can use multiple classes, spread the SYSOUT over a logical mix of classes.
If the application is selecting on route codes, be sure to spread the route codes across the three queues that can be used for a class. If there are route codes that traditionally build large queues (for example, they never actually get processed but age off the system), assign those to higher-collating route codes (U9999 for example). Place the smaller, more transient route codes in the lower route codes. This avoids having to step over a large queue of output to get to frequently used output.
One mistake that often occurs is an application that by default creates the majority of its output using CLASS A with a destination of LOCAL. Then, the output never prints because there are no local printers left at the installation. Eventually the output gets deleted off the system because of age. By itself this is not a problem. But it can become a problem when installations decide that the convenience printers will also use CLASS A and a special local route code (using the form Uxxxx). As the class A local work starts to back up, the overhead to select the output for the convenience printers starts to increase.
To prevent this, address the convenience printers by using Rxxxx route codes or send them to other classes (always using DESTIDs to do the translation). Changing how the JCL DEST is handled is easy with DESTIDs, so periodically look at that.
Another mistake to be avoided is setting up applications that specify held output (OUTDISPs of HOLD or LEAVE). These output groups are placed on the hold queue in JES2 with no particular order to them. They were intended to be accessed through the job they are associated with (such as when using the TSO OUTPUT command). The SYSOUT API (SAPI) does support selecting held output. This is functionally compatible with what the PSO interface can do. However it has a much higher overhead than selecting ready output groups.
The period when you are migrating from JES3 to JES2 is a good time to set up new rules. This is when you can easily get work spread across multiple SYSOUT classes. The goal in all this is to use as many of the 108 SYSOUT queues JES2 provides as practical.
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset