How JES2 works
When making a migration decision from JES3, you will want to know what JES2 provides. This chapter is intended for people who are familiar with JES3 and want to understand how JES2 works. It helps you to understand situations where the same function has a different name in JES2 than in JES3.
In many ways, JES2 provides many of the same functions that JES3 does. However, it generally does them in a completely different way. These differences arose over the years, partly because of different functional requirements, but also because of a fundamental difference in philosophy. JES3’s basic philosophy is one of centralized control where one global system controls some number of local systems, freeing up the locals to do what is most important to them: run jobs. JES2 however, is a collection of peer systems that independently process and manage work on the input, execution, and output queues. Spool-related work is done, as much as possible, by the process requesting a function, or by the corresponding JES2 address space.
1.1 What is a job entry subsystem (JES)?
The concept of a job entry system originated around 1960. Computers were constantly getting faster, but unit record devices (printers, card readers, and card punches) with their mechanical processes were getting slower relative to processor speeds. When systems performed an input/output (I/O) operation to one of these devices, they would wait and do nothing until the I/O completed. In an effort to improve the utilization of large computer systems, IBM designed a way to use a separate peripheral processor to perform the relatively slow I/O to card readers, printers, and punches. This meant you could run more jobs and make more efficient use of the expensive system.
From this beginning, JES2 and JES3 evolved. The two products were originally designed to address different parts of the mainframe market, but over time their surface functions have been converging.
1.2 JES2 overview
At its most basic level, JES2 provides the same functions as JES3 (or any other batch management system). It manages job and output work queues, implements a SPOOL to temporarily store data (job streams and SYSOUT data sets), and deals with peripheral devices (card readers, card punches, and printer). Jobs are generally submitted to JES from TSO using the SUBMIT command that writes card images to an internal reader. An internal reader is just a logical card reader. Job control language (JCL) is written in 80-character card images with room for sequence numbers in columns 73-80 (a carry-over from the days when you could drop your JCL card deck and had to get it back in the correct sequence).
JES2 also supports input and output using network job entry (NJE) protocols and remote job entry (RJE). These are ways to connect other systems to JES2 either as peers (NJE) or as workstations (RJE).
JES2 has an initialization deck that defines how JES2 will operate, operator commands that allow users and operators to interact with JES2, and programming interfaces such as the subsystem interface (SSI) that allow JES2 to communicate with other processes.
Up to this point, the same description applies to both JES2 and JES3. So what is different about JES2? One of the big differences is philosophy (centralized control versus distributed). Another is the scope of the function provided. The two have a different command and initialization deck philosophy. It is these subtle items that make a world of difference in how the two JESs do the same basic processing so differently.
The purpose of this chapter is to help you understand how JES2 does things and, where applicable, contrast that with how JES3 does things. If you want to learn even more about JES2, refer to z/OS JES2 Introduction, SA22-75351 and z/OS JES2 Initialization and Tuning Guide, SA22-75322.
1.2.1 JES2 environment
In JES2, the work queue resides in a construct called the checkpoint. The checkpoint can be placed in a DASD data set or in a Coupling Facility structure. A JES2 member accesses this checkpoint and adds work to, or takes work from, one of the work queues. JES2 can have
1 - 32 members accessing the checkpoint. All members must also have access to a set of data sets that contain the SPOOL data. A MAS (also referred to as a JES2 JESplex) is therefore the collection of JES2 members that access the same checkpoint and SPOOL data sets. All members in a JES2 MAS must be in the same z/OS sysplex. You can have multiple JESplexes in a single sysplex, including JES2 and JES3 JESPLEXes in the same sysplex. See Figure 1-1.
1 - 32 members accessing the checkpoint. All members must also have access to a set of data sets that contain the SPOOL data. A MAS (also referred to as a JES2 JESplex) is therefore the collection of JES2 members that access the same checkpoint and SPOOL data sets. All members in a JES2 MAS must be in the same z/OS sysplex. You can have multiple JESplexes in a single sysplex, including JES2 and JES3 JESPLEXes in the same sysplex. See Figure 1-1.
Figure 1-1 JES2 systems sharing checkpoint data sets
JES2 supports a concept called poly-JES. This is also known as secondary subsystems. In z/OS, when subsystems are defined to the SSI3 using the IEFSSNxx Parmlib member, one subsystem is defined as the primary job entry subsystem and others are considered secondary subsystems. z/OS, by default, routes requests (for example START commands) to the primary subsystem. However, z/OS supports any number of secondary subsystems to be defined. JES2 supports being defined as a primary or a secondary subsystem. This implies that you can define a primary and a secondary JES2 on the same z/OS image. See Figure 1-2.
Figure 1-2 JES2 systems with poly-JES
You can also define a JES3 as the primary and JES2 as a secondary subsystem (however running JES2 and JES3 on the same z/OS image is not currently supported, though there are no known problems). If JES3 is running on a system, it must be the primary job entry subsystem. You cannot run JES2 as the primary subsystem and JES3 as a secondary subsystem.
A secondary JES2 can be in the same MAS as the primary or in a separate MAS. There are multiple reasons to run a secondary JES2. The first is for testing purposes. Some customers test user exits or JES2 service in a secondary subsystem that is in a separate MAS from the primary but with access to the production environment. Other customers use secondary JES2s with a separate MAS to move work off the production MAS due to capacity issues. One case of this was a customer that had a flood of SYSOUT data sets created in their production MAS. The number of SYSOUT data sets was overwhelming their production archiver, so they used NJE to transfer most of the SYSOUT data sets to a secondary MAS using NJE and then brought them back to the production MAS a few thousand at a time to be archived.
A secondary JES2 in the same MAS as the primary has also been used to isolate functions (for reliability or performance). One example of this is placing SNA networking connections in a secondary JES2. Networking is CPU intensive for the JES2 main task but does not require access to the JES2 checkpoint. Moving it to a secondary JES2 allows frees up resources in the primary JES2 for other work. Of course, moving to NJE over TCP/IP achieves the same thing, but by doing all the work in a separate address space, without the need for a secondary JES2.
Each JES2 address space operates in a generally autonomous way. Most communication between JES2 address spaces traditionally occurs through the checkpoint data set. More recently, JESXCF is used to broadcast notifications and request specific functions from a member, and multi-system Event Notification Facilities (ENFs) are used to communicate job and outputs transitions. But the checkpoint is still the primary way JES2 members communicate.
All members in JES2 are capable of performing all JES2 functions. There are some functions for which a single member acts as coordinator (such as a checkpoint reconfiguration or SPOOL configuration manipulation) and others that only get performed on one member (such as priority aging). In these cases, if the member coordinating or performing the function fails, the processing moves to another member automatically.
Functions like RJE and NJE can be performed on any member of a MAS (including secondary JES2s). From an NJE point of view, the MAS is a single NJE node (just like JES3). However, a single z/OS image can have multiple JES2s in different MASes. So a single z/OS image can have multiple NJE node names (one for each MAS). NJE can be done between any two JES2s that are in separate MASes, including between a primary JES2 and a secondary one, or between two secondary JES2s. Of course, NJE can also be done between JES2 and JES3 or any other operating system that supports the NJE protocol.
The implementation of NJE over TCP/IP in JES2 and JES3 is implemented by a common set of services in using a separate NETSERV address space. Though the externals that manage NJE are different between JES2 and JES3, the underlying code is the same.
Figure 1-3 Address spaces associated with JES2
A JES2 member is composed of a number of address spaces and data areas. The JES2 address space is the started task that is created by the S JES2 command. As part of the initialization of JES2, two additional address spaces are created, JES2AUX and JES2MON (note that “JES2” is replaced with the actual subsystem name specified on the start command). The JES2AUX address space is the owner of various resources associated with JES2. It also runs some code to process requests from other MAS members. It cannot be canceled or forced. The JES2MON address space monitors the health of JES2 and ensures that it is processing normally.
JES2 uses XCF messaging to communicate between members and to determine member status. Similar to JES3, JES2 uses JESXCF to manage this communication. The JES2 messages consist of smaller messages between all members of the MAS. This differs from JES3 where messages can be larger and are primarily between the global and the local. JES2 also uses messaging to transport unwritten buffers between address spaces for spool data set browse.
A number of other optional address spaces are associated with JES2 based on functions used by JES2. Print processing can be done using the Functional Subsystem Interface (FSI). This interface, similar to the JES3 FSI, is used in JES2 to interact with products such as PSF. JES3 also uses the FSI for other processing, but JES2 only uses it for printing. NJE over TCP/IP is implemented using NETSERV address spaces. Finally, as of z/OS 2.1, JES2 supports invoking both the MVS converter and interpreter during the conversion phase of job processing. If this option is active, the processing is done in a JES2CI address space.
In addition to these address spaces, there is code that implements the JES2 side of the SSI and code that implements the SPOOL access method (known as HAM or HASP Access Method). These interfaces connect running address spaces and z/OS functions with JES2. Even when the JES2 address space terminates (such as when it ABENDs), these interfaces are still active. They continue to function so long as they do not require functions from the JES2 address space. If the requests require services from the JES2 address space (on that member), they will either wait for JES2 to be restarted, or fail the request.
Whether an interface can continue to function without the JES2 address space depends on where the code that implements the function runs. This differs greatly between JES2 and JES3. For example, creating, opening, and writing to a SYSOUT data set does not require any processing in the JES2 address space. However, creating and writing to a SYSOUT data set does require spool space. Each address space has some spool space available from its last allocation of space and JES2 maintains a cache of spool space (called the BLOB) for use by running address space. If there is available space, address spaces can continue to create and write to SYSOUT data. But once that space is exhausted, the address space waits for spool space.
Other functions always require the local JES2 address space to be active. For example, submitting a job, placing a job into execution, taking a job out of execution, processing a start command, or logging on to TSO all require services from the JES2 address space. But it is only the JES2 address space on the member where the request originated that must be active. Loss of a JES2 address space only impacts the work associated with that JES2, not work associated with other systems.
When shutting down JES2, you can either stop just the JES2 address space or try to completely shut down JES2. Completely shutting down JES2 is done using a $P JES2 command with no operands, or with the QUICK operand. The command is only accepted if there are no address spaces connected to JES2 and if there are no active JES2 processes. If a $P JES2 command is successfully processed, not only will the JES2 address space be stopped, but the JES2AUX, JES2MON, and any JES2CI address spaces are stopped. JES2 also will disconnect from the SSI. This returns z/OS to a state similar to what it was had JES2 never been started. This is called a clean JES2 shutdown (or just a $P JES2).
Often a clean shutdown of JES2 is not practical for the primary subsystem due to the number of address spaces that connect to JES2. But a secondary JES2 can more easily be shut down cleanly, allowing more possibilities for how secondaries are restarted.
After a clean shutdown of all JES2 members of a MAS, the systems are in a state where an all-member warm start or even a cold start could be performed. So strictly speaking, a cold start does not require any systems to be IPLed. However, in the case of a primary subsystem, it is often the most practical thing to do.
1.2.2 JES2 PROC JCL
JES2 (and in fact any JES subsystem) runs as a started task under the master subsystem of z/OS. As a started task, it requires a PROC that is accessible to the master subsystem (in other words the PROC must be in a concatenation defined in the IEFPDSI DD in the MSTJCLxx member of Proclib). There are two styles you can use when defining the JES2 PROC. The traditional style has Data Definition (DD) cards pointing to the initialization deck, PROCLIB concatenations, and often a STEPLIB DD. Then there is the minimalist approach, where you only have a PROC and EXEC card. Both are described here. However, you can also use some middle ground between the two approaches.
To STEPLIB or not to STEPLIB
One of the first questions you need to decide is how to package your JES2 load modules. JES2 supports placing all its load modules (and your exit modules) into a single data set. You can then place that data set in a STEPLIB in the JES2 PROC or add it to link list. This is useful if you want to run a secondary JES2 that has a different level of code from the primary JES2. Each JES2 can point to separate data sets that they will use to load the JES2 code.
Alternatively, you can place the load modules whose names start with HASC into LPA and place the remaining modules in the LINKLST concatenation. There is a small advantage to placing the HASC* modules into read-only LPA because that protects them from being overlaid. However, depending on your setup, it might be less error prone to just put all the JES2 modules into a single data set. If that is your choice, placing it in STEPLIB in the JES2 PROC with a parameter on the PROC that allows you to start from an alternate STEPLIB can give you a back-out path in case a bad fix is placed in the library.
You can also use a combination of these two methods by placing the code for the primary JES2 into the LINKLST or LPA, and use a STEPLIB for your secondary JES2.
JES2 initialization statements
JES2 initialization statements can be obtained from either the HASPPARM DD in the JES2 PROC or from the logical PARMLIB concatenation defined to z/OS. The HASPPARM DD points to a sequential data set (it can be a concatenation of sequential data sets or PDS members) that is read to obtain the JES2 initialization statements. The logical PARMLIB is a z/OS facility that allows programs to read initialization statements from a logical concatenation of PARMLIB data sets without having to have a DD statement pointing at the library.
Most installations do not place their JES2 initialization decks in the normal PARMLIB data set, preferring instead to keep them in a separate JES2PARM data set. However, in an effort to simplify your JES2 PROC, you can create a single member in the PARMLIB data set that has a single statement to include additional initialization streams from the JES2PARM data set. This allows a separate data set for the JES2 initialization statements but does not require a HASPPARM DD in the JES2 PROC.
PROCLIB concatenations
JES2 supports multiple PROCLIB concatenations, each of which can be associated with particular job classes (including TSO logons and started tasks) or specified in JCL. Traditionally, these are allocated as DDs in the JES2 PROC. The DD name was generally of the form PROCxx where xx is a 2-digit concatenation number.
However, PROCLIB data sets can now be allocated using the PROCLIB statement in the JES2 initialization deck or by using the $ADD PROCLIB command. The advantage is a simplified JES2 PROC and less chance of JES2 encountering a JCL error when you try to start it. You can also update PROCLIB concatenations using the $T PROCLIB command.
Putting it all together (sample JES2 PROCs)
Given all this, the most basic JES2 PROC would be what is shown in Example 1-1.
Example 1-1 Simplified JES2 PROC
//JES2 PROC
//IEFPROC EXEC PGM=HASJES20,TIME=1440,REGION=0M,PARM=(WARM,NOREQ)
The PROC in Example 1-1 assumes that JES2’s load modules are in LINKLST, the PROCLIBs are defined in the JES2 initialization deck, and the JES2 initialization parameters are contained in the HASJES2 member of the logical PARMLIB. There is not much that can go wrong with this PROC (in terms of JCL errors with the PROC). And there is nothing that ever needs to be changed.
On the other extreme, Example 1-2 contains a JES2 PROC that specifies all the possible parameters that are traditionally in the JES2 PROC.
Example 1-2 Traditional JES2 PROC
//JES2 PROC DSN1=’SYS1.PROCLIB’, * STANDARD PROCLIB. *
// DSN2=’SYS1.PROCLIB2’, * SYSTEM SUPPORT PROCS. *
// DSN3=’SYS1.PROCSPP’, * PROG. PRODUCT PROCS. *
// DSN4=’SYS2.PROCLIB’, * USER PROCLIB. *
// STEPLIB=’SYS1.JES2.SHASLNKE’, * JES2 PGM LIBRARY *
// MBR=HASIPARM * DEFAULT PARMLIB MEMBER *
//* *
//IEFPROC EXEC PGM=HASJES20,TIME=1440,REGION=0M,
// PARM=(WARM,NOREQ) SPECIFY START OPTIONS
//STEPLIB DD DSN=&STEPLIB,DISP=SHR (MUST BE AUTHORIZED)
//* *
//******** DEFAULT PROCEDURE LIBRARIES LIST ***********************
//* *
//PROC00 DD DSN=&DSN1,DISP=SHR
// DD DSN=&DSN2,DISP=SHR
// DD DSN=&DSN3,DISP=SHR
// DD DSN=&DSN4,DISP=SHR
//* *
//******** INDIVIDUAL PROCXX CARDS FOR EACH PROCLIB. ***************
//* *
//PROC01 DD DSN=&DSN1,DISP=SHR SPECIFIC DECLARATIONS
//PROC02 DD DSN=&DSN2,DISP=SHR FOR USE WITH JOBPARM PROC=
//PROC03 DD DSN=&DSN3,DISP=SHR PARAMETERS.
//PROC04 DD DSN=&DSN4,DISP=SHR *
//HASPPARM DD DSN=SYS1.PROCLIB(&MBR),DISP=SHR
//PARMBACK DD DSN=SYS1.JESPARM(&MBR),DISP=SHR ALTERNATE
//HASPLIST DD DDNAME=IEFRDER LISTING
As you can see, the PROC in Example 1-2 has defined all the various types of DDs that can be specified in a JES2 PROC. PARMBACK is an alternate DD name to get the JES2 initialization deck from (this can be selected via PARM= when JES2 is started). HASPLIST is a DD name that JES2 sends a listing of the processed initialization deck to. This is very useful if your JES2 initialization deck uses system symbols and you want to see how they were resolved.
Because of the number of DDs in the traditional JES2 PROC, there are more opportunities for JCL errors that could prevent JES2 from starting. The use of symbols allows you to fix the problem by simply overriding one of the statements. But diagnosing the problem and the correct override can be complicated if JES2 is not active.
The two styles of JES2 PROC can be combined based on your requirements. You can pick some DDs from the traditional JES2 PROC and use them in the simplified JES2 PROC. The two that are the least risky and most useful are the HASPLIST and the STEPLIB DDs.
You can use the HASPLIST DD to get a listing of the initialization statements that were processed with the symbol substitution resolved. If you have never used IEFRDER, it is a DD created by Start processing that picks up any DD keywords specified on the Start command. If you use the traditional JES2 PROC and issue the following command, the HASPLIST DD will be assigned to the specified data set name:
S JES2,DSN=SYS1.JESPARM.LISTING,DISP=OLD
Starting with z/OS 2.1, you can display information about how JES2 was started using the $D INITINFO command. This displays the command used to start JES2, the initialization decks read, and the STEPLIB specified in the JES2 PROC.
Example 1-3 Sample output of the $D INITINFO command
$HASP825 INITINFO --- Command used to start JES2
$HASP825 S JES2,SUB=MSTR
$HASP825 --- HASPPARM data sets read
$HASP825 DSN=SYS1.JESPARM(JES2Z7TS),VOLSER=SYSAQ4,
$HASP825 CARDS=1763,
$HASP825 DSN=SYS1.JESNODE(CONNECTS),VOLSER=SYSAQ4,
$HASP825 CARDS=4041,
$HASP825 DSN=SYS1.JESNODE(NODES),VOLSER=SYSAQ4,
$HASP825 CARDS=4058
$HASP825 --- STEPLIB Concatenation
$HASP825 DSN=SYS1.SHASLNKE,VOLSER=PRIPKB
1.2.3 MAS members
In a JES2 MAS, all members are peers and can perform all JES2 functions. You can choose to run certain functions on one member and not another (for example NJE or printing). Any members can join or leave a MAS at any time without impacting other members. Every MAS is defined as having 32 members even if only one member actually exists. Members do not have to be predefined before initializing. A new member can define itself to a MAS as part of its initialization.
One implication of this processing is there is no such thing as a single system environment in JES2. Even when only one member is active, it is considered a single member MAS. As such, JES2 processing is the same if there is one or multiple members in a MAS.
Each member of a JES2 MAS starts independently, processing the initialization statements read from the initialization deck. The member joins the JESXCF XCF group using group and member information from the initialization deck. This is to inform any other MAS members of its existence and to open a communication path to other members.
At this point, the address space is not yet considered part of the MAS because it has not been assigned a member number. Using the JESXCF group, the member sends a message to other members, asking where the checkpoint data set is located. This is in case the checkpoint had been moved but the initialization deck was not updated to reflect the change. The starting member then obtains the checkpoint data set lock and reads the checkpoint into memory. It is at this point that the JES2 address space assigns itself a member number and joins the MAS. The initialization parameters are validated against the checkpoint and updates made based on the start type.
All start types are processed in the same way until after the checkpoint is initially processed (including cold starts). Only after the checkpoint is initially read does the code start to distinguish the start type. This allows the initializing member to determine the status of other members of the MAS and perform the appropriate processing. On a cold start, if other active members are detected, the cold starting member fails initialization.
There are only two external values specifiable for the JES2 start type: COLD and WARM. A cold start clears the work queues, the checkpoint, and the SPOOL. A warm start is the normal way JES2 is initialized. The various types of warm starts that JES2 performs is based on the environment JES2 discovers after reading the checkpoint data set. If the SSI environment is active from a previous instance of the JES2 address space (the JES2 address space ABENDed), JES2 performs a hot start. Otherwise, the start is based on the status of other members and the state of the work queues. If there are no other JES2s active (no other JES2 address spaces or SSI environments in this MAS), an all-member warm start is performed. If there are other address spaces active, the start is a either a single system warm start or a quick start. The difference is that for a quick start, there are no residual traces of a previous instance of the member in the job queue (such as jobs in the job queue indicating they are active on the member initializing).
1.2.4 SPOOL
The SPOOL is the bulk data repository in JES2. It primarily contains JES-managed data sets that contain job input (instream data) and output (SYSOUT). This includes job-oriented data sets like the JCL, the output of the converter (internal text), and restart information (the job journal). It also contains a number of job-related control blocks that are used to manage the characteristics of a job and the data sets that it owns.
The SPOOL is composed of up to 253 standard z/OS data sets. Each data set can only have a single extent (this is due to the way the data set is managed). In a MAS, JES2 can only access one SPOOL data set per volume. This is because JES2 uses the volume serial number (VOLSER) that the SPOOL data set resides on to manage the SPOOL data set. Because of this, JES2 uses the terms “SPOOL volume” and “SPOOL data set” interchangeably when referring to a SPOOL extent.
The SPOOL configuration is established when JES2 is cold started, but can be altered via operator command. The SPOOL configuration on a warm start must match the configuration that other MAS members are using. The exception is an all-member warm start, where, in certain situations, a missing SPOOL volume can be removed from the configuration.
There are two ways that JES2 determines the SPOOL configuration on a cold start:
•The traditional method uses the SPOOL data set name and prefix (DSNAME= and VOLUME= on the SPOOLDEF initialization statement) to locate all volumes that match the specified prefix and have the required SPOOL data set on it. All data sets found this way are initialized as SPOOL data sets.
•If all your SPOOL data sets and volumes do not conform to the same naming convention, you can use SPOOL initialization statements to define each SPOOL data set you want to place in your configuration. SPOOL initialization statements must be used if the SPOOL prefix specified on the SPOOLDEF initialization statement contains the generic specification characters “*” or “?”.
SPOOL volumes can be added after a cold start by using the $S SPOOL command. The actual data set can already exist on the volume, or it can be dynamically created on the specified volume by the command. The SPOOL configuration is stored in the JES2 checkpoint data set and restored when JES2 warm starts. Because of this, you are not required to update the JES2 initialization data set when a new volume is started. However, it is a good practice to update the initialization deck in case a cold start is ever needed.
You can also use the $P SPOOL operator commands to delete a SPOOL volume from the configuration. JES2 tracks at a job level what SPOOL volumes a job has ever used. Before a SPOOL volume can be deleted from the configuration, all jobs that ever allocated space on the volume need to be purged from the system. This can take a number of IPLs plus waiting for the jobs to age off the system. In many installations that do not IPL frequently, this can takes weeks or months.
Alternatively, there is a function called SPOOL migration that can move data from one volume to another using the $M SPOOL command. It can also merge SPOOL volumes together. This process does not remove the logical volume from the configuration. Rather, it allows you to stop using a physical volume and delete the SPOOL data set from that volume. The logical volume remains until it can be purged (with the same rules as the $P SPOOL command).
It is not required that All SPOOL data sets have the same data set name. However, at the time the volume is defined (either on the SPOOL initialization statement or on a $S SPOOL command) the data set name must match the current value on DSNMASK= on the SPOOLDEF statement. Using this mask (with generic characters) limits the format of SPOOL data set name, preventing typing errors from creating or using an unintended data set for SPOOL.
The VOLSER of a SPOOL volume must match the value of VOLUME= on the SPOOLDEF statement, either as a prefix if there are no generic characters in VOLUME= on SPOOLDEF, or match using the generic characters. This restriction prevents an unintended volume from being used for SPOOL.
If you have multiple JESplexes in your environment, it is recommended that the VOLSER or data set name for SPOOL clearly indicate what JESplex the SPOOL data set is intended to be used in.
The SPOOL data sets are formatted with records that match the buffer size specified on the BUFSIZE= parameter on the SPOOLDEF statement. The buffer size can be 1944 - 3992 bytes (though most customers use the default size of 3992). The data set can be formatted before its first use, or JES2 formats it before adding it to the configuration. All volumes must have the same buffer size, and the buffer size cannot be altered without performing a cold start.
SPOOL space is allocated in units called “track groups”. A track group consists of multiple of tracks on the volume, however it is specified in terms of records using the TGSIZE= parameter on the SPOOLDEF statement. The number of tracks per track group on a SPOOL volume is determined when the volume is started (either by command or cold start). Different volumes can have a different number of tracks per track group. The value of TGSIZE= is set based on the type of jobs and data sets that are typically created. If most SYSOUT data sets are large, a higher value for TGSIZE= performs better but could result is less efficient use of SPOOL space. If most SYSOUT data sets are small spin data sets, a lower value for TGSIZE= will make more efficient use of SPOOL space but could slow the performance when creating larger SYSOUT data sets.
Another setting on SPOOLDEF, TRKCELL=, specifies the number of records per track cell. When a SYSOUT data set is written to a SYSOUT class that specifies TRKCELL=YES, space is assigned to the data set in track cell size increments. This improves the efficiency of reading in the data sets later when they are printed (or accessed by applications via SAPI or spool data set browse). The number of records per track will be divided into an even number of track cells. Tracks cells are generally full or half tracks.
SPOOL space ownership is tracked in internal control blocks and using data stored in record zero of the first track in each track group of SPOOL. Record zero in DASD architecture is an 8-byte data record that exists at the beginning of every track. JES2 stores the job number and job key (unique identifier) of the job that owns the SPOOL space. This ensures the integrity of ownership of every track group of SPOOL space. It also allows for a garbage collector to reclaim any SPOOL space whose owning jobs have lost track of it.
One of the disadvantages of using record zero is that JES2 is one of the few functions that uses the record. As a result, there might be applications that copy or move data sets or volumes that might not correctly preserve record zero. Before using any process to clone or move a SPOOL volume or data set, you must ensure that the process preserves the contents of record zero.
The following SHARE presentation provides more information about how to manage the JES2 SPOOL:
1.2.5 Checkpoint data set
The JES2 checkpoint is used to store various control blocks that represent the various work queues that JES2 manages. It serves two purposes:
•It is used when JES2 restarts to restore the work queues to their previous state.
•It serves as the primary communication between members of a MAS.
Because of this dual role, it needs to be stored on a non-volatile medium, and it will have very short access times.
Each member of the MAS has a complete copy of the checkpoint data in memory. This allows rapid access to the work queues that are placed in the checkpoint. When designing JES2 enhancements, one issue is whether to place data on SPOOL or in the checkpoint. This is a trade-off between the delay created by reading data from SPOOL and the overhead of placing the data in the checkpoint where every member has to maintain a current copy of it. The deciding factor is often if the data is needed for an operator command. To ensure commands can complete quickly, the data needed is generally placed in the checkpoint. Also, data in the checkpoint is available in extended status terse requests and displayed in the SDSF main view. Data on SPOOL is placed in the verbose section in extended status and often only displayed in the alternate view in SDSF.
The checkpoint is shared by MAS members in a time-sliced manner. Each member gets a lock on the checkpoint data set, reads in the changes made by other members, processes the queues, writes updated control blocks back to the checkpoint, and finally releases the lock. It then waits before trying to get the checkpoint again. The amount of time that a member can hold the checkpoint for, and the time it waits before trying to reacquire the checkpoint is controlled by the HOLD= and DORMANCY= parameters on the MASDEF statement.
HOLD= specifies the time (in hundredths of a second) that a member holds the checkpoint. It will be set long enough that a member can select or add items to the work queues. However, specifying too large a value can keep other members from accessing the checkpoint.
DORMANCY= has two parameters (also specified in hundredths of a second). The first parameter is the minimum amount of time a member waits after releasing the checkpoint lock before trying to acquire it again. This is called “the politeness interval”. Once the politeness interval ends, the member will only access the checkpoint if it needs it. If there is no process that needs the checkpoint, the member reacquires the checkpoint after the time specified on the second parameter passes. Since new work for a member is only discovered by reading the checkpoint, the second parameter controls the latency of the MAS (the delay between when work is added to the queue and when it is processed).
The settings of these parameters can greatly affect the performance of JES2. When first setting up JES2, there are rules of thumb that can be used to establish initial values, but they will be examined periodically (weekly or monthly at first, and later quarterly or yearly) to see if adjustments are needed. They will also be examined when workloads change.
The checkpoint can be placed on DASD or in a Coupling Facility structure. There are two advantages to placing the checkpoint in a Coupling Facility: better performance and better lock management. When the checkpoint is on DASD, a hardware reserve is used to serialize access to the data set. Some customers running GRS star mode have converted this reserve into a SYSTEMS ENQ. But this lock is often in contention, which increases GRS overhead. When the checkpoint is in a Coupling Facility structure, a lock element on the list structure is used to serialize access, meaning that no RESERVE (or ENQ) is issued. The list structure lock has less overhead and is more “fair”. When the checkpoint is on DASD, a RESERVE request will either be granted or will fail (because someone else is holding the RESERVE). If it fails, that system will try again later, with the same chance of success. As a result, it is possible for a large system to dominate access to the checkpoint. When the checkpoint is in a Coupling Facility, if another system is holding the checkpoint when this system looks for it, this system will be placed in a queue. This means that small systems cannot get constantly locked out of the checkpoint by a much larger one.
In general, if there are more than three members in a MAS, placing the checkpoint in a CF structure has advantages. A three-member MAS is a toss up and a two-member MAS generally runs better when checkpoint is on DASD.
JES2 supports two checkpoint data sets: CKPT1 and CKPT2. Depending on the mode the checkpoint is running in (duplex mode or dual mode), these are either primary and backup (duplex mode) data sets, or alternating primary data sets (dual mode). If the checkpoint is in a CF structure, then you must use duplex mode. You can use the $TCKPTDEF MODE= command to alter the mode as needed.
When introduced, dual mode provided a significant performance advantage by trading off extra CPU overhead for a reduced I/O size. But with modern DASD, the benefit of the reduced I/O size is minimal. Because of the extra CPU cost of dual mode, duplex mode would be the better option for most customers.
IBM strongly recommends always running with two checkpoint data sets in use, and with CKPT2 always on DASD. Further, IBM recommends running with DUPLEX=YES on all members whenever running in duplex mode.
When the checkpoint is in a CF structure, JES2 uses the structure in much the same way it uses a checkpoint on DASD. It defines list elements for each of the checkpoint records in the structure. The amount of space needed by JES2 in the structure does not change based on load or number of jobs in the system. Also, the data in the structure is not used as a staging area to improve performance. The structure is used as a permanent storage medium for the checkpoint data. Loss of a structure that contains current checkpoint data results in some amount of data loss (the difference between what is in CKPT1 and CKPT2). Before a coupling facility is serviced, the checkpoint data must be moved to another coupling facility or back to a DASD checkpoint.
JES2 also supports two backup checkpoints that can be used in the event of an error on the primary checkpoint data sets. These are specified as NEWCKPT1 and NEWCKPT2. When an error occurs, JES2 performs a process called checkpoint reconfiguration. For I/O errors, this can either be a manual process or an automated process. If OPVERIFY=NO and the appropriate NEWCKPT1 or NEWCKPT2 is specified, JES2 will automatically move the checkpoint data sets to the backup locations in the event of an I/O error.
You can also use the checkpoint reconfiguration process to manually move the checkpoint from its current location to a new one. This is done either to relocate the checkpoint (for example, to move it back to its primary location after the I/O error has been fixed) or to move it to a larger data set for expansion purposes.
Operations staff will familiarize themselves with the reconfiguration process to avoid potential delays or outages if it is ever started unexpectedly.
|
Note: The JES2 checkpoint will not be confused with checkpoint restart processing or printer checkpoints. All are often referred to a checkpoint processing.
|
1.2.6 Functional subsystem interface (FSI)
The functional subsystem interface (FSI) was originally created to address the problem of JES having to provide specific support whenever a new printer was created. In a sense, it was the print driver code of that time. Before the FSI, JES wrote directly to the printer, using the appropriate Channel Command Words (CCWs). JES had to be aware of all the capabilities of the printer and provide support for them.
The FSI addresses this issue by moving the direct communications to the printer into a separate product. JES provided a formal interface to access SYSOUT data sets and for control of the device.
The FSI is what is known as a JES-centric interface. There are definitions for the interface in JES, the commands to start, stop and alter the devices are JES commands, and JES decides (via JES printer parameters) which SYSOUT to process on the printer. It is designed to lock and act like a JES printer.
In contrast, interfaces like process SYSOUT (PSO) and the SYSOUT application programming interface (SAPI) are application-centric interfaces. JES has no definitions or control of the processes that use these interfaces.
JES2 has never used the FSI for anything but printers (whereas JES3 has used the FSI for Converter/Interpreter processing). In the FSI, there are functional subsystems (or FSS address spaces) and functional subsystem applications (FSAs). An FSS is defined in JES2 using the FSS initialization statement. This defines the MVS PROC used to start the address space and the characteristics of the address space.
The FSAs are printers within an FSS address space. Each printer that will be running in an FSS specifies the name of the owning subsystem on the FSS= statement in the printer definition.
Starting an FSS mode printer will start the FSS address space and activate the FSA within that FSS to process requests. Stopping an FSS mode printer deactivates the FSA and potentially stops the FSS if it is the last active printer and AUTOSTOP=YES was specified on the FSS definition.
1.2.7 Remote job entry (RJE)
Remote job entry (RJE) in JES2 works in the same way as remote job processing (RJP) in JES3. It was created to support unit record devices (card readers and punches and printer) outside the traditional data center. In RJE, the JES2 subsystem acts as the server, receiving jobs and commands from the remote workstation, and sending command output and SYSOUT to the remote printers and punches attached to the workstation. Both JESs support BSC, BSC multi-leaving, and SNA protocols when using RJE.
JES2 RJE consoles do not receive all the messages that JES3 RJP consoles do. The JES2 consoles can only receive JES2 messages routed to the console. JES3 RJP consoles can receive other operating system messages in addition to the JES3 messages.
JES2 supports a concept of remote pooling for RJE output. This allows a pool of remotes that sign on as individual RJE devices to be pooled together when printing output. For example, if remotes 2, 3, and 4 are pooled, any SYSOUT routed to one of the remotes can print on any of the remotes.
Since all JES2 address spaces can perform all JES2 functions, RJE devices can be attached to any member of the MAS (including secondary JES2 subsystems).
1.2.8 Network job entry (NJE)
Where RJE is a connection of workstations to servers with the intent to support unit record devices outside the data center, network job entry (NJE) is a connection between peer systems for the purpose of exchanging jobs and SYSOUT.
An NJE network is a collection of nodes. In JES2, a MAS is a single NJE node. NJE connections can be established by any member of a MAS. JES2 supports BSC, SNA, and TCP/IP protocols when connecting to other nodes. Unlike JES3, JES2 is not limited to a single protocol when connecting to other nodes in the network. NJE can be used to send jobs and SYSOUT between JES2, JES3, IBM z/VM® (using RSCS), IBM z/VSE® (using IBM POWER®), and IBM System i® systems.
NJE supports both directly-connected and indirectly-connected nodes. If a node is indirectly connected, NJE jobs are sent to a directly connected intermediate node and then forwarded to the destination node. These intermediate nodes are called store and forward nodes.
Network routing is the process of determining how to send NJE jobs to other nodes, potentially through intermediate store and forward nodes. To facilitate this, JES2 has a function called the network path manager. In an NJE network of JES2 nodes, there are connection status records that are shared across NJE connections. These connection records allow JES2 to dynamically determine the routing needed to reach a destination node. Connections that involve non-JES2 nodes are called non-path manager connections. Connections between JES2 nodes can also be defined as non-path manager connections. JES2 must have CONNECT statements defined to represent connections between non-path manager nodes.
The JES2 path manager uses the path manager connection records combined with static CONNECT statements to determine the path (or routing) to get to nodes in the NJE network. When defining an NJE network, you must ensure that the proper connect statements and other network parameters are set to ensure consistent flow of data through the network.
The following SHARE presentation provides more information on various JES2 NJE parameters:
1.2.9 JESXCF
JES2 uses the checkpoint data set as the primary way to communicate between members in a MAS. However, some messages need a more direct and immediate mechanism. For this, JES2 uses XCF messaging managed by the JESXCF component. JESXCF extends the standard XCF messaging model to be more compatible with the subdispatching needs of JES2. It also provides member and system status updates to JES2.
JESXCF is started automatically during z/OS initialization processing. All JES2 address spaces running on a system use the same JESXCF address space. There are multiple JESXCF/XCF groups that JES2 members join to perform different functions. The main group defaults to the NJE node name of the MAS the member is in. However, this can be altered using the XCFGRPNM= parameter on the MASDEF statement.
JES2 uses JESXCF to process the following:
•Member and system status tracking. The status information is used to control processes like resetting failed members and to managing processes such as starting and stopping SPOOL volumes.
•The checkpoint reconfiguration process.
•SPOOL migration coordination.
•Cross member data retrieval (such as in storage buffers from other members).
•Posting members for various queue status changes.
JES2 also uses XCF indirectly when it issues multi-system ENFs 58, 70, and 78. These do not go through JESXCF, but they do use similar XCF resources. ENFs 58 and 70 are used by JES2 instances to update each other about what jobs or SYSOUT groups are active on devices reported in the device status SSI. They can also be used by other products to track jobs and SYSOUT as they move through various phases of processing.
1.2.10 JES2 initialization deck
Whenever a JES2 address space initializes, regardless of the start type, it reads the JES2 initialization decks. The initialization statements are read from either the HASPPARM DD in the JES2 PROC, or from the logical PARMLIB concatenation defined to z/OS.
As each statement is read, JES2 performs system symbol substitution on each initialization statement. This allows installations to customize the initialization of JES2 on a system by system basis, but without having to have unique JES2 parm members for each system.
JES2 supports an INCLUDE initialization statement that can embed additional initialization statements from either another member of the current input data set, a member from the logical PARMLIB, or a fully specified data set name. Because JES2 does system symbol substitution, the data set or member name that is being included can contain a system symbol. For example:
INCLUDE DSN=SYS1.JES2PARM(&SYSNAME)
This would include a member from SYS1.JES2PARM that matches the name of the system being initialized.
If the member you want to refer to resides in the logical PARMLIB, you would specify the member name as MEMBER= in the PARM= for JES2. Otherwise, JES2 attempts to read from the HASPPARM DD for initialization statements. If there is no HASPPARM DD (or it cannot be opened) and MEMBER= was not specified in the PARM=, JES2 defaults to reading from the member HASjesx, where jesx is the name of the subsystem being started (normally JES2).
As part of streamlining the JES2 PROC, you might not want to have a HASPPARM DD in the JES2 PROC and instead use the logical PARMLIB concatenation to start JES2. But you might not want to keep your JES2 initialization decks in the SYS1.PARMLIB data set. This can be accomplished by placing an HASJES2 member in your PARMLIB concatenation that does an include of the appropriate data set and member that contains the JES2 initialization parameters (potentially using system symbol substitution as needed to select a data set or member). An alternative to this is to add the data set with the JES2 initialization decks to the default PARMLIB concatenation and then process initialization statements from there. This is a matter of installation preference.
This is all implying that there can be more than one member that contains your JES2 initialization statements. Generally, that is how customers do things. There are members that have general settings (such as SPOOL and checkpoint definitions, MAS-wide resource limits, MAS NJE parameters, and so forth), a network-wide definition of NJE nodes (shared with other MASes), and a MAS definition of DESTIDs. They then have local (member level) definitions of parameters, printers (and other devices), RJE workstations, and so forth. The number of members generally depends on both style and scope of management (perhaps one person manages NJE nodes so all that data is in a single member).
Besides organization, there is also the question of whether you want to have a general initialization deck that does an INCLUDE of the local definitions or a local initialization deck that does an INCLUDE for the general definitions. Again this is a question of style and preference.
All these member includes can cause some confusion about exactly which data sets and member were used during initialization. One suggestion to help improve understanding and help to debug problems is to include Display commands in your initialization decks. At the top of every initialization deck member, you can add a D INCLUDE statement. This will display (via WTO) where the D INCLUDE statement was read from (either PARMLIB member or full data set name and member). This lets you see what initialization decks were used. Another common practice is to display parameters like CKPTDEF or MASDEF at the end of the main initialization deck. This helps you see what was actually set by the initialization decks.
Beginning with z/OS 2.1, you can also use the $D INITINFO command to display where initialization statements were read from. Example 1-3 on page 11 has a sample of the output of this command.
Depending on the type of start that is being performed, some statements that cannot be processed on this type of start will be ignored. The type of start required to alter each JES2 parameter is listed after each keyword in the z/OS JES2 Initialization and Tuning Reference, SA22-7533. Statements that try to alter one of the MAS resources limits (like the number of jobs) on an inappropriate start will get a message similar to this:
Example 1-4 Message for ignored limit change
$HASP442 INITIALIZATION STATEMENTS CONFLICTING WITH SAVED VALUES FOLLOW:
$HASP496 JOBDEF JOBNUM=500 SAVED VALUE OF 410 WILL BE USED
Other initialization statements, such as altering characteristics of a job class, are ignored without a message.
1.2.11 JES2 monitor
To help you determine if JES2 is functioning properly, JES2 has a separate address space (one per JES2 address space) that monitors the JES2 main task. Its purpose is to detect problems in the JES2 main task that could prevent it from performing its normal functions. It does this both by monitoring activity of the JES2 main task and collecting data about resources that JES2 uses. It can also examine status indicators set by the main task that indicate that abnormal conditions exist.
The monitor is started as part of normal JES2 initialization and terminated on a clean shutdown of JES2. It issues messages when certain conditions are detected (such as the JES2 main task looping or waiting). It also supports commands to display abnormal conditions. All monitor commands start with the command prefix followed by the character J. For example, $JD MONITOR displays the status of the monitor address space.
The command $JD STATUS is the primary command to determine if JES2 is operating properly. It displays alert messages about abnormal conditions that the monitor has detected, resources that are above warning levels, and conditions that could affect normal JES2 processing.
Information that the monitor collects is also available from an SSI request and also displayed in SDSF (as prefix commands in the MAS panel and the RM panel).
1.3 JES2 internal components
At their most basic, JES2 and JES3 have the same structure. There is a single task within the JES2 address space that performs most of the processing. This is called the JES2 main task. There are subtasks that perform work that could possibly result in a WAIT or that need to be offloaded from the main task. JES2 connects to the SSI to handle requests from z/OS or other applications. There is also an interface that provides application access to JES data sets on SPOOL called the HASP access method (HAM).
However, the way that JES2 works internally is very different from how JES3 works internally. JES2 has no equivalent to a dynamic service program (DSP) or to the concept of a job segment scheduler (JSS). There is no way to directly specify which phases a job in JES2 will pass through. All jobs move through the same processing phases, affected only by their success or failure of each particular phase.
1.3.1 Processor control element (PCE)
JES2 processing centers around a control block called the processor control element (PCE). This is the unit of work (or thread) that is dispatched under the JES2 main task TCB by the JES2 dispatcher. A PCE can represent a device such as a printer. It can represent a phase of processing such as output processing or purge processing. Or it can represent some work process such as process SYSOUT (PSO) or the SYSOUT API (SAPI). All work done by the JES2 main task that is outside the JES2 dispatcher runs under the control of a PCE.
Because JES2 (and JES3) subdispatches a single TCB to perform much of its work, functions running in the main task cannot enter an MVS wait. If this were to happen, it stops all JES work until the wait is resolved. This can cause deadlocks if a JES function is needed to resolve the wait.
Most PCEs are created during JES2 initialization. However some, like PCEs that help perform warm start processing, can be created and deleted as needed. PCEs are defined in a table using the $PCETAB macro. But they can also be defined in installation exits using dynamic tables and the same $PCETAB macro.
1.3.2 Daughter task element (DTE)
Similar to how a PCE represents a subunit of work running in the JES2 main task, a DTE represents an MVS subtask that is running in the JES2 address space. Subtasks are created to perform work that either cannot be done under the JES2 main task or for performance reasons is best done in a separate task.
Operating system functions like WTO, allocation, and conversion can all result in an MVS WAIT. Because of this, these functions must be performed in a subtask. Some subtasks in JES2 are associated with a particular PCE, such as the conversion subtask. Others are associated with a function such as the WTO subtask. Still others are general-purpose subtasks that can perform any work that must be run outside the JES2 main task.
With the introduction of JES2CI address spaces that perform the conversion and interpretation process during the JES2 conversion phase, there can now be subtasks (DTEs) in other address spaces. When the JES2CI address space is being used, the converter subtask is started in the JES2CI address space instead of the JES2 address space. Otherwise, it functions in much the same way as a subtask in the JES2 address space.
1.3.3 Device control table (DCT)
JES2 does have various devices associated with it, though this is not as extensive as JES3 managed devices. The devices that JES2 manages are represented by DCT data areas. Devices include the physical devices like a printer, or a BSC TP line (CTC) and also logical devices like SNA lines or offload devices. The DCT manages the settings and status for the device. Many devices have a direct relationship with a PCE (such as a printer). In other cases, multiple devices can be associated and managed by a single PCE (such as lines). And there is also a concept of subdevices.
A subdevice is either a logical or physical device that is part of a larger device. An example would be a line that is used for NJE can have various transmitter and receiver devices associated with it. These manage the various job and SYSOUT streams that can be active on an NJE connection at any time. Another example is an RJE line can have various printers and readers associated with the RJE workstation. Each RJE device has its own DCT to manage it.
Subdevices are named as part of the primary device. So an NJE line called LINE15 could have a SYSOUT transmitter named L15.ST1. Similarly, remote 6 can have a printer named R6.PR1.
1.3.4 HASP communication tables (HCT and HCCT)
There are two anchor control blocks in JES2. In the JES2 address space, there is the HASP communication table (HCT). This table is pointed to by register 11 when code is running in the main task. It points to the major data areas in the JES2 address space and has fields that reflect the options specified during initialization. The HCT is divided into two sections: Set of local data areas and a set of checkpoint data areas. Any change to the checkpointed section is automatically propagated to other members using the JES2 checkpoint.
In common storage, the HASP common storage communication table (HCCT) anchors the common storage data areas used by JES2. It also has pointers to the data spaces that JES2 used to store some data areas. Code running in the SSI points to this area using register 11. The HCCT also has copies of some of the options that are in the HCT.
1.3.5 Checkpointed data areas
The JES2 checkpoint holds the primary job and output queues, data needed to manage the spool, and other areas JES2 needs to keep members synchronized. It contains data that is needed to start or restart a member. The checkpoint is divided into control information (such as the portion of the HCT that is shared between members), and data blocks called CTENTs. Each CTENT has an array of homogeneous control blocks.
For example, there is a CTENT that tracks what member an RJE device is signed on. Each defined remote has an element in this array. To find the entry for a particular RJE device, you index into the array to find the corresponding entry.
All CTENTs can be thought of as arrays of control blocks. However, most are not directly accessed like the remote sign-on table, but instead contain various chains of array elements. There are queue heads and chain pointers used to run the elements in the section using the array index.
Each CTENT is mapped to 4 K records in the checkpoint data set. The size of the array (and thus the number of records in the checkpoint) is established when JES2 is cold stared. Many of the arrays can have their size altered via operator command. Most arrays can be increased or decreased in size. Increasing the size of an array depends on the availability of space in the checkpoint data sets (or structures).
The size of many of the arrays (CTENTs) in the checkpoint constrains the number of objects JES2 can manage. These limits are monitored by JES2 and reported using the $HASP050 message. You will understand what is stored in the various CTENTs and how reaching the limits can affect your system.
Job queue element (JQE)
The job queue element (JQE) represents a job, started task, or TSO logon. These elements constitute the job queue in the checkpoint. Each active JQE is on one of the phase chains in the checkpoint (or on a free chain). JQEs are in no particular order, but another CTENT, the job index (JIX), can be used to locate the JQE for a given job number. If you run out of JQEs, you cannot submit any jobs, log on to TSO, issue a start command, or receive jobs or SYSOUT over NJE. The number of JQEs is controlled by the JOBDEF JOBNUM= specification. It can be increased or decreased via operator command.
Job output element (JOE)
When a job created output, it is represented by the Peripheral Data Definition Block (PDDB) on spool. Each instance of a data set has its own PDDB to describe the properties of that instance. When a job completes execution (or when a spin data set is spun), the PDDBs are gathered into job output elements (JOEs) for printing. All SYSOUT data sets that can be accessed by processes like printers, SAPI, and NJE are represented in JOEs.
The number of JOEs is controlled by the OUTDEF JOENUM= specification. It can be increased or decreased via operator command. If JES2 runs out of JOEs, it cannot make PDDBs available for processing. However, the SYSOUT data sets will still exist on spool. As JOEs are freed up, waiting PDDBs will be processed and placed into the available JOEs. For normal job output (not spin data sets), this has little impact on the system (except for the delay in making the data sets available for processing). But if a job cannot get a JOE for a spin data set, JES2 must go through an I/O intensive process to locate the spin data sets that need to be processed. This can impact the performance of a running system.
Block extension reuse table (BERT)
One of the major limitations of the checkpoint structure is that the CTENTs are arrays of fixed size element. This presents two problems. First, it is difficult to expand the size of the elements in the array. This is due to how the data is accessed by JES2. Second, if the element size is expanded, all elements of a particular type get larger. This requires JES2 to maintain space in the elements for the maximum size of all possible data that could be associated with an element. This is expensive if the data is variable in size (such as the accounting string) or only applies to a small subset of jobs (such as SYSLOG jobs).
The block extension reuse tables (BERTs) were created to address both these problems. Using BERTs, JES2 can add extensions to other existing data areas (such as JOEs and JQEs). These extensions can contain variable and optional data. This allows JES2 to maintain data for some jobs (such as the system name associated with a SYSLOG job) that do not consume space in the checkpoint for jobs that do not need it.
In addition, BERTs can be used to create other control blocks that do no warrant their own section in the checkpoint. This is either because there are too few instances of the data areas, or the number of data areas needed is not predictable. Examples are data areas that track duplicate job names (DJBs) or data areas that represent job classes (CATs).
Since their introduction, BERTs have become more integral to normal JES2 operation. However, this also implies that running out of BERTs (they are still an array in the checkpoint) can have varied and serious impact. So the question is always asked, how many BERTs is enough? The number of BERTs is specified on CKPTSPACE BERTNUM=. The number can be increased or decreased via operator command. But the number you need is very dependent on an installation’s work stream. As stated, BERTs contain variable data that is stored with various control blocks. One example is accounting strings. When a job goes through input processing, the accounting string is associated with the JQE and stored in BERTs. They remain with the job until it completes execution. At that point, the data is removed and the BERTs freed. So the number of BERTs needed to store the accounting string depends on the length of the accounting string at your installation and the number of jobs that are typically queued for execution.
Another example is tracking duplicate job names. When multiple jobs with the same job name are submitted, JES2 tracks them from after conversion until they complete execution. This is done to reduce the overhead of ensuring that two jobs with the same name are not executing at the same time. The tracking is done regardless of whether duplicate jobs matter (DUPL_JOB= parameter). The overhead of this then depends on the number of job names that have duplicates awaiting execution.
The bottom line is that installation must monitor the usage of BERTs over time and ensure that there are an adequate number defined. The cost of over defining BERTs is low, so often customers do that. A general rule of thumb is one BERT per JQE plus one quarter the number of JOEs.
Job numbers
JES2 assigns job numbers from a pool that is limited using the JOBDEF RANGE= specification. This can be changed at any time with an operator command. There is no direct relationship between the job number pool and the JQE pool that represents jobs. The range is typically greater than the number of JQEs defined to the MAS, reducing how often the job numbers wrap. The job index (JIX) is used to determine what job numbers are available for a new job.
Jobs arriving from NJE are assigned their original job number if it is available. If the original job number is outside the current range but available, it will be assigned to the job if JOBDEF RASSIGN=YES. If the job number is not available, the new job will be assigned a number within the range. If RASSIGN=NO, all jobs must be assigned a job number within the current range.
The job unidentified is a character representation of the job number. In JES2, the job identifier also indicates the type of job, batch job, started task, or TSO logon. The exact format of the JES2 job ID depends on the job number range. If the upper limit of the range is less than 100,000, then job identifiers start with a three character prefix (JOB, STC, TSU) followed by the job number. For example, a batch job assigned the number 1234 would have a job ID of JOB01234. If the upper level of the range is 100,000 or greater, the job ID prefix is reduced to a single character (J, S, T). In this case, the job assigned job number 1234 would have a job ID of J0001234. Jobs arriving via NJE that have an original job number of 100,000 or greater and with RASSIGN=YES will keep their original job number and have a job ID that starts with a single character regardless of the range setting.
Spool space
Spool space is represented in the checkpoint by two main data areas (CTENTs). First the Direct access spool block (DAS) represents each spool volume (data set). These are defined in 32 volume increments using the SPOOLDEF SPOOLNUM= specification. This value can be increased by operator command but not decreased. The value limits the number of spool volumes that can be defined.
The track group map (TGM) keeps track of what spool space is available. Each track group has 2 bits that represent it, one tracks the availability of the track group and the other tracks whether there were any problems accessing the track group. The number of tracks in the track group map is controlled by SPOOLDEF TGSPACE=MAX= specification. Track group space is specified in increments of about 16,000. The maximum number of track groups can be increased by operator command but not decremented.
1.4 JES2 start types
When JES2 is started, the value specified in PARM= can control the start type and the options associated with the start. There are two specifiable types of starts:
•COLD starts clear the entire checkpoint and logically clears the spool. All data that was stored on spool is logically lost. There is another option FORMAT that can be specified with COLD or in place of it to request that all spool volumes are formatted. On a cold start, JES2 detects whether a spool volume is formatted for use by JES2. This is done by inspecting specific tracks on the volume to see if they are formatted. If you are cold starting and are not certain that all volumes are properly formatted, you can specify the FORMAT option. Formatting large volumes can take up to an hour depending on the environment. Because of this, the FORMAT option will only be used when needed. An alternative to the FORMAT option would be to use IEBDG to format the spool volumes. Though this is not necessarily faster, it can be done before the IPL.
•WARM starts read in the existing checkpoint information and the initializing member joins the MAS. Warm starts take various forms based on the environment when the member is started. These forms are:
– All member warm start. This form of warm start occurs when this member is the first member to join a MAS (there are no other active members). The member being initialized is also not resuming after an ABEND of JES2 without an IPL (a hot start, described below). There are certain parameters that can only be changed on an all member warm start. In addition, there is a more complete validation of all jobs and SYSOUT data structures that only occur on an all member warm start.
On an all member warm start, you can request that the spool track group be rebuilt by specifying SPOOL=VALIDATE with WARM on PARM=. This causes JES2 to read each jobs spool allocation data area and rebuild the track group map based on what tracks jobs are actually allocated on. This can be a time consuming process and is generally only done if there are serious problems with the spool data structures.
On an all member warm start, you can request that the spool track group be rebuilt by specifying SPOOL=VALIDATE with WARM on PARM=. This causes JES2 to read each jobs spool allocation data area and rebuild the track group map based on what tracks jobs are actually allocated on. This can be a time consuming process and is generally only done if there are serious problems with the spool data structures.
– Single member warm start. This form of warm start occurs when a member is joining an existing MAS (there is at least one member active or hot startable in the MAS). The member was active previously and did not terminate cleanly (implying that there still might be work associated with the member in the job or output queue). The member being initialized is also not resuming after an ABEND of JES2 without an IPL (a hot start, described below).
Single member warm starts require the member to examine the job and output queue to determine if there are any jobs that are associated with the restarting member. If so, processing is required to process the incomplete processing of this work. This process takes time and can delay the JES2 start. To avoid this, when a member fails and the image it is running on is IPLed, there is an option to perform this restart processing on an active member of the MAS. This can be done using the $E MEMBER command or automatically using the MASDEF AUTOEMEM combined with the MASDEF RESTART specification. AUTOEMEM cleans up failed members when JES2 detects via XCF (JESXCF) that the member has terminated.
– Quick start. This is the same as a single member warm start except that the member is either new to the MAS (this is the first time it is starting) or it was in the MAS previously but came down cleanly (via a $P JES2), or the work was reset after an IPL using $E MEMBER or AUTOEMEM processing. As its name implies, this is the quickest start of JES2.
– Hot start. A hot start is the restarting of a JES2 address space after it has ABENDed. The subsystem interface (SSI) is still active. JES2 restarts the address space, deals with any work that was active in the address space when JES2 ABENDed, and then reconnects to the existing SSI. The intent of this type of start is to be as minimally disruptive as possible.
– All member hot start. This is referenced in $ACTIVATE processing as a case where a starting member can hot start and change the activation level with the start. In this type of start, there are no JES2 address spaces active anywhere in the MAS (they are either not active or the JES2 address space on the member has been ABENDed). There is certain processing that can be done on this type of start.
As you read the JES2 documentation, there are descriptions of the types of starts needed to make certain changes. As you can see, the warm start types are not options on the start command (or PARM=) but rather environments that must be created before starting JES2.
1.5 Activation levels
To avoid cold starts and also enable various functions, JES2 uses a function called $ACTIVATE. What this command does is alter what is referred to as the checkpoint level, checkpoint mode, or activation level (different terms for the same concept) to enable or disable functions. The process involves verifying that all members have the appropriate level of code, that certain functions are properly enabled or disabled, and that the checkpoints are large enough for the new checkpoint level. Performing an activation often involves reformatting data areas in the checkpoint or adding section (CTENTs) to the checkpoint. When a new level of code is activated, this generally prevents older level of code from joining the MAS. This ensures that before a function can be used, all members have the correct code to support that function. Also, the use of new functions can create data areas that prevent activation to a lower level.
Generally, there are two levels that any release of JES2 can run in. One is a compatibility level that supports the older releases. The other is the activated or full function level that enables the new function. After older releases fall off the coexistence support, compatibility mode is generally dropped and customers must activate full function mode before migrating. The design does support more than two levels of activation at a time, but this has never been needed.
Not all releases include a new activation level. Generally, they are only needed every few releases.
In some instances, full function mode might require changes to JES2 exits. This is because an exit might be accessing the data areas that are changing in format. All of the JES2 code supports dynamically changing modes using operator commands. However, your exits might not have been written to support this, or can only support moving in one direction.
In preparation to change your activation level, you will issue a $D ACTIVATE command. This displays the current checkpoint level and any requirements that have not been met to go to the other checkpoint level (assuming that there are only 2 allowed for this release). In some cases, an option might need to be turned on or off (depending on the direction of activation). In other cases, the checkpoint data set might need to be enlarged. There also might be a requirement to increase the limit on a parameter (for example the number of BERTs). The command also ensures that there are no errors in the checkpoint queues. All these must be set before issuing the $ACTIVATE command.
The most common way to alter the checkpoint level is to use the command:
$ACTIVATE,LEVEL=Z11
This will update the activate level to the Z11 level (the full function level for z/OS 1.11 to z/OS 2.1). To go back to compatibility mode, the command would be:
$ACTIVATE,LEVEL=Z2
This restores the activate level to the Z2 level (the compatibility level for z/OS 1.11 to z/OS 2.1).
On a cold start, JES2 by default starts in the full function mode for the current release. However, you can use the OPTSDEF COLD_START_MODE= initialization statement to cause JES2 to cold start in a particular mode. This parameter is only used on a cold start. The default is to start in full function mode of the current release. The valid values are the same as the LEVEL= keyword on the $ACTIVATE command.
Another way to affect the activation level is to use the UNACT parameter on the PARM= when starting JES2. This parameter allows you to cold start in compatibility mode or to switch to compatibility mode on an all member warm start or an all member hot start.
1.6 JES2 job processing
The life of a job starts when it is submitted to JES2 via one of its input sources. From there, it progresses through a series of phases that include the running of the job, processing of the output, and ultimately in the purging of the job. There are two primary ways jobs are processed by JES2:
•Pre-execution jobs that include JCL that describes how to run the job.
•Post-execution jobs that arrive via NJE and contain essentially SYSOUT data sets to be processed.
NJE also includes the concept of store and forward job streams that can be pre or post executions. These are jobs that are transitioning through a node on the way to their final destination.
1.6.1 Pre-execution job phases
Jobs submitted to JES2 though an input device (such as a reader or job receiver) go through the phases shown in Figure 1-4. Jobs (JQEs) move from phase to phase by moving from one work queue to another. Members of the MAS select work from a queue based on their ability to process the job, and for some phases, the affinity associated with the job.
Figure 1-4 Standard JES2 job phases
Input phase processing
The first phase that a job goes through is the input phase. This phase performs the following tasks:
•Creates the control blocks needed for the job in the checkpoint (JQE) and spool (JCT and IOT)
•Parses the JOB card extracting fields needed by JES2 to process manage the job
•Parses, validates, and builds the data areas associated with the JES2 JECL cards
•Separated the instream data from the JCL and created the needed instream data sets on spool
•Passes the security information to the security product (via SAF VERIFYX call) to determine the authority that the job will run under
In addition, if this is an internal reader (INTRDR) then symbols processing occurs for symbols passed on the internal reader allocation and to extract and return symbols related to the job submission.
Where the actual processing for the input phase occurs depends on the type of input device. Internal readers and NJE over TCP/IP processing occurs in the address space that is writing the JCL. So a TSO user issuing a SUBMIT has the bulk of the input phase processing occur as the records are written to the internal reader. This spreads the overhead (and billing) for this work to the address spaces that submit the jobs. There is still code in the JES2 address space that creates the job in the job queue and then moves it to the next phase. But this is relatively little overhead compared to the rest of input processing.
Input processing for all other devices (RJE readers, SNA, BSC and offload job receivers, and so on) occurs in the JES2 main task under a separate PCE for each device.
Any JES2 main task processing for the input phrase occurs on the member where the job is being input.
If the job is being routed to another node due to an XMIT or a ROUTE XEQ card in the JCL, the job is then queued to the XMIT phase/queue for processing by NJE. Otherwise, it is queued to the conversion phase. There are a few errors that can cause the job to be queued directly to purge processing. Also, before z/OS 2.1, there were cases where the job could be queued to the output phase.
Conversion phase processing
The conversion phase is responsible for calling the z/OS converter to parse the various JCL statements and convert them to internal text. Also, with z/OS 2.1, the conversion phase can also invoke the z/OS interpreter to convert the internal text into SWA control blocks (and detect additional JCL errors). This phase performs the following tasks:
•Determines the PROCLIB concatenation to be used by the job
•Passes the job to the conversion subtask
•Under the conversion subtask, the following occurs:
– JES2 opens the appropriate PROCLIB
– Reads in any symbols passed via the internal reader
– Calls the z/OS converter
– Processes any instream data creation requests from the converter
– Checks if the interpreter needs to be called and invokes it if needed
•A service class is assigned to the job based on WLM classification rules
•Duplicate jobname processing is performed
•The job is queued based on the return codes from the services called
Conversion processing can be done on any member of the MAS. There is one PCE for every conversion subtask. The conversion subtasks can be in the JES2 or JES2CI address space (depending on whether the interpreter is being called).
At the end of conversion processing, the job can be queued for the processing (execution) phase or the output phase if errors are detected.
Processing (execution) phase processing
The processing (or execution) phase is where the job runs. Jobs are queued to this phase by job class (one queue per class). In addition, if this is a batch job class that is managed by WLM, it is also added to a service class queue. Started tasks and TSO logons (also known as demand select jobs) are queued to the STC and TSU classes only.
The batch job classes have properties that can help an installation manage where and how many jobs execute in a given class. These control for both WLM and non-WLM managed job classes. There are limits on what members a class can run on, the number of jobs that can execute in the class anywhere in the MAS, and the number of jobs that can execute on each member of the MAS. In addition, if the job is in a WLM managed class, then there are controls to determine what members that WLM service class is active on.
If a batch job class is WLM managed, every job in that class is also queued to a service class queue in JES2. Each JES2 member will register the service class queue with WLM if the service class is active on that member. However, there is a restriction in WLM that only one address space per system can register to process a particular service class queue. The registration includes a MAS identifier, so the restriction is one service class queue manager per system in each MAS. So if there are 2 JES2s (JES2 and JESA) on the same system in the same MAS, only the JES2 will register with WLM and run WLM managed jobs. The system that registers the WLM service queue is referred to as the boss system.
Jobs in WLM managed classes are run in initiators that are started and stopped by WLM. These initiators are started and stopped based on the importance of work in the service class, the service class goals, the size of the queue of jobs waiting to execute and the capacity of the various systems in the MAS to absorb more initiators. The size of the queue of work can be influenced by the job class limits previously discussed. The size is also limited by the affinity associated with the jobs, and the systems where the job class is active.
JES2 is responsible for selecting the next job to execute. Even with WLM management, all WLM controls is the number of WLM managed initiators active on a member. JES2 selects the specific job to be placed in an initiator.
An alternative to WLM managed initiators is the traditional JES2 managed initiators. These must be defined in the JES2 initialization deck and are started and stopped by operator commands. Where the initiators are started and the number active is controlled by operator command.
There is also the capability to start batch jobs via operator command. This sends a request to WLM (even if the job is in a JES managed job class) to create an address space to run the job in. WLM will select a system (within the affinity limits for the job) that has capacity to run the job and then start an initiator to process that job. JES2 will then assign the job to that initiator.
After a job is assigned to an initiator and starts running, the JES2 address space does not get involved in running the job. It will process specific SSI requests that might be made by the address space (such as a SAPI or PSO SSI request) and provide spool space to the job (via the CSA space cache called the BLOB). The JES2 address space does not get involved when data sets are allocated or when SYSOUT is created. However, if a spin SYSOUT data set is created, a request is queued to the JES2 address space to build the JOE for the data set. This is an asynchronous request that the address space makes to the owning JES2 address space.
When a job completes running, JES2 examines how the job terminated and queues it to either:
•The output phase under normal conditions
•The spin phase if the job could still have unprocessed spin requests
•Back into execution if the job needs to be rerun
Spin phase processing
The spin phase exists to ensure that there are no outstanding spin SYSOUT data set requests for the address space. The spin phase is processed by the same set of PCEs that processes the spin SYSOUT data set requests. For each job in this phase, the spin work queue is examined. If there are no requests pending, the job is queued to the output phase. If any requests are found, the job remains in the spin phase until those requests are processed.
Output phase processing
This phase takes the non-spin SYSOUT data sets that were created during job execution and groups them into job output elements. Each instance of a data set has had a PDDB built for it. PDDBs with like characteristics are grouped into a JOE. The JOEs are then placed on the SYSOUT queues for processing by the various output processes (such as SAPI and printers).
It is during the output phase that job completion notification is issued. It is done at this point because all jobs (except for certain error scenarios) go through the output phase. This phase also reads the spool data areas, which are not read in the JES2 address space during the processing phase.
It is also during this phase that JES2 cleans up data areas that are no longer required for the job. Data stored in BERTs that is not longer required is deleted.
After a job completes the output phase, it will either be queued to the hardcopy phase (if there are any JOEs associated with the job) or the purge phase (if there are no JOEs for the job).
Hardcopy phase
Jobs that have completed the output phase and have any JOEs associated with them are placed on the hardcopy phase while their JOEs are processed. From the jobs standpoint, nothing happens to the job during this phase.
As JOEs are selected, processed, and purged, a check is made to see if there are any JOEs left for the job. When the last JOE for a job is purged, the job is moved to the purge queue by the process that eliminated the last JOE.
Purge phase
This is the last phase in the life of a job. Before a job is queued to this phase, all the JOEs have been removed and all that remains is the JQE and the spool space. The job’s control blocks that represent the spool space for the job are read in and the spool space purged. The security product is called to audit that the job has been deleted. Finally, the JQE is removed from the job index (JIX) freeing the job number and then the JQE is placed on the free queue.
1.6.2 Post-execution job phases
Post-execution jobs arrive via NJE or offload SYSOUT receivers. These jobs are assigned a job number and JQE in much the same way a pre-execution job does. However, these jobs are placed on the receive queue instead of the input queue. The processing in the receive queue is done in the NETSERV address space for NJE over TCP/IP and in the JES2 address space for NJE over BSC or SNA. The primary difference is that there is no JCL being processed for these jobs as they come in. Instead, NJE job and data set headers are used to build the needed job structures and SYSOUT PDDBs. The SYSOUT data sets are written to spool by the receiver processing.
When a job completes the receive phase, it is queued to the output phase similar to jobs completing execution. The only difference is that if the SYSOUT data sets are spin data sets, the jobs is also processed by the spin PCEs to create the JOEs for the spin data sets. Though the spin PCEs are processing the jobs, they are never actually placed on the spin phase queue.
After the output phase, the job progresses through the hardcopy and purge phases as normal.
1 http://www-01.ibm.com/support/knowledgecenter/SSLTBW_2.1.0/com.ibm.zos.v2r1.hasa800/abstract.htm?lang=en
2 http://www-01.ibm.com/support/knowledgecenter/SSLTBW_2.1.0/com.ibm.zos.v2r1.hasa300/abstract.htm?lang=en
3 For a discussion of SSI, see ftp://ftp.software.ibm.com/s390/jes2/Share102/NewInter.pdf.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.