InfiniBand technical description
In this chapter, we describe the technical implementation of the InfiniBand technology on IBM zEnterprise and System z processors.
zEnterprise and System z CPCs use InfiniBand technology for interconnecting CPCs in a Parallel Sysplex environment.
We discuss the following topics:
•InfiniBand connectivity
•InfiniBand fanouts
•Fanout plugging
•Adapter ID assignment and VCHIDs
•InfiniBand coupling links
•InfiniBand cables
2.1 InfiniBand connectivity
zEnterprise and System z CPCs benefit from the high speed and low latency offered by InfiniBand technology. This technology provides improved reliability, scalability, and performance, which are all attributes important in a Parallel Sysplex.
Because we are dealing with increased I/O data rates and faster CPCs, we need a way to connect two CPCs with a faster and more flexible interconnection. Also, as enterprises move from Sysplex Timers to STP for time coordination, interconnectivity between the CPCs in the Common Time Network (CTN) is required. InfiniBand provides all of this functionality.
Figure 2-1 provides an overview of the InfiniBand coupling implementation options that are available with InfiniBand on zEnterprise and System z CPCs. The connectivity options are:
•Any-to-any coupling and timing-only 12X IFB mode connectivity between zEC12, zBC12, z196, z114, and z10 CPCs.
•Any-to-any coupling and timing-only 12X IFB3 mode connectivity between zEC12, zBC12, z196, and z114 CPCs.
•Any-to-any coupling and timing-only 1X connectivity between zEC12, zBC12, z196, z114, and z10 CPCs (optical link - long reach).
Figure 2-1 InfiniBand connectivity options
Note that Parallel Sysplex InfiniBand (PSIFB) coupling link channel-path identifiers (CHPIDs) can be shared or spanned across logical partitions and channel subsystems on zEnterprise and System z CPCs. However, the total number of CF link CHPIDs (including ICP, ISC, and InfiniBand CHPIDs) must not exceed 128 (64 on CPC generations prior to z196). See 3.1, “Planning considerations” on page 38 for more information.
|
Note: Whenever the text refers to coupling links or coupling link fanouts, this also applies to STP timing-only links.
|
Table 2-1 provides more information about the InfiniBand options that are available on zEnterprise and System z CPCs. We describe these options in more detail in subsequent sections.
Table 2-1 Available InfiniBand options for zEnterprise and System z CPCs
|
Fanout type
|
Description
|
System z9
|
System z10
|
zEC12
aBC12
zEnterprise 196
zEnterprise 114
|
Maximum distance
|
Data link
rate
|
|
HCA1-O
|
12X IB-SDR
|
Optical
coupling link
|
N/A
|
N/A
|
150 meters
(492 feet)
|
3 GBps
|
|
HCA2-O
|
12X IB-DDR
12X IB-SDR
|
N/A
|
Optical
coupling link
|
Optical
coupling link
|
150 meters
(492 feet)
|
6 GBps
3 GBps
|
|
HCA2-O LR1
|
1X IB-DDR
1X IB-SDR
|
N/A
|
Optical coupling link
|
Optical coupling link (carry forward)
|
10 km
(6.2 miles)2
|
5 Gbps
2.5 Gbps
|
|
HCA2-C3
|
12X IB-DDR
|
N/A
|
Copper
I/O cage link
|
Copper
I/O cage link
|
1 - 3.5 meters
(3.2 - 11.4 feet)
|
6 GBps
|
|
HCA3-O
|
12x IB-DDR
|
N/A
|
N/A
|
Optical
coupling link4
|
150 meters
(492 feet)
|
6 GBps
|
|
HCA3-O LR
|
1x IB-DDR
1x IB-SDR
|
N/A
|
N/A
|
Optical
coupling link
|
10 kma
(6.2 miles)b
|
5 Gbps
2.5 Gbps
|
1 RPQ 8P2340 for extended distance is available on client request.
2 An extended distance of 175 km (108 miles) is supported with DWDM. The data rate (DDR or SDR) depends on the capability of the attached equipment.
3 These are only used for internal connections within the CPC. HCA2-C fanouts cannot be used for STP or for connecting to a CF. They are only included here for completeness.
4 The improved IFB3 protocol can be used if two HCA3-O fanouts are connected.
2.2 InfiniBand fanouts
|
Note: InfiniBand links can be used to connect a z9 to a z10 or later. However, they cannot be used to connect two z9 CPCs to each other.
|
This section describes the various InfiniBand fanouts that are offered on the zEnterprise and System z CPCs.
There are six fanout types that are based on InfiniBand technology:
•HCA1-O (z9 only).
•HCA2-O (z196, z114, and z10. They can be carried forward to a zEC12 or a zBC12 on an upgrade).
•HCA2-O LR (Orderable on z10 only - but can be carried forward to z196, z114, zEC12, or zBC12 on an MES).
•HCA2-C (z196, z114, and z10 - depending on the ordered I/O configuration. They cannot be used for timing links or coupling to a CF.)
•HCA3-O (z114, z196, zBC12, and zEC12).
•HCA3-O LR (z114, z196, zBC12, and zEC12).
|
Note: HCA1-O, HCA2-O, HCA2-O LR, HCA3-O, and HCA3-O LR adapters are used exclusively by InfiniBand coupling and timing-only links. Throughout this document, we refer to them as PSIFB fanouts.
|
Each PSIFB fanout has either two ports or four ports (for the HCA3-O LR) to connect an optical cable (see Figure 2-2 and Figure 2-3 on page 21).
2.2.1 Adapter types
This section provides further information about the different types of InfiniBand coupling link adapters:
•HCA1-O
This fanout is only available on System z9. It provides interconnectivity for Parallel Sysplex and STP connections between zEnterprise 196, zEnterprise 114, System z10, and System z9.
The fanout has two optical multifiber push-on (MPO) ports. The link operates at a maximum speed of 3 GBps.
HCA1-O fanouts can be connected only to HCA2-O fanouts on z196, z114, and z10 CPCs. Ports on the HCA1-O fanout are exclusively used for coupling links or STP timing-only links and cannot be used for any other purpose.
|
Note: z9-to-z9 PSIFB link connections are not supported.
|
•HCA2-O
This fanout, shown in Figure 2-2, can be ordered on z196, z114, and z10 CPCs. It provides interconnectivity for Parallel Sysplex and STP connections between these CPCs, and supports connection to a HCA1-O fanout on a z9.
Figure 2-2 HCA2-O fanout and HCA2-O LR fanout
The fanout has two optical MPO ports. The link can operate at either double data rate (if connected to another HCA2-O or a HCA3-O) or single data rate (if connected to an HCA1-O).
Ports on the HCA2-O fanout are used exclusively for coupling links or STP timing-only links and cannot be used for any other purpose.
HCA2-O fanouts can also be carried forward during an MES upgrade from a System z10 to a zEnterprise system.
|
Note: On zEnterprise or later CPCs, the only reason for ordering HCA2-O fanouts is if you need to connect to a z9 CPC using PSIFB. In all other situations, order HCA3-O (or HCA3-O LR) fanouts.
|
•HCA2-O LR
This fanout (also shown in Figure 2-2 on page 20) can only be ordered on System z10. HCA2-O LR fanouts can be carried forward to z196 and z114 by way of an MES upgrade. It provides interconnectivity for Parallel Sysplex and STP connections between zEC12, zBC12, z196, z114, and z10.
The fanout has two optical Small Form-Factor Pluggable (SFP) ports. The link operates at either 5 Gbps or 2.5 Gbps depending on the capability of the attached equipment.
Ports on the HCA2-O LR fanout are used exclusively for coupling links or STP timing-only links and cannot be used for any other purpose.
•HCA2-C
This fanout is available on z196, z114, and z10 CPCs, depending on the I/O configuration. The number of HCA2-C fanouts is not chosen by you; it is determined by the number and type of I/O cards that are ordered. HCA2-C fanouts are only relevant to coupling from the perspective that the number of HCA2-C fanouts that are installed can have an impact on the number of fanouts that can be used for coupling.
The HCA2-C fanout provides the connection between the CPC complex and the IFB-MP cards that are installed in the I/O cages or drawers. Ports on the HCA2-C fanout are exclusively used for I/O interfaces and cannot be used for any other purpose.
•HCA3-O
This fanout (shown on the left side in Figure 2-3) is only available on z196 and later CPCs. It provides interconnectivity for Parallel Sysplex and STP connections.
The fanout has two optical MPO ports. Each link operates at 6 GBps.
If the connection is made between two HCA3-O fanouts and there are no more than four CHPIDs per port defined, the connection will automatically use the improved IFB3 protocol mode.
Ports on the HCA3-O fanout are exclusively used for coupling links or STP timing-only links and cannot be used for any other purpose.
Figure 2-3 HCA3-O and HCA3-O LR fanout
•HCA3-O LR
This fanout (shown on the right side in Figure 2-3 on page 21) is only available on z196 and later CPCs. It provides interconnectivity for Parallel Sysplex and STP connections between zEC12, zBC12, z196, z114, and z10 CPCs.
The fanout has four optical Small Form-Factor Pluggable (SFP) ports. Each link operates at either 5 Gbps or 2.5 Gbps depending on the capability of the attached equipment.
Ports on the HCA3-O LR fanout are exclusively used for coupling links or STP timing-only links and cannot be used for any other purpose.
Table 2-2 summarizes the InfiniBand interconnectivity options.
Table 2-2 InfiniBand interconnectivity options
|
|
HCA1
|
HCA2 12X
|
HCA2 1X
|
HCA3 12X
|
HCA3 1X
|
|
HCA1
|
No
|
Yes
|
No
|
No
|
No
|
|
HCA2 12X
|
Yes
|
Yes (IFB)
|
No
|
Yes (IFB)
|
No
|
|
HCA2 1X
|
No
|
No
|
Yes
|
No
|
Yes
|
|
HCA3 12X
|
No
|
Yes (IFB)
|
No
|
Yes (IFB3)
|
No
|
|
HCA3 1X
|
No
|
No
|
Yes
|
No
|
Yes
|
2.3 Fanout plugging
This section describes the fanout plugging rules for the zEnterprise and System z CPCs.
|
Note: For the fanout plugging rules for zEC12 or zBC12, refer to the IBM Redbooks documents IBM zEnterprise EC12 Technical Guide, SG24-8049, or IBM zEnterprise BC12 Technical Guide, SG24-8138.
|
2.3.1 Fanout plugging rules for zEnterprise 196
With the introduction of the zEnterprise CPCs, there are now six fanout types available. Depending on the number of I/O cages or drawers installed and the I/O domains in use, you have different numbers of fanout slots available for use as coupling or timing-only links. A maximum of 16 coupling link fanouts are supported for a z196.
Note that a z196 CPC that is fully populated for I/O connectivity has a maximum of 12 HCA slots remaining for coupling link fanouts. It has six I/O drawers or up to three I/O cages installed and uses up to 24 I/O domains. To connect each of the domains, you need 24 I/O interfaces, which can be provided by 12 I/O interface fanouts. This means that only 12 fanout slots are left to install coupling link fanouts (either HCA2-Os, HCA2-O LRs, HCA3-Os, HCA3-O LRs, or any combination of the four).
The fanout plugging rules vary and are dependent upon the z196 model. Figure 2-4 on page 23 can serve as a reference, but use the IBM configuration tool to determine the correct allocation.
Figure 2-4 z196 fanout positions
Positions D1 and D2 are not available on z196 models M66 and M80. For model M49, the positions D1 and D2 in the book positions LG10 and LG15 are not available.
For more information about this topic, see IBM zEnterprise 196 Technical Guide, SG24-7833.
2.3.2 Fanout plugging rules for zEnterprise 114
The z114 CPC has two hardware models. In the hardware model M05, only the first CPC drawer is installed with a maximum of four fanout slots. The hardware model M10 has both CPC drawers installed and provides up to eight fanout slots; see Figure 2-5 on page 24.
Depending on the number of installed I/O drawers, a different number of I/O interconnection fanouts are used. This can range from zero I/O interconnection fanouts for a dedicated stand-alone Coupling Facility model, through a Model M05 with a maximum of four legacy I/O drawers (where all four fanout slots are used for I/O interconnection fanouts), to a model M10 with three I/O drawers (where a maximum of six fanout slots will be used for I/O interconnection fanouts). So, depending on the model and the I/O connectivity that is required, there will be zero to eight fanout slots available to install coupling link fanouts (either HCA2-Os, HCA2-O LRs, HCA3-Os, HCA3-O LRs, or any combination of these).
Figure 2-5 zEnterprise 114 fanout positions
For more information, refer to IBM zEnterprise 114 Technical Guide, SG24-7954.
2.3.3 Fanout plugging rules for System z10 EC
System z10 supports three fanout types.
The HCA2-C fanout provides the I/O interfaces. Depending on the number of I/O cages installed and the I/O domains in use, you have different numbers of fanout slots available for use as coupling links. A fully populated CPC has three I/O cages installed and uses 21 I/O domains. To connect each of them, you need 24 I/O interfaces, which can be provided by 12 HCA2-C fanouts. That means a maximum of 12 fanout slots are available to install coupling link fanouts (either HCA2-Os, HCA2-O LRs, or MBAs, or any combination of the three).
Depending on the System z10 model, the plugging rules for fanouts vary. Figure 2-6 on page 25 can serve as a reference, but use the IBM configuration tool to determine the correct allocation.
Figure 2-6 System z10 EC fanout positions
Positions D1 and D2 are not available in z10 models E56 and E64. For model E40, the positions D1 and D2 in the book positions LG10 and LG15 are not available.
More information about the plugging rules for System z10 EC is available in IBM System z10 Enterprise Class Technical Guide, SG24-7516.
2.3.4 Fanout plugging rules for System z10 BC
The System z10 BC offers the possibility to work without any I/O drawers and can act as a dedicated Coupling Facility CPC without the need for HCA2-C fanouts. Depending on the number of installed I/O drawers, a different number of HCA2-C fanouts for I/O connections are needed.
A fully-populated CPC has four I/O drawers with a total of eight I/O domains installed and uses four fanout slots for I/O connections; see Figure 2-7. A CPC without an I/O drawer installed will have six fanouts available for coupling connections. So there will be between two and six fanout slots available to install coupling link fanouts (either HCA2-Os, HCA2-O LRs, or MBAs, or any combination of these).
Figure 2-7 System z10 BC fanout positions
For more information about the plugging rules for System z10 BC, see IBM System z10 Business Class Technical Overview, SG24-7632.
2.4 Adapter ID assignment and VCHIDs
This section describes the assignment of the Adapter IDs (AIDs) and explains the relationship between the AID, the virtual channel path identifier (VCHID), and the channel path identifier (CHPID). A solid understanding of these concepts facilitates the management and maintenance of PSIFB links from the HMC or SE.
|
Note: For adapter and VCHID information specific to zEC12 or zBC12, refer to the IBM Redbooks documents IBM zEnterprise EC12 Technical Guide, SG24-8049, or IBM zEnterprise BC12 Technical Guide, SG24-8138.
|
2.4.1 Adapter ID assignment
An adapter ID (AID) is assigned to every PSIFB link fanout at installation time. It is unique for the CPC. There is only one AID per fanout, so all ports on the fanout share the same AID. The adapter ID is:
•A number between 00 and 1F on z196, z10 EC, and z9 EC
•A number between 00 and 0B on z114
•A number between 00 and 05 on z10 BC
•A number between 08 and 0F on a z9 BC
In the input/output configuration program (IOCP) or hardware configuration definition (HCD), the AID and port number are used to connect the assigned CHPID to the physical location of the fanout.
There are distinct differences between zEnterprise systems, System z10, and System z9 for the assignment and handling of the AID; for example:
•For z196, the AID is bound to the serial number of the fanout. If the fanout is moved, the AID moves with it. For newly-built systems or newly-installed books, you can determine the AID from Table 2-3.
Table 2-3 Initial AID number assignment for zEnterprise 196
|
Adapter
location
|
Book
|
|||
|
|
Fourth
|
First
|
Third
|
Second
|
|
D1
|
00
|
08
|
10
|
18
|
|
D2
|
01
|
09
|
11
|
19
|
|
D3
|
N/A
|
N/A
|
N/A
|
N/A
|
|
D4
|
N/A
|
N/A
|
N/A
|
N/A
|
|
D5
|
02
|
0A
|
12
|
1A
|
|
D6
|
03
|
0B
|
13
|
1B
|
|
D7
|
04
|
0C
|
14
|
1C
|
|
D8
|
05
|
0D
|
15
|
1D
|
|
D9
|
06
|
0E
|
16
|
1E
|
|
DA
|
07
|
0F
|
17
|
1F
|
|
Note: The fanout positions D3 and D4 are reserved for Functional Service Processor (FSP) cards and cannot be used for fanouts. Also, positions D1 and D2 are not available in zEnterprise 196 models M66 and M80. For model M49, the positions D1 and D2 in the book positions LG10 and LG15 are not available.
|
•For zEnterprise 114, the AID is bound to the serial number of the fanout. If the fanout is moved, the AID moves with it. For newly-built systems you can determine the AID from Table 2-4.
Table 2-4 Initial AID number assignment for zEnterprise 114
|
Fanout position
|
D1
|
D2
|
D3
|
D4
|
D5
|
D6
|
D7
|
D8
|
|
CEC Drawer 1 AID
|
08
|
09
|
N/A
|
N/A
|
N/A
|
N/A
|
0A
|
0B
|
|
CEC Drawer 2 AID
|
00
|
01
|
N/A
|
N/A
|
N/A
|
N/A
|
02
|
03
|
|
Note: The fanout positions D3, D4, D5, and D6 are reserved for the Functional Service Processors and Oscillator cards and cannot be used for fanouts.
|
•For z10 EC, the AID is bound to the serial number of the fanout. If the fanout is moved, the AID moves with it. For newly-built systems or newly-installed books, you can determine the AID from Table 2-5.
Table 2-5 Initial AID number assignment for System z10 EC
|
Adapter
location
|
Book
|
|||
|
|
Fourth
|
First
|
Third
|
Second
|
|
D1
|
00
|
08
|
10
|
18
|
|
D2
|
01
|
09
|
11
|
19
|
|
D3
|
N/A
|
N/A
|
N/A
|
N/A
|
|
D4
|
N/A
|
N/A
|
N/A
|
N/A
|
|
D5
|
02
|
0A
|
12
|
1A
|
|
D6
|
03
|
0B
|
13
|
1B
|
|
D7
|
04
|
0C
|
14
|
1C
|
|
D8
|
05
|
0D
|
15
|
1D
|
|
D9
|
06
|
0E
|
16
|
1E
|
|
DA
|
07
|
0F
|
17
|
1F
|
|
Note: The fanout positions D3 and D4 are reserved for Functional Service Processor (FSP) cards and cannot be used for fanouts. Also, positions D1 and D2 are not available in System z10 models E56 and E64. For model E40 the positions D1 and D2 in the book positions LG10 and LG15 are not available.
|
•For z10 BC, the AID is bound to the serial number of the fanout. If the fanout is moved, the AID moves with it. For newly-built systems, you can determine the AID from Table 2-6.
Table 2-6 Initial AID number assignment for System z10 BC
|
Fanout position
|
D1
|
D2
|
D3
|
D4
|
D5
|
D6
|
D7
|
D8
|
D9
|
DA
|
|
AID
|
N/A
|
N/A
|
00
|
01
|
02
|
03
|
04
|
05
|
N/A
|
N/A
|
|
Note: The fanout positions D1, D2, D9, and DA are reserved for the Functional Service Processors and Oscillator cards and cannot be used for fanouts.
|
•The AID for z9 is bound to the physical fanout position. If the fanout is moved to another slot, the AID changes for that specific fanout, and it might be necessary to adjust the input/output configuration data set (IOCDS).
The Physical Channel ID (PCHID) Report lists the assignments of the AIDs for new CPCs or miscellaneous equipment specification (MES) upgrades, and in the Hardware Management Console (HMC) and Support Element (SE) panels after installation. See 3.7, “Physical and logical coupling link capacity planning” on page 50 for an example of a PCHID Report.
2.4.2 VCHID - Virtual Channel Identifier
A physical channel identifier (PCHID) normally has a one-to-one relationship between the identifier and a physical location in the machine; see Figure 2-8 for an example.
Figure 2-8 The PCHID refers to the physical location
However, a PCHID in the range from 0700 to 07FF lacks the one-to-one relationship between the identifier and the physical location, either because they do not have a physical card (like Internal Coupling connections (ICPs)), or because they are administered through different identifiers (as for PSIFB links, with the AIDs). No one-to-one relationship is possible due to the capability to define more than one CHPID for a physical location. Therefore, these are sometimes referred to as Virtual Channel Path Identifiers (VCHIDs). Note that the SE and HMC still refer to these as PCHIDs; see Figure 2-9 for an example.
Figure 2-9 The VCHID refers to the physical location of the HCA and Port
VCHIDs for IC channels have been implemented for several generations of System z CPCs. However, prior to the introduction of PSIFB links, there was no requirement for the operators to interact with the VCHID objects on the SE. With the introduction of PSIFB, there might now be situations where you have to query or manage the VCHID objects.
To manage PSIFB links, the SE and HMC tasks have been changed to handle the VCHIDs that support the physical hardware. In the SE Channel task, the VCHID for every CHPID that is associated with a PSIFB link is shown, and all traditional channel operations can be carried out against each VCHID. The AID and port that the VCHID is assigned to can be found under the channel details for each VCHID (see Figure 2-9 for an example).
VCHIDs are assigned automatically by the system and are not defined by you in the IOCDS. A VCHID is also not permanently tied to an AID. Therefore the VCHID assignment can change after a Power-On Reset (POR) if the hardware configuration changed (if an HCA was added or removed, for example). Due to the automatic assignment of the VCHID at every POR, the client or SSR needs to make sure that the correlation for the channel that they intend to manipulate has not changed. The VCHID that is currently associated with a coupling CHPID can be found by issuing an MVS D CF,CFNM=xxxx command for the associated CF.
2.5 InfiniBand coupling links
This section discusses the various PSIFB coupling links that are available for each individual zEnterprise and System z CPC from the CPC point of view.
|
Note: For coupling link information specific to zEC12 or zBC12, refer to the IBM Redbooks documents IBM zEnterprise EC12 Technical Guide, SG24-8049, or IBM zEnterprise BC12 Technical Guide, SG24-8138.
|
2.5.1 12X PSIFB coupling links on System z9
|
Note: At the time of writing, upgrades for System z9 CPCs have been withdrawn from marketing.
|
An HCA1-O fanout is installed in a fanout slot in the front of a z9 book and takes an empty slot or the place of one of the previous MBAs. It supports the 12X IB-SDR link option and is used to connect a System z9 to a zEnterprise or a System z10. The fanout has two optical MPO ports and the order increment for the HCA1-O fanout is always one complete fanout with both ports enabled.
The point-to-point coupling link connection is established by connecting the HCA1-O fanout to a zEnterprise or System z10 HCA2-O fanout through a 50-micron OM3 (2000 MHz-km) multimode fiber optic cable. A HCA1-O fanout can only be connected to a HCA2-O fanout. The cable contains 12 lanes (two fibers per lane, one each for transmit and receive); 24 fibers in total. The maximum supported length for these connections is 150 meters (492 feet). The link bandwidth is 3 GBps for a single data rate connection (SDR) and is auto-negotiated.
Each HCA1-O fanout has an AID that is bound to the physical position in which it is installed. That means that if you move the fanout to another position, the AID changes and you need to adjust the AID in the IOCP or HCD. See 2.4, “Adapter ID assignment and VCHIDs” on page 26 for more information.
It is possible to define up to 16 CHPIDs per fanout (AID), which can be freely distributed across both ports.
|
Important: For optimal performance, define no more than four CIB CHPIDs per port.
|
A maximum of 8 HCA1-O fanouts per book are supported on a System z9, providing a total of 16 ports. Regardless of how many fanouts are installed, the maximum combined total of 64 CF link CHPIDs per CPC applies.
|
Note: HCA1-O adapters cannot be carried forward on an upgrade to a z10 or z196. Upgrades to either of those CPCs requires replacing the HCA1 adapters with HCA2 or HCA3 adapters.
|
2.5.2 12X PSIFB coupling links on System z10
An HCA2-O fanout is installed in a fanout slot in the front of the System z10 CPC cage (or CPC drawer for the z10 BC). It supports the 12X IB-DDR link option. This link connects to a zEnterprise, a System z10, or a System z9 CPC, in a point-to-point coupling link connection. The fanout has two optical MPO ports (see Figure 2-2 on page 20). The order increment for the HCA2-O fanout is always one complete fanout with both ports enabled.
The connection is established by connecting the HCA2-O to the other system’s PSIFB fanout (either an HCA1-O for a System z9, a HCA2-O for a zEnterprise or a System z10, or a HCA3-O for a zEnterprise) through a 50-micron OM3 (2000 MHz-km) multimode fiber optic cable. The cable contains 12 lanes (two fibers per lane, one each for transmit and receive); 24 fibers in total. The maximum supported length for these connections is 150 meters (492 ft). The maximum bandwidth is 6 GBps for a double data rate connection (DDR) or 3 GBps for a single data rate connection (SDR) and is auto-negotiated.
Each HCA2-O fanout has an AID, which is bound to the HCA serial number. See 2.4, “Adapter ID assignment and VCHIDs” on page 26 for more information.
It is possible to define up to 16 CHPIDs per fanout (AID), which can be freely distributed across both ports.
|
Important: For optimal performance, define no more than four CIB CHPIDs per port.
|
A maximum of 16 fanouts for InfiniBand coupling are supported on a z10 EC. All 16 can be HCA2-O fanouts providing a total of 32 ports. On a z10 BC, a maximum of 6 HCA2-O fanouts are supported, providing a total of 12 ports. Regardless of the number of fanouts installed or used, the maximum of 64 CF link CHPIDs per CPC still applies (including IC, Inter-Cluster Bus-4 (ICB4), active InterSystem Coupling Facility-3 (ISC3), and PSIFB).
|
Note: The InfiniBand link data rate of 6 GBps or 3 GBps does not represent the performance of the link. The actual performance depends on many factors, such as latency through the adapters, cable lengths, and the type of workload.
|
2.5.3 PSIFB Long Reach coupling links on System z10
An HCA2-O LR fanout is installed in a fanout slot in the front of the z10 CPC cage (or CPC drawer for the z10 BC). It supports the 1X IB-DDR LR link option. This link connects either to a zEnterprise or to a z10 CPC in a point-to-point coupling link connection. The fanout has two optical SFP ports, and the order increment for the HCA2-O LR fanout is always one complete fanout with both ports enabled.
The connection is established by connecting the HCA2-O LR to the other system’s HCA2-O LR or HCA3-O LR port through a 9-micron single mode fiber optic cable. The cable contains one lane with one fiber for transmit and one for receive. The maximum supported length for these connections is 10 km1 (6.2 miles) unrepeated and 175 km (108 miles) when repeated through a DWDM. The maximum bandwidth is 5 Gbps for a double data rate connection (DDR) or 2.5 Gbps for a single data rate connection (SDR) and is auto-negotiated.
Each HCA2-O LR fanout has an AID, which is bound to the HCA serial number. See 2.4, “Adapter ID assignment and VCHIDs” on page 26 for more information.
It is possible to define up to 16 CHPIDs per fanout (AID), which can be freely distributed across both ports.
|
Important: For optimal performance, it is best to avoid defining more than four CHPIDs per port. However, if the link is being used to provide connectivity for sysplexes with low levels of Coupling Facility activity over greater distances, it might be acceptable to assign more than four CHPIDs per port to be able to utilize a greater number of subchannels.
|
A maximum of 16 fanouts for coupling are supported on a z10 EC. All 16 can be HCA2-O LR fanouts providing a total of 32 ports. On a z10 BC, a maximum of 6 HCA2-O LR fanouts are supported, providing a total of 12 ports. Nevertheless, the maximum value of 64 CF link CHPIDs per system (including IC, Inter-Cluster Bus-4 (ICB-4), active InterSystem Coupling Facility-3 (ISC-3), and PSIFB) still applies.
|
Note: The InfiniBand link data rate of 5 Gbps or 2.5 Gbps does not represent the performance of the link. The actual performance depends on many factors, such as latency through the adapters, cable lengths, and the type of workload.
|
2.5.4 12X PSIFB coupling links on z196 and z114
An HCA3-O or HCA2-O fanout is installed in a fanout slot in the front of the z196 CPC cage (or CPC drawer for the z114). It supports the 12X IB-DDR link option. This link connects to a zEnterprise, a System z10, or a System z9 CPC, in a point-to-point coupling link connection. The fanout has two optical MPO ports (see Figure 2-2 on page 20), and the order increment for the HCA3-O or HCA2-O fanout is always one complete fanout with both ports enabled.
The connection is established by connecting the HCA3-O or HCA2-O to the other system’s PSIFB fanout (either an HCA1-O for a z9, a HCA2-O for a zEnterprise or z10, or a HCA3-O for a zEnterprise) through a 50-micron OM3 (2000 MHz-km) multimode fiber optic cable. The cable contains 12 lanes (two fibers per lane, one each for transmit and receive); 24 fibers in total. The maximum supported length for these connections is 150 meters (492 ft). The maximum bandwidth is 6 GBps for a double data rate connection (DDR) or 3 GBps for a single data rate connection (SDR) and is auto-negotiated.
With the introduction of the HCA3-O fanout, it is possible to utilize an improved IFB protocol which is called IFB3. This new protocol provides improved service times for the PSIFB 12X link. However, certain conditions must be met to utilize it:
•The IFB3 protocol will only be used when both ends of the link are connected to an HCA3-O fanout.
•The IFB3 protocol will only be used if a maximum of four CHPIDs are defined per HCA3-O fanout port for all logical partitions (LPAR) combined.
For example, IFB3 mode will be used in the following situations:
– Four CHPIDs are assigned to a HCA3-O port, and all four CHPIDs are shared across z/OS LPARs that are in the same sysplex.
– Four CHPIDs are assigned to a HCA3-O port, and each CHPID is in use by a different sysplex.
– Four CHPIDs are assigned to a HCA3-O port. The CHPIDs are defined as SPANNED, and are shared across z/OS LPARs in multiple CSSs.
IFB3 mode will not be used in the following cases:
– More than four CHPIDs are assigned to the port.
– More than four CHPIDs are assigned to the port, but some of the CHPIDs are offline, bringing the number of online CHPIDs below five. The port will still run in IFB mode.
The PSIFB link will automatically detect if the given requirements are met and will auto-negotiate the use of the IFB3 protocol. The two ports of an HCA3-O fanout are able to work in different protocol modes. It is possible to determine from the Support Element which protocol is currently being used on any given HCA3-O port. See “Analyze Channel Information option” on page 226 for more information.
|
Important: In the case where a dynamic I/O configuration change results in an IFB protocol mode change on an HCA3-O port, the physical port will automatically perform a reinitialization. This will result in all defined CHPIDs on this port being toggled offline concurrently and then back online. As a result, all connectivity to any connected Coupling Facilities and STP through this port will be lost for a short period of time.
This means that you must ensure that all your CFs are connected through at least two physical links, and that any change you make is not going to affect more than one port.
|
Each HCA3-O or HCA2-O fanout has an AID, which is bound to the HCA serial number. See 2.4, “Adapter ID assignment and VCHIDs” on page 26 for more information.
It is possible to define up to 16 CHPIDs per fanout (AID), which can be freely distributed across both ports. For optimum performance, especially when using HCA3-O links, do not define more than four CHPIDs per port. However, if the link is being used to provide connectivity for sysplexes with low levels of Coupling Facility activity, it might be acceptable to assign more than four CHPIDs per port.
A maximum of 16 fanouts for coupling are supported on a z196. All 16 can be HCA3-O fanouts, HCA2-O fanouts, or a mix of both, providing a total of 32 ports. On z114, a maximum of eight HCA3-O fanouts, HCA2-O fanouts, or a mix of both, is supported, providing a total of 16 ports. Even though the maximum number of CF link CHPIDs was raised to 128 for zEnterprise systems, remember that this includes IC, InterSystem Coupling Facility-3 (ISC3), and PSIFB connections.
|
Note: The InfiniBand link data rate of 6 GBps or 3 GBps does not represent the performance of the link. The actual performance depends on many factors, such as latency through the adapters, cable lengths, and the type of workload.
|
2.5.5 Long Reach PSIFB coupling links on zEnterprise 196 and 114
An HCA3-O LR or HCA2-O LR fanout is installed in a fanout slot in the front of the z196 CPC cage (or CPC drawer for the z114). It supports the 1X IB-DDR LR link option. This link connects to either a zEnterprise or a z10 CPC in a point-to-point coupling link connection. The fanout has four optical SFP ports for HCA3-O LR or two optical SFP ports for HCA2-O LR, and the order increment for the HCA3-O LR fanout is always one complete fanout with all ports enabled. The HCA2-O LR fanout can no longer be ordered for a zEnterprise CPC. It can only be carried forward through an MES from a z10 or a z196 at Driver level 86.
The connection is established by connecting the HCA3-O LR or HCA2-O LR to the other system’s HCA3-O LR or HCA2-O LR port through a 9-micron single mode fiber optic cable. The cable contains one lane with one fiber for transmit and one for receive. The maximum supported length for these connections is 10 km2 (6.2 miles) unrepeated and 175 km (108 miles) when repeated through a DWDM. The maximum bandwidth is 5 Gbps for a double data rate connection (DDR) or 2.5 Gbps for a single data rate connection (SDR) and is auto-negotiated.
Each HCA3-O LR and HCA2-O LR fanout has an AID, which is bound to the HCA serial number. See 2.4, “Adapter ID assignment and VCHIDs” on page 26 for more information.
It is possible to define up to 16 CHPIDs per fanout (AID), which can be freely distributed across both ports.
An overall maximum of 16 coupling link fanouts is supported on a z196 and 12 of those can be long reach coupling link fanouts. Therefore, the maximum number of up to 48 long reach coupling ports can be reached if all 12 long reach fanouts installed are HCA3-O LRs.
On z114, a maximum of eight HCA3-O LR fanouts, HCA2-O LR fanouts, or a mix of both, is supported, providing a total of up to 32 ports.
Even though the maximum number of CF link CHPIDs was raised to 128 for zEnterprise systems, it still has to be taken into consideration because it includes IC, InterSystem Coupling Facility-3 (ISC3), and PSIFB connections.
|
Note: The InfiniBand link data rate of 5 Gbps or 2.5 Gbps does not represent the performance of the link. The actual performance depends on many factors, such as latency through the adapters, cable lengths, and the type of workload.
|
2.5.6 PSIFB coupling links and Server Time Protocol
External PSIFB coupling links can also be used to pass time synchronization signals using Server Time Protocol (STP). Therefore, you can use the same coupling links to exchange both time synchronization information and Coupling Facility messages in a Parallel Sysplex. See 3.4, “Considerations for Server Time Protocol” on page 45 for more information.
Note that all the links between a given pair of CPCs must be defined as coupling links or as timing-only links; you cannot have a mix of coupling and timing-only links between one pair of CPCs.
|
Note: To avoid a single point of failure, use at least two physical connections for all CPCs that are connected using InfiniBand, and spread those connections over multiple adapters (different AIDs).
|
2.6 InfiniBand cables
Two cable types are used for PSIFB connections in the zEnterprise, System z10 and System z9 environment.
|
Note: Fiber optic cables and cable planning, labeling, and placement are client responsibilities for new installations and upgrades.
|
The cable types are:
•Standard 9 µm single mode fiber optic cable (see Figure 2-10 on page 35) with LC Duplex connectors for PSIFB LR (1X) links.
Figure 2-10 Single mode fiber optic cable with LC Duplex connectors
•The 50-micron OM3 (2000 MHz-km) multimode fiber cable with MPO connectors (see Figure 2-11) for PSIFB 12X links.
The 50-micron OM3 fiber cable is an InfiniBand Trade Association (IBTA) industry standard cable. It has one pair of fibers per lane (24 fibers in total) for a 12X connection.
Figure 2-11 Optical InfiniBand cable, including TX and RX labels
The sender and receiver are clearly marked with either RX or TX, and the connectors are keyed. Also, on all field-replaceable units (FRUs) using IFB optical modules, the keys face upward, the Transmitter module (TX) is on the left side, and the Receiver module (RX) is on the right side.
To avoid problems or lengthy problem determination efforts, use the IBM part numbers to order cables that are designed to provide 99.999% availability.
|
Important: The fiber optic modules and cable connectors are sensitive to dust and debris. Component and cable dust plugs must always cover unused ports and cable ends.
|
1 Refer to RPQ 8P2340 for information about extending the unrepeated distance to 20 km.
2 Refer to RPQ 8P2340 for information about extending the unrepeated distance to 20 km.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.