This appendix introduces the native Peripheral Component Interconnect Express (PCIe) adapters management on IBM Z servers (IBM z14, z13, and z13s)¹. The appendix also includes concepts of the integrated firmware processor (IFP) and resource groups (RG).

This appendix includes the following topics:

 

The native PCIe adapter is a new category of features that was introduced in zEC12. IBM z14 ZR1 supports the following native PCIE features:

These adapters are installed into a PCIe+ I/O drawer and are identified by a physical channel ID (PCHID) that is assigned according to the physical location.

For adapters that are installed in a PCIe+ I/O drawer, management functions in the form of device drivers and diagnostic tools are always implemented to support virtualization of the adapter, service, and maintenance.

Traditionally, these management functions are integrated on the adapter with specific hardware design (FICON Express, OSA-Express). For most of the newly introduced native PCIe adapters, these functions are moved out of the adapter and are now handled by an Integrated Firmware Processor (IFP) which runs the Resource Group (RG) firmware.

For traditional I/O adapters, such as the Open Systems Adapter (OSA) and Fibre Channel connection (FICON) cards, the application-specific integrated circuit (ASIC) chip on the adapter always downloads the device drivers and diagnostic tools from the Support Element (SE) and runs the management functions on the adapter. With the native PCIe adapter design, no ASIC chip is used for management (including virtualization) function on the native PCIe adapters.

For the RoCE Express, Coupling Express Long Reach, and zEDC, device drivers and diagnostic tools are now running on the IFP and use four RGs. Management functions, including virtualization, servicing and recovery, diagnostics, failover, firmware updates against an adapter, and other functions are implemented within the RG microcode. For the zHyperlink Express adapters, only service (diagnostics) and firmware maintenance are handled for the Resource Group microcode.

The IFP is a processor unit (PU) that is exclusively used to manage native PCIe adapters that are installed in the PCIe+ I/O drawer. It is allocated from the system PU pool and is not counted in the PUs available for characterization.

If a native PCIe adapter is installed in the system, the system allocates and initializes an IFP during its power-on reset (POR) phase. Although the IFP is allocated to one of the physical PUs, it is not visible to the users. In an error or failover scenario, PU sparing also applies for IFP, with the same rules as other PUs.

The IFP allocates four resource groups for running the management functions of native PCIe adapters. A native PCIe adapter is managed by one of the resource groups according to the adapter location in the PCIe+ I/O drawer.

As shown in Figure C-1, each I/O domain in a PCIe+ I/O drawer of a z14 ZR1 server is driven by a PCIe switch that is connected to the PCIe I/O feature that is in the CPC drawer. Each slot in a PCIe_ I/O drawer is attached to one resource group. The native PCIe I/O adapters are managed by their respective RG for device drivers and diagnostic tools functions.

Up to four PCIe+ I/O drawers are supported on z14 ZR1 servers. The same type of native PCIe adapter is always assigned to different I/O domains and in different resource groups (and different PCIe I/O drawers if the configuration includes them) to eliminate the possibility of a single point of failure.

The IFP and resource groups perform the following management tasks on the native PCIe adapters:

The firmware of native PCIe adapters and resource groups is part of the system’s microcode and can be updated by a Microcode Change Level (MCL) upgrade. MCL upgrades on adapters or on the code of the resource groups require the specific adapter or all native PCIe adapters that are managed by the specific resource group (depending on the type of microcode that it applied) to be offline during activation of the MCL.

However, to maintain availability, MCLs can be applied to only one resource group at a time. While one resource group is offline, the other three resource groups and all their adapters remain active. An MCL application for a native PCIe adapter or resource group is not possible if an error condition exists within the other resource groups.

If an error occurs in one of the resource groups or features that are assigned to one of the resource groups, the IFP manages error recovery and collects error data. The error data is sent by the IFP to the SE, which then provides a message on the SE and the Hardware Management Console (HMC). If an error requires maintenance, a call home to the IBM Support system is started by the HMC.

Any maintenance of a native PCIe feature is managed by the IFP, including testing or replacing a feature card. Before a feature is configured offline, the IFP ensures that the same type of feature is available in the same or one of the other resource groups (if applicable).

The following maximum number of native PCIe adapters can be installed in a z14 ZR1 server:

Considering availability, install adapters of the same type in slots of different I/O domains, drawers, fanouts, and resource groups.

The next sections provide more information about achieving a highly available configuration.

During the ordering process of the native PCIe adapters, such as the zEDC Express and RoCE Express/Express2, adapters of the same type are evenly spread across resource groups 1, 2, 3, and 4 for availability and serviceability.

A sample PCHID report of a z14 ZR1 configuration with four zEDC Express adapters and four 10GbE RoCE Express2 adapters is shown in Figure C-2 on page 423. The following information is listed for each adapter:

The native PCIe adapters are not part of the traditional channel subsystem (CSS). Although they do not include a channel-path identifier (CHPID) assigned, they do include a PCHID that is assigned according to their physical location in the PCIe+ I/O drawer.

To define the native PCIe adapters in the HCD or HMC, a new I/O configuration program (IOCP) FUNCTION statement is introduced that includes several feature-specific parameters.

The IOCP example that is shown in Figure C-3 defines zEDC Express and 10GbE RoCE Express2 adapters to LPARs LP14 and LP15.

The FUNCTION identifier (FID) is a hexadecimal number 000 - FFF that you use to assign a PCHID to the FUNCTION to identify the specific adapter in the PCIe+ I/O drawer. Because the FUNCTION is not related to a channel subsystem, all LPARs on a central processor complex (CPC) can be defined to it. However, a FUNCTION cannot be shared between LPARs. It is only dedicated or reconfigurable by using the PART parameter. The TYPE parameter is required for z14 ZR1.

If you want several LPARs to use a zEDC Express or RoCE Express or Express2 adapter, you must use a Virtual Function (VF) number. A VF number is a number 1 - nnn, where nnn is the maximum number of LPARs that the feature supports. The maximum is 15 for the zEDC Express feature, 31 for the 10GbE RoCE Express (FC 0411), 62 for the RoCE Express2 adapter (31 VFs per physical port), and 254 for the zHyperLink Express feature⁴.

The physical network ID (PNETID) is required to set up the Shared Memory Communications over Remote Direct Memory Access (SMC-R) communication between two RoCE Express or Express2 features. Each FUNCTION definition supports up to four PNETIDs.