Central processor complex hardware components
This chapter introduces the z14 ZR1 central processor complex (CPC) hardware components. It also describes the significant features and functions with their characteristics and options.
This chapter describes the z14 ZR1 hardware building blocks and how these components interconnect.
This chapter includes the following topics:
2.1 System overview: Frame and drawers
The IBM z14 Model ZR1 system is designed in an industry standard 19-inch form factor rack (frame) that can be easily installed in any data center. The design uses power distribution unit (PDU)-based power along with redundant power, cooling, and power cords.
Redesigned CPC drawer I/O infrastructure also lowers power costs, reduces the footprint, and allows installation in virtually any data center. The z14 ZR1 server is rated at ASHRAE class A31data center operating environment.
The system is designed with one CPC Drawer and up to four new Peripheral Component Interconnect Express Generation 3 (PCIe Gen3) I/O drawers (named PCIe+ I/O drawers). The components that are included in the rack are described in the following sections.
The z14 ZR1 server is an air-cooled system (see Figure 2-1) without front covers and doors.
Figure 2-1 z14 ZR1 server
The single rack includes the following major components (from top to bottom of the rack):
•Two redundant Support Element (SE) 1U servers that are installed at the top of the rack.
•Two redundant Ethernet switches that provide external and internal communications to manage the server.
•Up to four new PCIe+ I/O drawers. The number of PCIe+ I/O drawers depends on the I/O configuration of the z14 ZR1 server.
•A pullout Keyboard, Mouse, Monitor (KMM) tray (new design for z14 ZR1), accessible at the front of the rack. The KMM devices are connected to an internal Keyboard, Video, Mouse (KVM) switch that is mounted in the rear of the rack, which is used to alternate the console between the two Support Elements.
•A new CPC drawer, which houses the PU and SC Single Chip Modules (SCMs), Memory, PCIe fanouts for I/O drawer connectivity, ICA SR adapters, and the necessary power elements and cooling fans.
•Depending on the configuration, two or four intelligent Power Distribution Units (PDUs) are mounted vertically on each side at the rear of the rack. All of the internal components receive their power from these intelligent PDUs. The PDUs are cabled for redundancy.
•A newly designed vertical cable management guide (“spine”) can assist with proper cable management for fiber, copper, and coupling cables. The spine is shipped with configurations that contain three or four PCIe+ I/O drawers or with the 16U Reserved feature (FC 0617). All external cabling to the system (from top or bottom) can use the spine to minimize interference with the PDUs mounted on the sides of the rack.
•The z14 ZR1 server (as its predecessor, the z13s) has the option of ordering the infrastructure to support the top and bottom exit of fiber optic and copper cables.
The rack with the spine mounted is shown in Figure 2-2. It and includes removable hooks that can be placed in appropriate slots throughout the length of the spine.
Figure 2-2 Cable management spine
2.1.1 z14 ZR1 configurations
All system components are designed and integrated in an IBM 19-inch frame (rack)2. The possible configurations of the system (rear view) are shown in Figure 2-3 on page 24. Consider the following points:
•The z14 ZR1 system is built in a 42 EIA units rack (A-frame). The base rack is 40 EIA units high with a 2U removable top. FC 9975 is available if a height reduction is necessary.
•PCIe+ I/O drawers are provided as required by the number of I/O adapters ordered. The PCIe+ I/O drawers (1, 2, 3, and 4) and are installed in order in the following EIA locations: A14B, A01B, A23B, and A31B.
•The five possible configurations without FC 06173 with the view from the rear of the system (from left to right in Figure 2-3):
– The first configuration can be a coupling facility with coupling fanouts that are installed in the CPC Drawer only.
– The second configuration includes a single PCIe+ I/O drawer that is installed at EIA A14B and along with Configuration 1, requires only the upper pair of PDUs, which are mounted vertically at the rear of the system in columns N and Z.
– The third configuration includes the second PCIe+ I/O drawer that is installed at EIA A01B and now requires the extra lower pair of PDUs installed to provide necessary power.
– The fourth configuration includes the third PCIe+ I/O drawer that is installed at EIA A23B.
– The fifth configuration includes the fourth PCIe+ I/O drawer that is installed at EIA A31B (maximum I/O configuration).
Figure 2-3 ZR1 Rack configurations: Rear view (without FC 0617)
2.1.2 PCIe+ I/O drawer
A new PCIe+ I/O drawer is introduced with the z14 ZR1 (Machine Type 3907). The new drawer is shown in Figure 2-4 on page 25. Each PCIe+ I/O drawer (up to four) has 16 PCIe slots each to support the PCIe I/O infrastructure with a bandwidth of 16 GBps and includes the following features:
•Two I/O domains (0 and 1), each capable of hosting eight PCIe adapters, for a total of 16 I/O adapters.
•Two PCIe switch cards provide connectivity to the PCIe fanouts that are installed in the CPC drawer. Each I/O domain features the PCIe slots that are allocated over two PCIe support partitions to manage the native PCIe adapters.
•Two Flexible Support Processor (FSP) cards that are used to control the drawer.
•Two Power Supply Units (PSUs) in a redundant configuration.
•Six hot-swappable cooling fan modules at the front of the drawer.
Figure 2-4 PCIe+ I/O drawer front and rear view
PCIe I/O infrastructure
|
Terminology: Throughout this chapter, the terms adapter and card are interchangeable and refer to a feature that is installed in a PCIe+ I/O drawer.
|
The PCIe I/O infrastructure uses the PCIe fanouts that are installed in the processor (CPC) drawer to connect to the PCIe+ I/O drawer. The PCIe adapters include the following features:
•FICON Expresss16S+ (two port card), long wavelength (LX) or short wavelength (SX), which contains two physical channel IDs (PCHIDs)
•FICON Express16S (two port card), long wavelength (LX) or short wavelength (SX), which contain two PCHIDs (only for carry-forward MES)
•FICON Express8S (two port card), long wavelength (LX) or short wavelength (SX), which contain two PCHIDs (only for carry-forward MES)
•Open Systems Adapter (OSA)-Express7S 25GbE Short Reach (SR) - New feature
•Open System Adapter (OSA)-Express6S:
– OSA-Express6S 10 Gb Ethernet (single port card, Long Reach (LR) or Short Reach (SR), one PCHID)
– OSA-Express6S Gb Ethernet (two port card, LX or SX, one PCHID)
– OSA-Express6S 1000BASE-T Ethernet (two port card, RJ-45, one PCHID)
•Open System Adapter (OSA)-Express5S and 4S features (for carry-forward MES only):
– OSA-Express5S and 4S 10 Gb Ethernet (one port card, LR or SR, and one PCHID)
– OSA-Express5S and 4S Gb Ethernet (two port card, LX or SX, and one PCHID)
– OSA-Express5S 1000BASE-T Ethernet (two port card, RJ-45, and one CHPID)
•Crypto Express6S (new build) and Crypto Express5s (only for carry-forward MES). Each feature holds one PCIe cryptographic adapter. Each adapter can be configured as:
– Secure IBM Common Cryptographic Architecture (CCA) coprocessor
– Secure IBM Enterprise Public Key Cryptography Standards (PKCS) #11 (EP11) coprocessor
– Accelerator
•zHyperLink Express is a two-port card that is directly connected to storage subsystem controller (supported DS8880 series) that is designed to increase the scalability of IBM Z transaction processing through increased throughput and lower I/O latency. This feature is installed in the PCIe+ I/O drawer.
•Coupling Express LR (CE LR) is a two-port card that is used for long-distance coupling connectivity and uses CL5 coupling channel type. The card uses 10GbE RoCE technology and is designed to drive distances up to 10 km (6.21 miles) and support a link data rate of 10 Gigabits per second (Gbps). This feature is installed in the PCIe+ I/O drawer. Each port supports up to four CHPIDs per port and use point-to-point connectivity only.
•New feature - 25GbE RoCE Express2 - Remote Direct Memory Access (RDMA) over Converged Ethernet. The next generation RoCE with improved virtualization scale, better performance, and RAS. It is a two-port card and supports up to 31 virtual functions per port and 62 VFs per PCHID (feature)
•10GbE RoCE Express2 - Remote Direct Memory Access (RDMA) over Converged Ethernet. The next generation RoCE with improved virtualization scale, better performance, and RAS. It is a two-port card and supports up to 31 virtual functions per port and 62 VFs per PCHID (feature).
•10GbE RoCE Express - Remote Direct Memory Access (RDMA) over Converged Ethernet (only for a carry-forward MES). It is a two-port card and supports up to 31 virtual functions per adapter.
•zEnterprise Data Compression (zEDC) Express. The zEnterprise Data Compression Express card occupies one I/O slot, but it does not have a CHPID type. Up to 15 partitions can share the feature concurrently.
2.2 16U Reserved feature (FC 0617)
The z14 ZR1 can be ordered with FC 0617 (16U Reserved), which provides a Fit-for-Purpose solution in a single footprint by allowing z14 ZR1 configurations with two or less PCIe+ I/O drawers the use of open space in the rack for non-Z hardware4.
The 16U Reserved feature allows clients to create all-in-one solutions to run their entire business or independent application or cloud solutions. Consider the following points:
•FC 0617 is “16U Reserved” space in the rack for non-Z equipment (components) to be installed in the Z rack. It allows for clients to integrate other hardware into the single 19-inch rack, which reduces data center footprint requirements.
•Reserved space can be especially useful for space-constrained data centers or to keep rack-mounted HMC and other rack-mounted equipment together with the Z server.
EIA units 22 - 38 space is designated for the 16U Reserved (FC 0617), as shown in Figure 2-5.
Figure 2-5 Possible configurations with the 16U Reserved feature
For more information about the 16U Reserved feature, see Appendix G, “16U Reserved feature” on page 465.
2.3 CPC drawer
The z14 ZR1 server continues the design of z13s by packaging processors in drawers. Unlike z13s, the processor drawer was designed to fit in the 19-inch rack. The single z14 ZR1 Central Processor Complex (CPC) drawer includes the following features:
•Up to four Processor Unit (PU) Single Chip Modules (SCMs)
•One System Control (SC) SCM
•Memory DIMMs (up to four banks or five DIMMs each)
•Connectors to support PCIe+ I/O drawers (through PCIe fanout hubs)
•Coupling links to other CPCs by using the Integrated Coupling Adapter Short Reach fanout
The z14 ZR1 CPC drawer and its components are shown in Figure 2-6.
Figure 2-6 z14 ZR1 CPC drawer layout details
The z14 ZR1 CPC drawer size is determined by the number of PU SCMs that is installed and is feature-driven (four CPC drawer size features).
Depending on the CPC drawer feature, the CPC drawer contains the following components:
•Four CPC Drawer size configurations (CPC drawer has always ONE SC SCM):
– FC 0636 = 1 PU SCM (maximum 4 characterizable PUs)
– FC 0637 = 2 PU SCMs (maximum 12 characterizable PUs)
– FC 0638 = 4 PU SCMs (maximum 24 characterizable PUs)
– FC 0639 = 4 PU SCMs (maximum 30 characterizable PUs)
The following SCMs are available:
– PU SCM uses 14nm SOI technology, 17 layers of metal, 14.4 miles of wire, core running at 4.5GHz: 10 PUs/SCM (with 5, 6, 7, 8, 9 active cores).
– System Controller (SC) SCM, 17 layers of metal, 13.8 miles of wire, 672 MB L4 cache.
•Each PU SCM has one memory controller that drives five DDR4 dual inline memory modules (DIMMs), for a maximum of 20 DIMMs per drawer.
DIMMs are plugged into 5, 10, 15, or 20 DIMM groups, which provides 160 - 10240 GB of physical memory (RAIM protected) that results in 128 - 8192 GB of addressable memory (64 - 8128 customer usable memory).
•Up to eight PCIe Generation 3 I/O slots that can host two, four, or eight PCIe Gen3 x16 fanouts (16 GBps bandwidth) populated by:
– PCIe Gen3 I/O fanout for PCIe+ I/O drawer (always ordered and used in pairs for availability)
– ICA SR PCIe fanout for coupling
•Management elements:
– Two flexible service processor (FSP) cards for system control (N+1 redundancy).
– Two Oscillator cards to provide system clocking (N+1 redundancy)
•CPC drawer power infrastructure consists of:
– Two or four Power Supply Units (PSUs) that provide power to the CPC drawer. The loss of one power supply leaves enough power to satisfy the drawer’s power requirements (N+1 redundancy). The power supplies can be concurrently removed and replaced (one at a time).
– Three to six Voltage Regulator Modules that plug next to the memory DIMMs.
– Two Power Control cards to control the five CPC fans at the front of the CPC drawer.
The front view of the CPC drawer, which includes the cooling fans and power control cards, is shown in Figure 2-7.
Figure 2-7 Front view of the CPC drawer
The rear view of a fully populated CPC Drawer is shown in Figure 2-8. PCIe I/O fanouts are plugged in specific slots for best performance and availability. Redundant FSP cards (two) and Oscillator cards (two) are always installed.
Figure 2-8 Rear view of a fully populated CPC Drawer
A top view of the CPC drawer is shown in Figure 2-9.
Figure 2-9 CPC drawer top view and feature code comparison
The following PU SCM feature codes relate to the resources that are enabled:
•FC 0636: One PU SCM, five DIMMs, two PCIe slots enabled
This configuration supports one PCIe+ I/O drawer or two ICA SR adapters.
•FC 0637: Two PU SCMs, up to 10 DIMMs, four PCIe slots enabled
This configuration supports up to two PCIe+ I/O drawers or four ICA SR adapters or one PCIe+ I/O Drawer and two ICA SR Adapters.
•FC 0638 and FC 0639: Four PU SCMs, up to 20 DIMMs, eight PCIe slots enabled
This configuration supports up to four PCIe+ I/O drawers or eight ICA SR adapters or a combination not to exceed eight PCIe slots in the CPC drawer.
Memory is connected to the SCMs through memory control units (MCUs). Up to four MCUs are available in a drawer (one per PU SCM) to provide the interface to the controller on memory DIMM. A memory control unit drives five DIMM slots.
The CPC drawer logical diagram is shown in Figure 2-10 on page 31.
Figure 2-10 CPC drawer logical diagram
The buses are organized in the following configurations:
•The PCIe I/O buses provide connectivity for PCIe fanouts and can sustain up to 16 GBps data traffic per bus direction.
•The X-bus provides interconnects between SC SCM to PU SCM and PU SCMs to each other, in the same node.
•Processor support interfaces (PSIs) are used to communicate with FSP cards for system control.
•Configurations with four PU SCMs operate as two Logical PU clusters with two PU SCMs per logical cluster.
•Configurations with two PU SCMs or one PU SCM operate in one logical PU cluster.
2.3.1 Oscillator cards
The z14 ZR1 CPC drawer contains the two oscillator cards (OSCs): One primary and one backup. If the primary OSC fails, the secondary detects the failure, takes over transparently, and continues to provide the clock signal to the CPC. The two oscillators have Bayonet Neill-Concelman (BNC) connectors that provide pulse per second signal (PPS) input for synchronization to an external time source with PPS output.
The SEs provide the Simple Network Time Protocol (SNTP) client function. When Server Time Protocol (STP) is used, the time of an STP-only Coordinated Timing Network (CTN) can be synchronized to the time that is provided by a Network Time Protocol (NTP) server. This configuration allows time-of-day (TOD) synchronization in a heterogeneous platform environment and throughout the LPARs running on the CPC.
The accuracy of an STP-only CTN is improved by using an NTP server with the PPS output signal as the External Time Source (ETS). NTP server devices with PPS output are available from several vendors that offer network timing solutions. A cable connection from the PPS port on the OSC to the PPS output of the NTP server is required when z14 ZR1 uses STP and is configured in an STP-only CTN that uses NTP with PPS as the external time source. The z14 ZR1 server cannot participate in a mixed CTN; it can participate in an STP-only CTN only.
STP with PPS timing signal accuracy
STP tracks the highly stable and accurate PPS signal from the NTP server and maintains an accuracy of 10 µs to the ETS, as measured at the PPS input of the z14 ZR1 server.
STP without PPS timing signal accuracy
If STP uses an NTP server without PPS, a time accuracy of 100 ms to the ETS is maintained.
The OSCs cards are plugged into the rear of the CPC drawer in slots LG11 and LG12.
|
Tip: STP is available as FC 1021. It is implemented in the Licensed Internal Code (LIC), and allows multiple servers to maintain time synchronization with each other and synchronization to an ETS. For more information, see the following publications:
•Server Time Protocol Planning Guide, SG24-7280
•Server Time Protocol Implementation Guide, SG24-7281
•Server Time Protocol Recovery Guide, SG24-7380
|
2.3.2 System control
The various system elements are managed through the Flexible Server Processors (FSPs). An FSP is based on the IBM PowerPC® microprocessor technology. Each FSP card has two ports to connect to two internal Ethernet LANs through the internal network switches (SW1 and SW2). The FSPs communicate with the SEs and provide a subsystem interface (SSI) for controlling components.
An overview of the system control design is shown in Figure 2-11.
Figure 2-11 Conceptual overview of system control elements
|
Note: The maximum z14 ZR1 configuration has one CPC drawer and four PCIe+ I/O drawers. The various supported FSP connections are referenced in Figure 2-11.
|
A typical FSP operation is to control a power supply. An SE sends a command to the FSP to start the power supply. The FSP (by using SSI connections) cycles the various components of the power supply, monitors the success of each step and the resulting voltages, and reports this status to the SE.
Most system elements are duplexed (N+1), and each element has at least one FSP. Two internal Ethernet LANs and two SEs, for redundancy, and crossover capability between the LANs, are available so that both SEs can operate on both LANs.
The Hardware Management Consoles (HMCs) and SEs are connected directly to one or two Ethernet Customer LANs. One or more HMCs can be used.
2.3.3 CPC drawer power
The power for the CPC drawer is a new design. It uses the following combinations of Power Supply Units (PSUs), POL5s, VRMs, and Power Control Cards:
•PSUs: Provide AC to 12V DC bulk/standby power and are installed at the rear of the CPC. The quantity that is installed depends on the following configurations:
– Four PSUs for configurations with four PU SCMs
– Two PSUs for configurations with one or two PU SCMs
•POLs: Point of Load N+2 Redundant cards are installed next to the Memory DIMMs.
•VRMs: Voltage Regulator Modules are derivative of z13s design (N+2 redundancy).
•Power Control card: Redundant processor power and control cards connect to the CPC trail board. The control function is powered from 12V standby that is provided by the PSU.
The Power Control card also includes pressure, temperature, and humidity sensors.
The Power Control card also includes pressure, temperature, and humidity sensors.
2.4 Single chip modules
The Single Chip Module (SCM) is a multi-layer metal substrate module that holds one PU chip or an SC chip. Both PU and SC chip size is 696 mm2 (25.3 mm x 27.5 mm). Each CPC drawer has one, two or four PU SCMs (6.1 billion transistors each), and one SC SCM (9.7 billion transistors).
The two types of SCMs (PU and SC) are shown in Figure 2-12.
Figure 2-12 Single Chip Modules (PU SCM and SC SCM)
Both PU and SC SCMs use CMOS 14 nm process, 17 layers of metal, and state-of-the-art Silicon-On-Insulator (SOI) technology.
The SCMs are plugged into a socket that is part of the CPC drawer packaging.
2.4.1 Processor Unit Single Chip Module
|
Note: The terms PU SCM (packaged PU chip) and PU chip are used interchangeably in this section.
|
The z14 ZR1 PU chip shares the design with the z14 M0x PU chip and is an evolution of the z13s design. It includes the following features:
•CMOS 14nm SOI technology
•Pipeline enhancements, dynamic improved simultaneous multithreading (SMT), enhanced single-instruction multiple-data (SIMD), and redesigned, larger on-chip caches
Each PU chip includes up to nine active cores (10 cores by design) that run at 4.5 GHz, which means that the cycle time is 0.222 ns. The PU SCMs come in five versions: 5, 6, 7, 8 or 9 active cores. A schematic representation of the PU chip is shown in Figure 2-13 on page 35.
Figure 2-13 PU SCM floor plan
The PU chip contains the following enhancements:
•Cache Improvements:
– New power efficient logical directory design
– 33% larger L1 I$ (128 KB), private
– 2x larger L2 D$ (4 MB), private
– 2x larger L3 cache with symbol ECC, shared
•New Translation/TLB2 design:
– Four concurrent translations
– Reduced latency
– Lookup that is integrated into L2 access pipe
– 2x Consolidated region and segment table entries (CRSTE) growth
– 1.5x Page table entry (PTE) growth
– New 64 entry 2 GB TLB2
•Pipeline Optimizations:
– Improved instruction delivery
– Faster branch wake-up
– Reduced execution latency
– Improved Operand store compare (OSC) avoidance
– Optimized second-generation SMT2
•Better Branch Prediction:
– 33% Larger BTB1 & BTB2
– New Perceptron Predictor
– New Simple Call Return Stack
2.4.2 Processor Unit (Core)
Each processor unit (see Figure 2-14) or core is a superscalar and out-of-order processor that includes 10 execution units.
Figure 2-14 PU Core layout
Consider the following points:
•Fixed-point unit (FXU): The FXU handles fixed-point arithmetic.
•Load-store unit (LSU): The LSU contains the data cache. It is responsible for handling all types of operand accesses of all lengths, modes, and formats as defined in the z/Architecture.
•The instruction fetch and branch (IFB) (prediction) and instruction cache and merge (ICM) sub units contain the instruction cache, branch prediction logic, instruction fetching controls, and buffers. Its relative size is the result of the elaborate branch prediction.
•Instruction decode unit (IDU): The IDU is fed from the IFU buffers, and is responsible for parsing and decoding of all z/Architecture operation codes.
•Translation unit (XU): The XU has a large translation lookaside buffer (TLB) and the Dynamic Address Translation (DAT) function that handles the dynamic translation of logical to physical addresses.
•Instruction sequence unit (ISU): This unit enables the out-of-order (OoO) pipeline. It tracks register names, OoO instruction dependency, and handling of instruction resource dispatch.
•Recovery unit (RU): The RU keeps a copy of the complete state of the system that includes all registers, collects hardware fault signals, and manages the hardware recovery actions.
•Dedicated Co-Processor (COP): The dedicated coprocessor is responsible for data compression and encryption functions for each core.
•Core pervasive unit (PC) for instrumentation, error collection.
•Vector and Floating point Units (VFU).
•Binary floating-point unit (BFU): The BFU handles all binary and hexadecimal floating-point and fixed-point multiplication operations.
•Decimal floating-point unit (DFU): The DU runs floating-point, decimal fixed-point, and fixed-point division operations.
•Vector execution unit (VXU).
•Level 2 cache (L2) for instructions and data (L2I/L2D).
2.4.3 PU characterization
The PUs are characterized for client use. The characterized PUs can be used in general to run supported operating systems, such as z/OS, z/VM, and Linux on Z. They also can run specific workloads, such as Java, XML services, IPSec, and some Db2 workloads, or functions, such as the Coupling Facility Control Code (CFCC). For more information about PU characterization, see 3.5, “Processor unit functions” on page 83.
The maximum number of characterizable PUs depends on the ZR1 CPC drawer feature code. Some PUs are characterized for system use; some are characterized for client workload use.
By default, one spare PU is available to assume the function of a failed PU. The maximum number of PUs that can be characterized for client use are listed in Table 2-1.
Table 2-1 Number of PUs per z14 model
|
Feature
|
CPs
|
IFLs
|
zIIPs
|
ICFs
|
IFPs
|
Standard SAPs
|
Add’l SAPs
|
Spares
|
|
|
Max30
|
0 - 6
|
0 - 30
|
0 - 12
|
0 - 30
|
1
|
2
|
0 - 2
|
1
|
|
|
Max24
|
0 - 6
|
0 - 24
|
0 - 12
|
0 - 24
|
1
|
2
|
0 - 2
|
1
|
|
|
Max12
|
0 - 6
|
0 - 12
|
0 - 8
|
0 - 12
|
1
|
2
|
0 - 2
|
1
|
|
|
Max4
|
0 - 4
|
0 - 4
|
0 - 2
|
0 - 4
|
1
|
2
|
0 - 2
|
1
|
|
2.4.4 System Controller SCM (chip)
The System Controller (SC) SCM uses the CMOS 14nm SOI chip technology, with 17 layers of metal. It measures 25.3 x 27.5 mm, and has 9.7 billion transistors. One SC SCM is available per system for all CPC drawer features.
A schematic representation of the SC chip is shown in Figure 2-15 on page 38. Consider the following points:
•X-Bus6 (PU-PU and PU-SC): Significant changes allow SC to fit more X-Bus connections
•672 MB shared eDRAM L4 Cache is available
•L4 Directory is built with eDRAM
•New L4 Cache Management:
– L3 to L4 cache capacity ration was increased with the z14 ZR1 processor design
– New on-drawer Cluster-to-Cluster (topology change) management
Figure 2-15 SC chip layout
2.5 Memory
The maximum physical memory size is feature-dependent. The orderable memory sizes for z14 ZR1 are listed in Table 2-2.
Table 2-2 z14 ZR1 Memory sizes
|
Feature
|
Number of PU SCMs
|
Standard memory (GB)1
|
|
Max4
|
1
|
64-1984
|
|
Max12
|
2
|
64-4032
|
|
Max24
|
4
|
64-8128
|
|
Max30
|
4
|
64-8128
|
1 Actual amount of customer usable memory.
The minimum physical installed memory is 160 GB. The minimum initial amount of memory that can be ordered for z14 ZR1 is 64 GB. The maximum customer memory size depends on the number of PU SCMs installed (CPC drawer feature) and is based on the physical installed memory minus the RAIM (20% of physical memory) and minus the hardware system area (HSA) memory, which has a fixed amount of 64 GB.
The memory ordering granularity (installed customer memory) is listed in Table 2-3.
Table 2-3 Memory ordering granularity
|
Memory increment (GB)
|
Offered memory sizes (GB)
|
|
8
|
64, 72, 80, 88, 96
|
|
32
|
128, 160, 192, 256, 288, 320, 352, 384
|
|
64
|
448, 512, 576
|
|
128
|
704, 832, 960
|
|
256
|
1216, 1472, 1728, 1984, 2240 … 4032
|
|
512
|
4544, 5056 … 8128
|
2.5.1 Memory subsystem topology
The z14 ZR1 memory subsystem uses high-speed, differential-ended communications memory channels. An overview of the CPC drawer memory topology of a z14 ZR1 server is shown in Figure 2-16.
Figure 2-16 CPC drawer memory topology
The CPC drawer includes 5, 10, 15, or 20 DIMMs (up to four populated memory banks). DIMMs are connected to the memory control unit (MCU) on each PU SCM. Each PU SCM has one MCU, which uses five channels: one DIMM per channel, which implements a RAIM protection scheme (4 +1 design). Each CPC drawer can have one, two, or four populated memory banks.
DIMMs are used in 32, 64, 128, 256, and 512 GB sizes with five DIMMs of the same size included in a memory feature (160, 320, 640, 1280, and 2560 GB RAIM array size).
2.5.2 Redundant array of independent memory
The z14 ZR1 server uses the RAIM protection scheme against hardware memory failures. The RAIM design detects and recovers from failures of dynamic random access memory (DRAM), sockets, memory channels, or DIMMs.
The RAIM design requires the addition of one memory channel that is dedicated for reliability, availability, and serviceability (RAS), as shown in Figure 2-17.
Figure 2-17 RAIM configuration (four PU SCMs)
The fifth channel in each MCU enables memory to be implemented as a Redundant Array of Independent Memory (RAIM). RAIM features significant error detection and correction capabilities: bit, lane, DRAM, DIMM, socket, and complete memory channel failures can be detected and corrected, including many types of multiple failures.
The RAIM design provides the following layers of memory recovery:
•ECC with 90B/64B Reed Solomon code.
•DRAM failure, with marking technology in which two DRAMs can be marked and no half sparing is needed. A call for replacement occurs on the third DRAM failure.
•Lane failure with CRC retry, data-lane sparing, and clock-RAIM with lane sparing.
•DIMM failure with CRC retry, data-lane sparing, and clock-RAIM with lane sparing.
•DIMM controller ASIC failure.
•Channel failure.
2.5.3 Memory configurations
Consider the following general plugging rules for memory:
•5, 10, 15, or 20 DIMMs are plugged, depending on the configuration (minimum 5 DIMMs: one bank must be installed)
•No mixing of DRAM technology within a Memory Controller (that is, all DDR4)
•Each five slot DIMM bank must have the same DIMM size
•A mix of DIMM sizes can be used between different DIMM banks
•Another 64 GB of memory is added to the total amount of memory that is specified by the customer. The extra 64 GB of memory is reserved for use by HSA.
•The CPC drawer can have available unused memory, which can be ordered as a memory upgrade and enabled by LIC without DIMM changes.
•DIMM changes are disruptive (require machine power off) on z14 ZR1.
Memory location plugging by CPC drawer feature
Memory plugging rules for CPC drawer Max4 feature are listed in Table 2-4.
Table 2-4 Max4 FC 0636 physical memory configurations
|
|
Physical
|
PU SCM2
MD01-05
|
PU SCM3
MD06-10
|
PU SCM4
MD11-15
|
PU SCM5
MD16-20
|
GB
total
|
Customer capacity
|
Hardware
increment
|
|
1
|
160
|
32
|
N/A
|
N/A
|
N/A
|
128
|
64
|
N/A
|
|
2
|
320
|
64
|
N/A
|
N/A
|
N/A
|
256
|
192
|
128
|
|
3
|
640
|
128
|
N/A
|
N/A
|
N/A
|
512
|
448
|
256
|
|
4
|
1280
|
256
|
N/A
|
N/A
|
N/A
|
1024
|
960
|
512
|
|
5
|
2560
|
512
|
N/A
|
N/A
|
N/A
|
2048
|
1984
|
1024
|
Memory plugging rules for CPC drawer Max12 feature are listed in Table 2-5.
Table 2-5 Max12 FC 0637 physical memory configurations
|
|
Physical
|
PU SCM2
MD01-05
|
PU SCM3
MD06-10
|
PU SCM4
MD11-15
|
PU SCM5
MD16-20
|
GB
total
|
Customer capacity
|
Hardware
increment
|
|
1
|
320
|
32
|
32
|
N/A
|
N/A
|
256
|
192
|
N/A
|
|
2
|
640
|
64
|
64
|
N/A
|
N/A
|
512
|
448
|
256
|
|
3
|
1280
|
128
|
128
|
N/A
|
N/A
|
1024
|
960
|
512
|
|
4
|
2560
|
256
|
256
|
N/A
|
N/A
|
2048
|
1984
|
1024
|
|
5
|
3840
|
256
|
512
|
N/A
|
N/A
|
3072
|
3008
|
1024
|
|
6
|
5120
|
512
|
512
|
N/A
|
N/A
|
4096
|
4032
|
1024
|
Memory plugging rules for CPC drawer Max24 and Max30 features are listed in Table 2-6.
Table 2-6 Max24 FC 0638 and Max30 FC 0639 physical memory configurations
|
|
Physical
|
PU SCM2
MD01-05
|
PU SCM3
MD06-10
|
PU SCM4
MD11-15
|
PU SCM5
MD16-20
|
GB
total
|
Customer capacity
|
Hardware
increment
|
|
1
|
320
|
32
|
32
|
N/A
|
N/A
|
256
|
192
|
N/A
|
|
2
|
480
|
32
|
32
|
32
|
N/A
|
384
|
320
|
128
|
|
3
|
640
|
32
|
32
|
32
|
32
|
512
|
448
|
128
|
|
4
|
960
|
64
|
64
|
32
|
32
|
768
|
704
|
256
|
|
5
|
1280
|
64
|
64
|
64
|
64
|
1024
|
960
|
256
|
|
6
|
1600
|
128
|
128
|
32
|
32
|
1280
|
1088
|
256
|
|
7
|
2560
|
128
|
128
|
128
|
128
|
2048
|
1984
|
768
|
|
8
|
3840
|
128
|
128
|
256
|
256
|
3072
|
3008
|
1024
|
|
9
|
5120
|
256
|
256
|
256
|
256
|
4096
|
4032
|
1024
|
|
10
|
6400
|
256
|
256
|
256
|
512
|
5120
|
5056
|
1024
|
|
11
|
7680
|
256
|
256
|
512
|
512
|
6144
|
6080
|
1024
|
|
12
|
8960
|
256
|
256
|
512
|
512
|
7168
|
7104
|
1024
|
|
13
|
10240
|
512
|
512
|
512
|
512
|
8192
|
8128
|
1024
|
The View Hardware Configuration task on the Support Element lists all the hardware components in the system. It can be used to view the memory DIMM capacity that is installed by location in the system. An example of a system with 256 GB DIMMs installed in all 20 slots in the CPC Drawer is shown in Figure 2-18.
Figure 2-18 View Hardware Configuration task on the Support Element
The CPC drawer and DIMM locations for a z14 ZR1 are shown in Figure 2-19.
Figure 2-19 CPC drawer and DIMM locations
The physical memory DIMM plugging configurations by feature code from manufacturing when the system is ordered are listed in Table 2-7. The drawer columns for the specific model contain the memory configuration number for the specific drawer. Use available unused memory that can be enabled by LIC, when required.
If more storage is ordered by using other feature codes, such as Virtual Flash Memory or Preplanned memory, the extra storage is installed and plugged as necessary.
Table 2-7 Memory features
|
FC
|
Incr.
|
Mem.
Incr.
GB
|
Max4
|
Max12
|
Max24 / Max30
|
|||||||
|
SCM2
Qty 5
|
Dial
Max
|
SCM2
Qty 5
|
SCM2
Qty 5
|
Dial
Max
|
SCM2
Qty 5
|
SCM2
Qty 5
|
SCM2
Qty 5
|
SCM2
Qty 5
|
Dial
Max
|
|||
|
3539
|
8
|
64
|
32GB
|
64
|
32GB
|
32GB
|
192
|
32GB
|
32GB
|
X
|
X
|
192
|
|
3540
|
|
72
|
64GB
|
192
|
32GB
|
32GB
|
|
32GB
|
32GB
|
X
|
X
|
|
|
3541
|
|
80
|
64GB
|
|
32GB
|
32GB
|
|
32GB
|
32GB
|
X
|
X
|
|
|
3542
|
|
88
|
64GB
|
|
32GB
|
32GB
|
|
32GB
|
32GB
|
X
|
X
|
|
|
3543
|
|
96
|
64GB
|
|
32GB
|
32GB
|
|
32GB
|
32GB
|
X
|
X
|
|
|
3544
|
32
|
128
|
64GB
|
|
32GB
|
32GB
|
|
32GB
|
32GB
|
X
|
X
|
|
|
3545
|
|
160
|
64GB
|
|
32GB
|
32GB
|
|
32GB
|
32GB
|
X
|
X
|
|
|
3546
|
|
192
|
64GB
|
|
32GB
|
32GB
|
|
32GB
|
32GB
|
X
|
X
|
|
|
3547
|
|
224
|
128GB
|
448
|
64GB
|
64GB
|
448
|
32GB
|
32GB
|
32GB
|
X
|
320
|
|
3548
|
|
256
|
128GB
|
|
64GB
|
64GB
|
|
32GB
|
32GB
|
32GB
|
X
|
|
|
3549
|
|
288
|
128GB
|
|
64GB
|
64GB
|
|
32GB
|
32GB
|
32GB
|
X
|
|
|
3550
|
|
320
|
128GB
|
|
64GB
|
64GB
|
|
32GB
|
32GB
|
32GB
|
X
|
|
|
3551
|
|
352
|
128GB
|
|
64GB
|
64GB
|
|
32GB
|
32GB
|
32GB
|
32GB
|
448
|
|
3552
|
|
384
|
128GB
|
|
64GB
|
64GB
|
|
32GB
|
32GB
|
32GB
|
32GB
|
|
|
3553
|
64
|
448
|
128GB
|
|
64GB
|
64GB
|
|
32GB
|
32GB
|
32GB
|
32GB
|
|
|
3554
|
|
512
|
256GB
|
960
|
128GB
|
128GB
|
960
|
64GB
|
64GB
|
32GB
|
32GB
|
704
|
|
3555
|
|
576
|
256GB
|
|
128GB
|
128GB
|
|
64GB
|
64GB
|
32GB
|
32GB
|
|
|
3556
|
128
|
704
|
256GB
|
|
128GB
|
128GB
|
|
64GB
|
64GB
|
64GB
|
64GB
|
|
|
3557
|
|
832
|
256GB
|
|
128GB
|
128GB
|
|
64GB
|
64GB
|
64GB
|
64GB
|
960
|
|
3558
|
|
960
|
256GB
|
|
128GB
|
128GB
|
|
64GB
|
64GB
|
64GB
|
64GB
|
|
|
3559
|
256
|
1216
|
512GB
|
1984
|
256GB
|
256GB
|
1984
|
128GB
|
128GB
|
128GB
|
128GB
|
1984
|
|
3560
|
|
1472
|
512GB
|
|
256GB
|
256GB
|
|
128GB
|
128GB
|
128GB
|
128GB
|
|
|
3561
|
|
1728
|
512GB
|
|
256GB
|
256GB
|
|
128GB
|
128GB
|
128GB
|
128GB
|
|
|
3562
|
|
1984
|
512GB
|
|
256GB
|
256GB
|
|
128GB
|
128GB
|
128GB
|
128GB
|
|
|
3563
|
|
2240
|
|
|
256GB
|
256GB
|
3008
|
128GB
|
128GB
|
256GB
|
256GB
|
3008
|
|
3564
|
|
2496
|
|
|
256GB
|
256GB
|
|
128GB
|
128GB
|
256GB
|
256GB
|
|
|
3565
|
|
2752
|
|
|
256GB
|
256GB
|
|
128GB
|
128GB
|
256GB
|
256GB
|
|
|
3566
|
|
3008
|
|
|
256GB
|
256GB
|
|
128GB
|
128GB
|
256GB
|
256GB
|
|
|
3567
|
|
3264
|
|
|
512GB
|
512GB
|
4032
|
256GB
|
256GB
|
256GB
|
256GB
|
4032
|
|
3568
|
|
3520
|
|
|
512GB
|
512GB
|
|
256GB
|
256GB
|
256GB
|
256GB
|
|
|
3569
|
|
3776
|
|
|
512GB
|
512GB
|
|
256GB
|
256GB
|
256GB
|
256GB
|
|
|
3570
|
|
4032
|
|
|
512GB
|
512GB
|
|
256GB
|
256GB
|
256GB
|
256GB
|
|
|
3571
|
512
|
4544
|
|
|
|
|
|
256GB
|
256GB
|
256GB
|
512GB
|
5056
|
|
3572
|
|
5056
|
|
|
|
|
|
256GB
|
256GB
|
256GB
|
512GB
|
|
|
3573
|
|
5568
|
|
|
|
|
|
256GB
|
256GB
|
512GB
|
512GB
|
6080
|
|
3574
|
|
6080
|
|
|
|
|
|
256GB
|
256GB
|
512GB
|
512GB
|
|
|
3575
|
|
6592
|
|
|
|
|
|
256GB
|
512GB
|
512GB
|
512GB
|
7104
|
|
3576
|
|
7104
|
|
|
|
|
|
256GB
|
512GB
|
512GB
|
512GB
|
|
|
3577
|
|
7616
|
|
|
|
|
|
512GB
|
512GB
|
512GB
|
512GB
|
8128
|
|
3578
|
|
8128
|
|
|
|
|
|
512GB
|
512GB
|
512GB
|
512GB
|
|
2.5.4 Memory upgrades
Memory upgrades can be ordered and enabled by LICCC, by upgrading (replacing with higher capacity) the DIMM cards, or by adding DIMM cards.
If all or part of the added memory is enabled for use, it might become available to an active LPAR if the partition includes defined reserved storage. (For more information, see 3.7.3, “Reserved storage” on page 105.) Alternatively, the added memory can be used by an already-defined LPAR that is activated after the memory addition.
|
Note: Memory downgrades by way of LICCC are always disruptive.
|
2.5.5 Virtual Flash Memory
IBM Virtual Flash Memory (VFM) FC 0614 replaces the Flash Express features (0402 and 0403) that were available on the IBM z13s. It offers up to 2.0 TB of virtual flash memory in up to four 512 MB increments for improved application availability and to handle paging workload spikes.
No application changes are required to change from IBM Flash Express to VFM. Consider the following points:
•Dialed memory + zVFM = total hardware plugged
•Dialed memory + Plan Ahead memory + VFM = total hardware plugged
VFM is offered as a dialed 512GB memory increment size. Feature code 0614 represents one 512GB zVFM increment, as shown in the following example:
FC 0614 - Min=0, Max=4
VFM is designed to help improve availability and handling of paging workload spikes when z/OS V2.1, V2.2, or V2.3, or on z/OS V1.137 is run. With this support, z/OS is designed to help improve system availability and responsiveness by using VFM across transitional workload events, such as market openings and diagnostic data collection. z/OS is also designed to help improve processor performance by supporting middleware use of pageable large (1 MB) pages.
VFM can also be used in coupling facility images to provide extended capacity and availability for workloads that are use IBM WebSphere MQ Shared Queues structures. The use of VFM can help availability by reducing latency from paging delays that can occur at the start of the workday or during other transitional periods. It is also designed to help eliminate delays that can occur when collecting diagnostic data during failures.
VFM can help organizations meet their most demanding service level agreements and compete more effectively. VFM is easy to configure, and provides rapid time to value.
2.5.6 Preplanned memory
Preplanned memory helps plan for nondisruptive memory upgrades. The required hardware is pre-plugged based on a target capacity that is specified by the customer. This pre-plugged hardware is enabled by way of an LICCC order that is placed by the customer when they determine that more memory capacity is needed. The pre-plugged memory can be made available through a LICCC update.
For more information about ordering this LICCC, see the following resources:
•The IBM Resource Link website (login required)
•Your IBM representative
The installation and activation of any pre-planned memory requires the purchase of the required feature codes (FCs), as listed in Table 2-8.
Table 2-8 Feature codes for plan-ahead memory
|
Memory
|
ZR1 feature code
|
|
Pre-planned memory
Charged when physical memory is installed.
Used for tracking the quantity of physical increments of plan-ahead memory capacity.
|
•FC 1993 - 8 GB
•FC 1996 - 16 GB
|
|
Virtual Flash Memory (VFM) Pre-planned Memory
Charged when physical memory is installed.
Used for tracking the quantity of physical increments of plan-ahead VFM memory capacity.
|
FC 1999 - 64 GB
|
|
Pre-planned memory activation
Charged when plan-ahead memory is enabled.
Used for tracking the quantity of increments of plan-ahead memory that are being activated.
|
•FC 1739 (8 GB memory capacity Increments <128GB)
•FC 1740 (8 GB memory capacity Increments >=128GB)
•FC 1741(16 GB memory capacity Increments >1=28GB)
•FC 1742 (32GB memory capacity increments >=128GB
|
The payment for plan-ahead memory is a two-phase process. One charge occurs when the plan-ahead memory is ordered. Another charge occurs when the prepaid memory is activated for use. For more information about the exact terms and conditions, contact your IBM representative.
Pre-planned memory is installed by ordering FC 1993 (8 GB) or FC 1996 (16 GB). The ordered amount of plan-ahead memory is charged at a reduced price compared to the normal price for memory. One FC 1993 is needed for each 8 GB of usable memory (10 GB RAIM), or one FC 1996 is needed for each 16 GB of usable memory (20GB RAIM).
|
Reminder: Maximum amount of preplanned memory is limited to 2TB
|
2.6 Reliability, availability, and serviceability
IBM Z servers continue to deliver enterprise class RAS with IBM z14 ZR1 servers. The main philosophy behind RAS is about preventing or tolerating (masking) outages. It is also about providing the necessary instrumentation (in hardware, LIC and microcode, and software) to capture or collect the relevant failure information to help identify an issue without requiring a reproduction of the event. These outages can be planned or unplanned. Planned and unplanned outages can include the following situations (examples are not related to the RAS features of IBM Z servers):
•A planned outage because of added processor capacity
•A planned outage because of added I/O cards
•An unplanned outage because of a power supply failure
•An unplanned outage because of a memory failure
The IBM Z hardware has decades of intense engineering behind it, which results in a robust and reliable platform. The hardware has many RAS features that are built into it. For more information, see Chapter 9, “Reliability, availability, and serviceability” on page 327.
2.6.1 RAS in the CPC memory subsystem
Patented error correction technology in the memory subsystem continues to provide the most robust error correction from IBM to date. Two full DRAM failures per rank can be spared and a third full DRAM failure can be corrected.
DIMM level failures, including components, such as the memory controller application-specific integrated circuit (ASIC), power regulators, clocks, and system board, can be corrected. Memory channel failures, such as signal lines, control lines, and drivers and receivers on the MCM, can be corrected.
Upstream and downstream data signals can be spared by using two spare wires on the upstream and downstream paths. One of these signals can be used to spare a clock signal line (one upstream and one downstream). The following improvements were also added in the z14 ZR1 server:
•No cascading of memory DIMMs
•Independent channel recovery
•Double tabs for clock lanes
•Separate replay buffer per channel
•Hardware driven lane soft error rate (SER) and sparing.
2.6.2 General z14 ZR1 RAS features
The z14 ZR1 server includes the following RAS features:
•The Power/Thermal Subsystem is new for z14 ZR1. It uses switchable, intelligent Power Distribution Units (PDUs) instead of Bulk Power Assemblies in past generations. The z14 ZR1 server provides a true N+1 cooling function with fans.
•Redundant (N+1), number of PDUs is configuration-dependent (2 or 4).
Input power Single Phase, not cross-coupled. The loss of a single phase puts power subsystem into N-mode.
•CPC drawer is packaged to fit in the 19-inch rack. CPC drawer power and cooling includes:
– PSUs: AC to 12V bulk/standby (N+1 redundant). PSU Fans are not separate FRUs and are available in quantities of 2 or 4.
– POLs: N+2 Phase and Master Redundant and are available in quantities of 3 - 6.
– VRM sticks8: Derivative of z13s design (N+2 Phase and Master redundancy) and are available in quantities of 6.
– Power Control Card: New power control card to control CPC fans (N+1 redundant) and are available in quantities of 2.
– SCMs are all air-cooled and used new heat sinks for PU SCMs on z14 ZR1. The SC heat sink on the SC SCM is the same from z14 M0x.
– Fans: Drawer has five fans and are N+1 redundant.
– FSPs: Redundant (N+1).
•PCIe+ I/O Drawer Power/Thermal is all new for z14 ZR1:
– Two Power Supply assemblies: Power supply with dedicated on board fan, combined with I/O Power Control (Power Supply and I/O Power Control are separate FRUs) in N+1 configuration
– Fans: Drawer has six fans, N+1 redundant
– FSPs: Redundant (N+1)
The internal intelligent Power Distribution Unit (iPDU) provide the following capabilities:
•Switchable PDUs provide outlet control by way of Ethernet:
– Provide a System Reset capability
– Power cycle an SE if a hang occurs
– Verify a power cable at installation
•System Reset Function:
– No EPO switch is on the z14 ZR1. This function provides a means to put a server into a known state similar to past total power reset.
– This function does not provide the option to power down and keep the power down to the system. The power must be unplugged or the customer-supplied power is turned off at the panel.
•Other characteristics:
– iPDU Firmware can be concurrently updated
– Concurrently repairable
– Power redundancy check
•Cable verification test by way of IPDU:
– By power cycling individual iPDU outlets, the system can verify proper cable connectivity
– Power cable test runs during system Power On
– Runs at system Installation and at every system Power On until the test passes and erases the cable test file
The power service and control network (PSCN) is used to control and monitor the elements in the system and include the following components:
•Ethernet Top of Rack (TOR) switches provide the internal PSCN connectivity:
– Switches are redundant (N+1)
– Concurrently maintainable
– Each switch has on integrated power supply
– FSPs are cross wired to the Ethernet switches
•Redundant SEs
Each SE has two power supplies (N+1) and input power is cross-coupled from the PDUs.
IBM z14 ZR1 servers continue to deliver robust server designs through new technologies, hardening both new and classic redundancy.
For more information, see Chapter 9, “Reliability, availability, and serviceability” on page 327.
2.7 Connectivity
Connections to PCIe+ I/O drawers and Integrated Coupling Adapters (ICAs) are driven from the CPC drawer fanout cards. These fanouts are on the front of the CPC drawer.
The location of the fanouts for the CPC drawer is shown in Figure 2-20.
Figure 2-20 Location of the PCIe, FSP and OSC adapters
The number of available fanouts depends on the CPC drawer feature:
•Eight PCIe fanout slots are available for systems with Max24/Max30 CPC drawer. PCIe fanout locations are LG01 - LG04 and LG07 - LG10.
•Two PCIe fanouts are available for systems with Max4 CPC drawer. PCIe fanout locations are LG09 and LG10.
•Four PCIe fanouts are available for systems with Max12 CPC drawer. PCIe fanout locations are LG07- LG10.
The CPC drawer has two FSPs for system control. The location codes for the FSPs are LG05 and LG06.
A fanout can be repaired concurrently with the use of redundant I/O interconnect. For more information, see 2.7.1, “Redundant I/O interconnect” on page 49.
The following types of fanouts are available:
•PCIe Generation3 fanout card: This copper fanout provides connectivity to the PCIe switch cards in the PCIe+ I/O drawer.
•Integrated Coupling Adapter (ICA SR): This adapter provides coupling connectivity between z14 ZR1 and z14 M0x / z13 / z13s servers.
When you are configuring for availability, balance coupling links across adapters, and I/O features across PCIe+ I/O drawers. In a system that is configured for maximum availability, alternative paths maintain access to critical I/O devices, such as disks and networks. The CHPID Mapping Tool can be used to assist with configuring a system for high availability.
2.7.1 Redundant I/O interconnect
Redundancy is provided for PCIe I/O interconnects.
The PCIe+ I/O drawer supports up to 16 PCIe cards, which are organized in two hardware domains per drawer, as shown in Figure 2-21 on page 50.
Figure 2-21 Infrastructure for PCIe+ I/O drawer (system with two PCIe+ I/O drawers)
Each domain is driven through a PCIe Gen3 switch. The two PCIe switch cards (LG06 and LG16) provide a backup path (Redundant I/O Interconnect - RII) for each other through the passive connection in the PCIe+ I/O drawer backplane. During a PCIe fanout or cable failure, all 16 PCIe cards in the two domains can be driven through a single PCIe switch card (see Figure 2-22).
Figure 2-22 Redundant I/O Interconnect
To support Redundant I/O Interconnect (RII) between domain pair 0 and 1, the two interconnects to each pair must be driven from two different PCIe fanouts. Normally, each PCIe interconnect in a pair supports the eight features in its domain. In backup operation mode, one PCIe interconnect supports all 16 features in the domain pair.
|
Note: The PCIe Gen3 Interconnect (switch) adapter must be installed in the PCIe+ I/O drawer to maintain the interconnect across I/O domains. If the adapter is removed (for a service operation), the I/O cards in that domain (up to eight) become unavailable.
|
2.7.2 CPC drawer upgrades
All fanouts that are used for I/O and coupling links are rebalanced if the upgrade does not involve adding components inside the CPC drawer. If MES involves extra components, such as PU SCMs and Memory DIMMs, the change is disruptive.
When a z14 ZR1 is ordered, the PUs are characterized according to their intended usage. The PUs can be ordered as any of the following items:
CP The processor is purchased and activated. PU supports the z/OS, z/VSE, z/VM, z/TPF, and Linux on Z9 operating systems. It can also run Coupling Facility Control Code.
Capacity marked CP A processor that is purchased for future use as a CP is marked as available capacity. It is offline and not available for use until an upgrade for the PU is installed. It does not affect software licenses or maintenance charges.
IFL The Integrated Facility for Linux (IFL) is a processor that is purchased and activated for use by z/VM for Linux guests and Linux on Z9 operating systems.
Unassigned IFL A processor that is purchased for future use as an IFL. It is offline and cannot be used until an upgrade for the IFL is installed. It does not affect software licenses or maintenance charges.
ICF An internal coupling facility (ICF) processor that is purchased and activated for use by the Coupling Facility Control Code.
zIIP An “Off Load Processor” for workload that is restricted to Db2 type applications.
Additional SAP An optional processor that is purchased and activated for use as an SAP (System Assist Processor).
A minimum of one PU that is characterized as a CP, IFL, or ICF is required per system. The maximum number of characterizable PUs is 30 (any combination of 6 CPs, 30 IFLs, 30 ICF, two more SAP10s, and 12 ZIIPs). The maximum number of zIIPs is up to twice the number of PU that are characterized as CP.
|
Remainder: Not all PUs on a model must be characterized.
|
The following items are present in the z14 ZR1 server, but they are not part of the PUs that clients purchase and require no characterization:
•Standard SAP to be used by the channel subsystem. The number of standard SAPs is always two.
•One IFP (Integrated Firmware Processor), which is used in the support of “native” PCIe features.
•One spare PU, which can transparently assume any characterization if another PU a permanent fails.
The various feature code driven CPC drawer sizes are listed in Table 2-1 on page 37.
A capacity marker identifies the number of CPs that were purchased. This number of purchased CPs is higher than or equal to the number of CPs that is actively used. The capacity marker marks the availability of purchased but unused capacity that is intended to be used as CPs in the future. They often have this status for software-charging reasons. Unused CPs are not a factor when establishing the millions of service units (MSU) value that is used for charging monthly license charge (MLC) software, or when charged on a per-processor basis.
2.7.3 System upgrades
Concurrent upgrades of CPs, IFLs, ICFs, zIIPs, or SAPs are available for the z14 ZR1 server. However, concurrent PU upgrades require that more PUs are installed, but not activated.
The spare PU is used to replace defective PUs. In the rare event of a PU failure, a spare PU is activated concurrently and transparently and is assigned the characteristics of the failing PU.
If an upgrade request cannot be accomplished within the configuration, a hardware upgrade is required. The upgrade that requires the addition of one or more PU SCMs and DIMMs per to accommodate the wanted capacity.
All upgrades for z14 ZR1 that involve adding PU SCMs or memory are disruptive.
The upgrade paths for the z14 are shown in Figure 2-23.
Figure 2-23 z14 ZR1 system upgrade paths
Consider the following points regarding upgrades:
•Upgrade from a z14 LR1 to a z14 ZR1 is supported
•Upgrade from z14 ZR1 to z14 LR1 is not supported
•Upgrade from z14 ZR1 to z14 M0x is not supported
•Upgrade from z14 LR1 to z14 LM1 is not supported
You can upgrade a z13s (2965) server and preserve the CPC serial number (S/N). The I/O cards can also be carried forward (with certain restrictions) to the z14 ZR1 server. For frame roll MES from z13s to z14 ZR1, new frames are shipped. New PCIe+ I/O drawers are supplied with the MES for z13s to replace the PCIe I/O drawers.
|
Important: Upgrades from IBM z13s are always disruptive.
|
2.7.4 Concurrent PU conversions
Assigned CPs, assigned IFLs, and unassigned IFLs, ICFs, zIIPs, and SAPs can be converted to other assigned or unassigned PU feature codes.
Most conversions are nondisruptive. In exceptional cases, the conversion might be disruptive; for example, when a model ZR1 with 6 CPs is converted to an all IFL system. In addition, an LPAR might be disrupted when PUs must be freed before they can be converted. Conversion information is listed in Table 2-9.
Table 2-9 Concurrent PU conversions
|
To
From
|
CP
|
IFL
|
Unassigned IFL
|
ICF
|
zIIP
|
Additional SAP
|
|
CP
|
-
|
Yes
|
Yes
|
Yes
|
Yes
|
Yes
|
|
IFL
|
Yes
|
-
|
Yes
|
Yes
|
Yes
|
Yes
|
|
Unassigned IFL
|
Yes
|
Yes
|
-
|
Yes
|
Yes
|
Yes
|
|
ICF
|
Yes
|
Yes
|
Yes
|
-
|
Yes
|
Yes
|
|
zIIP
|
Yes
|
Yes
|
Yes
|
Yes
|
-
|
Yes
|
|
Additional SAP
|
Yes
|
Yes
|
Yes
|
Yes
|
Yes
|
-
|
2.7.5 Model capacity identifier
To recognize how many PUs are characterized as CPs, the Store System Information (STSI) instruction returns a Model Capacity Identifier (MCI). The MCI determines the number and speed of characterized CPs. Characterization of a PU as an IFL, ICF, or zIIP is not reflected in the output of the STSI instruction because characterization has no effect on software charging. For more information about STSI output, see “Processor identification” on page 321.
|
Capacity identifiers: Within a z14 ZR1 server, all CPs feature the same capacity identifier. Specialty engines (IFLs, zIIPs, and ICFs) operate at full speed.
|
Model capacity identifiers
All model capacity identifiers feature a related MSU value that is used to determine the software license charge for MLC software, as listed in Table 2-10 on page 54.
Table 2-10 Model capacity identifier and MSU values
|
Model
cap ID
|
MSU
|
Model
cap ID
|
MSU
|
Model
cap ID
|
MSU
|
Model
cap ID
|
MSU
|
Model
cap ID
|
MSU
|
Model
cap ID
|
MSU
|
|
A01
|
11
|
B01
|
12
|
C01
|
14
|
D01
|
16
|
E01
|
19
|
F01
|
21
|
|
A02
|
21
|
B02
|
23
|
C02
|
26
|
D02
|
30
|
E02
|
35
|
F02
|
40
|
|
A03
|
30
|
B03
|
33
|
C03
|
38
|
D03
|
44
|
E03
|
51
|
F03
|
59
|
|
A04
|
39
|
B04
|
44
|
C04
|
49
|
D04
|
58
|
E04
|
67
|
F04
|
77
|
|
A05
|
48
|
B05
|
53
|
C05
|
60
|
D05
|
71
|
E05
|
82
|
F05
|
94
|
|
A06
|
56
|
B06
|
63
|
C06
|
70
|
D06
|
83
|
E06
|
96
|
F06
|
110
|
|
G01
|
24
|
H01
|
27
|
I01
|
30
|
J01
|
34
|
K01
|
38
|
L01
|
42
|
|
G02
|
45
|
H02
|
51
|
I02
|
57
|
J02
|
64
|
K02
|
71
|
L02
|
80
|
|
G03
|
64
|
H03
|
74
|
I03
|
83
|
J03
|
93
|
K03
|
104
|
L03
|
116
|
|
G04
|
86
|
H04
|
96
|
I04
|
108
|
J04
|
121
|
K04
|
135
|
L04
|
150
|
|
G05
|
105
|
H05
|
118
|
I05
|
132
|
J05
|
147
|
K05
|
164
|
L05
|
184
|
|
G06
|
123
|
H06
|
138
|
I06
|
154
|
J06
|
173
|
K06
|
193
|
L06
|
216
|
|
M01
|
48
|
N01
|
53
|
O01
|
60
|
P01
|
67
|
Q01
|
75
|
R01
|
83
|
|
M02
|
90
|
N02
|
100
|
O02
|
112
|
P02
|
125
|
Q02
|
140
|
R02
|
156
|
|
M03
|
130
|
N03
|
145
|
O03
|
162
|
P03
|
181
|
Q03
|
203
|
R03
|
225
|
|
M04
|
168
|
N04
|
189
|
O04
|
211
|
P04
|
235
|
Q04
|
263
|
R04
|
294
|
|
M05
|
206
|
N05
|
231
|
O05
|
258
|
P05
|
288
|
Q05
|
322
|
R05
|
360
|
|
M06
|
241
|
N06
|
271
|
O06
|
303
|
P06
|
339
|
Q06
|
379
|
R06
|
423
|
|
S01
|
93
|
T01
|
104
|
U01
|
117
|
V01
|
130
|
W01
|
145
|
X01
|
162
|
|
S02
|
175
|
T02
|
195
|
U02
|
218
|
V02
|
243
|
W02
|
272
|
X02
|
304
|
|
S03
|
253
|
T03
|
282
|
U03
|
316
|
V03
|
353
|
W03
|
395
|
X03
|
442
|
|
S04
|
328
|
T04
|
367
|
U04
|
411
|
V04
|
459
|
W04
|
514
|
X04
|
575
|
|
S05
|
402
|
T05
|
450
|
U05
|
504
|
V05
|
563
|
W05
|
629
|
X05
|
705
|
|
S06
|
473
|
T06
|
529
|
U06
|
592
|
V06
|
663
|
W06
|
741
|
X06
|
828
|
|
Y01
|
178
|
Z01
|
195
|
|
|||||||
|
Y02
|
333
|
Z02
|
365
|
||||||||
|
Y03
|
484
|
Z03
|
531
|
||||||||
|
Y04
|
631
|
Z04
|
693
|
||||||||
|
Y05
|
772
|
Z05
|
848
|
||||||||
|
Y06
|
909
|
Z06
|
998
|
||||||||
|
A00: Model capacity identifier A00 is used for IFL-only or ICF-only configurations.
|
2.7.6 Capacity Backup Upgrade
Capacity Backup Upgrade (CBU) delivers temporary backup capacity in addition to the capacity that an installation might have available in numbers of assigned CPs, IFLs, ICFs, zIIPs, and optional SAPs. CBU has the following types:
•CBU for CP
•CBU for IFL
•CBU for ICF
•CBU for zIIP
•CBU for more (optional) SAPs
When CBU for CP is added within the same capacity setting range (indicated by the model capacity indicator) as the currently assigned PUs, the total number of active PUs (the sum of all assigned CPs, IFLs, ICFs, zIIPs, and optional SAPs) plus the number of CBUs cannot exceed the total number of PUs that are available in the system.
When CBU for CP capacity is acquired by switching from one capacity setting to another, no more CBUs can be requested than the total number of PUs available for that capacity setting.
CBU and granular capacity
When CBU for CP is ordered, it replaces lost capacity for disaster recovery. Specialty engines (ICFs, IFLs, and zIIPs) always run at full capacity, and when running as a CBU to replace lost capacity for disaster recovery.
When you order CBU, specify the maximum number of CPs, ICFs, IFLs, zIIPs, and SAPs to be activated for disaster recovery. If a disaster occurs, you decide how many of each of the contracted CBUs of any type to activate. The CBU rights are registered in one or more records in the CPC. Up to eight records can be active, which can contain various CBU activation variations that apply to the installation.
The number of CBU test activations that you can run for no extra fee in each CBU record is now determined by the number of years that are purchased with the CBU record. For example, a three-year CBU record includes three test activations, as compared to a one-year CBU record that has one test activation.
You can increase the number of tests up to a maximum of 15 for each CBU record. The real activation of CBU lasts up to 90 days with a grace period of two days to prevent sudden deactivation when the 90-day period expires. The contract duration can be set 1 - 5 years.
The CBU record describes the following properties that are related to the CBU:
•Number of CP CBUs that are allowed to be activated
•Number of IFL CBUs that are allowed to be activated
•Number of ICF CBUs that are allowed to be activated
•Number of zIIP CBUs that are allowed to be activated
•Number of SAP CBUs that are allowed to be activated
•Number of extra CBU tests that are allowed for this CBU record
•Number of total CBU years ordered (duration of the contract)
•Expiration date of the CBU contract
The record content of the CBU configuration is documented in IBM configurator output, which is shown in Example 2-1. In this example, one CBU record is made for a five-year CBU contract without more CBU tests for the activation of one CP CBU.
Example 2-1 Simple CBU record and related configuration features
On-Demand Capacity Selections:
NEW00001 - CBU - CP(1) - Years(5) - Tests(5)
Resulting feature numbers in configuration:
6817 Total CBU Years Ordered 5
6818 CBU Records Ordered 1
6820 Single CBU CP-Year 5 
In Example 2-2, a second CBU record is added to the configuration for two CP CBUs, two IFL CBUs, and two zIIP CBUs, with five more tests and a five-year CBU contract. The result is that a total number of 10 years of CBU ordered: Five years in the first record and five years in the second record. The two CBU records are independent and can be activated individually. Five more CBU tests were requested. Because a total of five years are contracted for a total of three CP CBUs (two IFL CBUs and two zIIP CBUs), they are shown as 15, 10, 10, and 10 CBU years for their respective types.
Example 2-2 Second CBU record and resulting configuration features
NEW00001 - CBU - Replenishment is required to reactivate
Expiration(06/21/2017)
NEW00002 - CBU - CP(2) - IFL(2) - zIIP(2)
Total Tests(5) - Years(5)
Resulting cumulative feature numbers in configuration:
6817 Total CBU Years Ordered 10
6818 CBU Records Ordered 2
6819 5 Additional CBU Tests 1
6820 Single CBU CP-Year 15
6822 Single CBU IFL-Year 10
6828 Single CBU zIIP-Year 10
CBU for CP rules
Consider the following guidelines when you are planning for CBU for CP capacity:
•The total CBU CP capacity features are equal to the number of added CPs plus the number of permanent CPs that change the capacity level. For example, if two CBU CPs are added to the current model D03, and the capacity level does not change, the D03 becomes D05, as shown in the following example:
(D03 + 2 = D05)
If the capacity level changes to a E06, the number of extra CPs (three) is added to the three CPs of the D03, which results in a total number of CBU CP capacity features of six:
(3 + 3 = 6)
•The CBU cannot decrease the number of CPs.
•The CBU cannot lower the capacity setting.
|
Remember: CBU for CPs, IFLs, ICFs, zIIPs, and SAPs can be activated together with On/Off Capacity on-Demand (CoD) temporary upgrades. Both facilities can be on a single system, and can be activated simultaneously.
|
CBU for specialty engines
Specialty engines (ICFs, IFLs, and zIIPs) run at full capacity for all capacity settings. This fact also applies to CBU for specialty engines. The minimum and maximum (min-max) numbers of all types of CBUs that can be activated on each of the models are listed in Table 2-11. The CBU record can contain larger numbers of CBUs than can fit in the current model.
Table 2-11 Capacity Backup matrix
|
Model
ZR1
|
Total PUs available
|
CBU
CPs
min - max
|
CBU
IFLs
min - max
|
CBU
ICFs
min - max
|
CBU
zIIPs
min - max
|
CBU
SAPs
min - max
|
|
Max30
|
30
|
0-6
|
0 - 30
|
0 - 30
|
0 - 12
|
0 - 2
|
|
Max24
|
24
|
0-6
|
0 - 24
|
0 - 24
|
0 - 12
|
0 - 2
|
|
Max12
|
12
|
0-6
|
0 - 12
|
0 - 12
|
0 - 8
|
0 - 2
|
|
Max4
|
4
|
0-4
|
0 - 4
|
0 - 4
|
0 - 2
|
0 - 2
|
2.7.7 On/Off Capacity on Demand and CPs
On/Off CoD provides temporary capacity for all types of characterized PUs. Relative to granular capacity, On/Off CoD for CPs is treated similarly to the way that CBU is handled.
On/Off CoD and granular capacity
When temporary capacity that is requested by On/Off CoD for CPs matches the model capacity identifier range of the permanent CP feature, the total number of active CPs equals the sum of the number of permanent CPs plus the number of temporary CPs ordered. For example, when a model capacity identifier D03 has two CPs added temporarily, it becomes a model capacity identifier D05.
When the addition of temporary capacity that is requested by On/Off CoD for CPs results in a cross-over from one capacity identifier range to another, the total number of CPs active when the temporary CPs are activated is equal to the number of temporary CPs ordered. For example, when a configuration with model capacity identifier D03 specifies four temporary CPs through On/Off CoD, the result is a server with model capacity identifier E05.
A cross-over does not necessarily mean that the CP count for the extra temporary capacity increases. The same D03 can temporarily be upgraded to a server with model capacity identifier F03. In this case, the number of CPs does not increase, but more temporary capacity is achieved.
On/Off CoD guidelines
When you request temporary capacity, consider the following guidelines:
•Temporary capacity must be greater than permanent capacity.
•Temporary capacity cannot be more than double the purchased capacity.
•On/Off CoD cannot decrease the number of engines on the CPC.
•The number of engines cannot be increased to more than what is installed.
For more information about temporary capacity increases, see Chapter 8, “System upgrades” on page 281.
2.8 Power and cooling
The z14 ZR1 power and cooling system is a change from previous systems because the system is packaged in an industry standard 19-inch rack form factor for all the internal system elements. The power subsystem is based on Power Distribution Units (PDUs) that are mounted at the rear of the system in pairs. The new PSCN structure uses industry standard Ethernet TOR switches that replace the previous IBM System Control Hubs (SCHs).
2.8.1 Considerations
The IBM Z systems operate with redundant power infrastructure. The z14 ZR1 is designed with a new power infrastructure that is based on intelligent (PDUs that are mounted vertically on the rear side of the 19-inch rack) and Power Supply Units for the internal components. The PDUs are single phase, 200 - 240 VAC and are controlled by using an Ethernet port.
The power supply units convert the AC power to DC power that is used as input for the Points of Load (POLs) in the CPC drawer and the PCIe+ I/O drawers.
The power requirements depend on the number of PU SCMs that is installed in the CPC drawer and number of PCIe+ I/O drawers (1 - 4) and I/O features that are installed in the PCIe+ I/O drawers.
The power subsystem in a z14 ZR1 includes the following main characteristics:
•Two or four single phase PDUs (200 - 240 VAC, 50/60 Hz), each with a 30A power cord.
•Maximum supported rack power is 9600W.
•No High-Voltage DC power option.
•No three-phase power, no 480 VAC.
•No Emergency Power Off (EPO) switch.
•No balanced power option or plan ahead power cables (installation of extar PDUs and PCIe+ I/O drawers is nondisruptive if enough PCIe fanouts are available in the CPC drawer).
•No Internal Battery Feature (IBF). The standard data center power protection (uninterruptible power supply based) can be used.
•If the 16U Reserved feature (FC 0617) is installed, the total power consumption for equipment that is installed under this feature is limited to 3400W. For more information, see Appendix G, “16U Reserved feature” on page 465.
The following previous features are not available on the ZR1:
•Internal Battery Feature (IBF)
•480V AC
•High-Voltage DC Power
•Balanced Power, no plan ahead power cords
•external EPO (Emergency Power Off) switch
•Three-Phase Power
•Single SE Display and keyboard, which includes a KVM switch
2.8.2 Power and weight estimation tool
By using the power and weight estimation tool for the z14 ZR1 server, you can enter your precise server configuration to obtain an estimate of power consumption. Log in to the Resource link with your user ID. Click Planning → Tools → Power and weight estimation Tools. Specify your server and the quantity for the features that are installed in your system. This tool estimates the power consumption for the specified configuration. The tool does not verify that the specified configuration can be physically built.
The exact power consumption for your system varies. The object of the tool is to estimate the power and weight requirements to aid you in planning for your system installation. Actual power consumption after installation can be confirmed by using the HMC Monitors Dashboard task and the CPC details panel (Energy Management tab), as shown in Figure 2-24.
Figure 2-24 CPC Details Panel: Energy Management tab
2.8.3 Cooling requirements
The z14 ZR1 is an air-cooled system. With the new design, no cables are used in the front of the system. This configuration results in better air flow. Air flow is front-to-rear (front is cold air input, rear is warm air exhaust). Chilled air, ideally coming from under a raised floor, is required to fulfill the cooling requirements. The chilled air is often provided through perforated floor tiles.
For more information about the amount of chilled air that is required for various temperatures under the floor of the computer room, see IBM 3907 Installation Manual for Physical Planning, GC28-6974.
The z14 ZR1 is classified as ASHRAE class A3 compliant.
If the 16U Reserved feature is present, the non-Z equipment that is installed in the space that is provided by the feature must conform with the specifications that are described in IBM 3907 Installation Manual for Physical Planning, GC28-6974.
The 16U Reserved feature includes the following basic requirements:
•Front-to-rear airflow
•No cables in front of the rack
•Visible conformity labels (according to country of installation requirements)
•Unused rack space must be covered with fillers to provide proper air flow
2.9 Summary
All aspects of the z14 ZR1 structure are listed in Table 2-12.
Table 2-12 System structure summary
|
Description
|
Max4
|
Max12
|
Max24
|
Max30
|
|
Number of CPC drawers
|
1
|
1
|
1
|
1
|
|
Number of SCMs
|
2
|
3
|
5
|
5
|
|
Total number of PU SCMs
|
1
|
2
|
4
|
4
|
|
Total number of SC SCMs
|
1
|
1
|
1
|
1
|
|
Total number of PUs
|
8
|
16
|
28
|
34
|
|
Maximum number of characterized PUs
|
4
|
12
|
24
|
30
|
|
Number of CPs
|
0 - 4
|
0 - 6
|
0 - 6
|
0 - 6
|
|
Number of IFLs
|
0 - 4
|
0 - 12
|
0 - 24
|
0 - 30
|
|
Number of ICFs
|
0 - 4
|
0 - 12
|
0 - 24
|
0 - 30
|
|
Number of zIIPs
|
0 - 2
|
0 - 8
|
0 - 12
|
0 - 12
|
|
Standard SAPs
|
2
|
2
|
2
|
2
|
|
Additional SAPs
|
0 - 2
|
0 - 2
|
0 - 2
|
0 - 2
|
|
Number of IFP
|
1
|
1
|
1
|
1
|
|
Standard spare PUs
|
1
|
1
|
1
|
1
|
|
Enabled memory sizes GB
|
64 - 1984
|
64 - 4032
|
64 - 8182
|
64 - 8182
|
|
L1 cache per PU (I/D)
|
128/128 KB
|
128/128 KB
|
128/128 KB
|
128/128 KB
|
|
L2 cache per PU
|
2/4 MB (I/D)
|
2/4 MB (I/D)
|
2/4 MB (I/D)
|
2/4 MB (I/D)
|
|
L3 shared cache per PU SCM
|
128 MB
|
128 MB
|
128 MB
|
128 MB
|
|
L4 shared cache per node
|
672 MB
|
672 MB
|
672 MB
|
672 MB
|
|
Cycle time (ns)
|
0.888
|
0.888
|
0.888
|
0.888
|
|
Clock frequency
|
4.5 GHz
|
4.5 GHz
|
4.5 GHz
|
4.5 GHz
|
|
Maximum number of PCIe fanouts
|
2
|
4
|
8
|
8
|
|
I/O interface per PCIe cable
|
16 GBps
|
16 GBps
|
16 GBps
|
16 GBps
|
|
Number of support elements
|
2
|
2
|
2
|
2
|
|
External AC power
|
Single-phase
|
Single-phase
|
Single-phase
|
Single-phase
|
1 For more information, see Chapter2, “Environmental specifications” in the IBM 3907 Installation Manual for Physical Planning, GC28-6974-00.
2 The components cannot be installed in a customer supplied rack. The term “frame” can also be used interchangeably with “rack”.
4 Examples of hardware that can be installed in the unused space are: IBM rack-mounted HMC (1U), IBM rack-mounted TKE (1U), IBM V7000 storage, IBM V9000 storage, SAN switches, Network switches or other equipment that is designed to fit in a 19-inch rack. Non-IBM equipment can also be installed in the space reserved.
5 POL - Point of Load, VRM - Voltage Regulator Module.
6 z14 ZR1 is a single CPC drawer system; therefore, the A-Bus is not used. Also, the S-Bus (SC-to-SC connectivity) that was available in z13s was eliminated because of single SC SCM that incorporates a single L4 cache.
7 z/OS V1.13 includes more requirements. For more information, see Chapter 7, “Operating system support” on page 209.
8 Voltage Regulator Module stick converts the DC bulk power that is delivered by the PSUs (12V) into localized low voltage that is used by the installed components (for example, PU SCMs, SC SCM, memory DIMMs, and other circuitry).
9 The KVM hypervisor is part of select Linux on Z distributions.
10 Two standard SAPs with every ZR1 system are always featured. Up to two more SAPs can be ordered. The extra SAPs are part of the customer characterizable PUs.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.