Introducing the IBM z14
This chapter describes the basic concepts and design considerations around IBM z14TM servers and includes the following topics:
1.1 Design considerations for the IBM z14
Digital technologies are driving major industry shifts, ushering in a new age. In the IBM Cognitive Era®,1 it is either disrupting or being disrupted. Tomorrow’s disruptors will be organizations that can converge digital business with a new level of digital intelligence. A key in digital transformation is the ability to accelerate the innovation of new business, such as Internet of Things (IoT) and Cognitive and Virtual Reality, while keeping complexity in check.
Information Technology (IT) today is experiencing a time of exponential growth in data and transaction volumes that are driven by digital transformation. New dynamics in the market provide opportunities for businesses to grab market share and win. As companies struggle to meet the demands of the new IT, which is dominated by cloud, analytics, mobile, social, and IoT, IT leaders are now being challenged to add value by opening their enterprises to new ways of doing business.
In this challenging landscape, businesses must manage, store, protect, and, most importantly, use the information for gaining competitive advantage. This need is creating the demand to apply intelligence and insight to the data to build new services that are wrapped for a customized user experience.
By creating applications that provide intelligent user experiences, companies can provide value to their partners and clients, which ultimately preserves loyalty. And most of all to succeed, organizations must provide their users with the peace of mind that no matter what device is being used, data is protected.
Organizations in every industry and sector must secure that growing data and comply with increasingly intricate regulations. This operational environment increases the pressure on IT to securely deliver services and support on time and on budget. By encrypting as much of your data and transactional pipeline as possible, you can reduce potential data breach risks and financial losses, and comply with complex regulatory mandates.
Cryptography always is in the DNA of IBM Z family. The IBM z14 continues that tradition with pervasive encryption to defend and protect your critical assets with unrivaled encryption and intelligent data monitoring without compromising transactional throughput or response times. Most importantly, this pervasive encryption requires no application changes. Pervasive encryption can dramatically simplify data protection and reduce the costs of regulatory compliance. By using simple policy controls, z14 pervasive computing streamlines data protection for mission critical IBM Db2® for z/OS, IBM IMS, and Virtual Storage Access Method (VSAM) datasets.
The Central Processor Assist for Cryptographic Function (CPACF), which is standard on every core, supports pervasive encryption and provides hardware acceleration for encryption operations. The new Crypto Express6S gets a performance boost on z14. Combined, these enhancements perform encryption more efficiently on the z14 than on earlier IBM Z servers.
The IBM z14 was designed specifically to meet the demand for new services and customer experiences, while securing the growing amounts of data and complying with increasingly intricate regulations. With up to 170 configurable cores, z14 has performance and scaling advantage over prior generations and 31% more capacity than the 141-way z13.
The new FICON Express16S+ delivers an increase in I/O rates and in-link bandwidth, and a reduction in single-stream latency, which provides the system the ability to absorb large applications and transaction spikes driven by unpredictable mobile and IoT devices.
Next-generation SMT on z14 delivers improved virtualization performance to benefit Linux. High-speed connectivity out to the data is critical in achieving exceptional levels of transaction throughput. The new IBM zHyperLink Express introduces disk I/O technology for accessing the IBM DS8880 storage system with low latency, which enables shorter batch windows and a more resilient I/O infrastructure with predictable and repeatable I/O performance.
With up to 32 TB of memory, z14 can open opportunities, such as in-memory data marts and in-memory analytics while giving you the necessary room to tune applications for optimal performance. By using the Vector Packed Decimal Facility that allows packed decimal operations to be performed in registers rather than memory, and by using new fast mathematical computations, compilers (such as Enterprise COBOL for z/OS, V6.2, Enterprise PL/I for z/OS, V5.2, z/OS V2.3 XL C/C++), the COBOL optimizer, Automatic Binary Optimizer for z/OS, V1.3, and Java, are optimized on z14. These compilers and optimizer are designed to improve application performance, reduce CPU usage, and reduce operating costs. Java improvements and the use of crypto acceleration deliver more improvements in throughput per core, which gives a natural boost to z/OS Connect EE, IBM WebSphere® Liberty in IBM CICS®, Spark for z/OS, and IBM Java for Linux on Z.
Smoothly handling the data tsunami requires robust infrastructure that is designed specifically for high-volume data transactions. To take advantage of new unstructured data,2 businesses on IBM z can use application programming interfaces (APIs) that can help with creating and delivering innovative new services.
Linux on IBM Z, which is optimized for open source software, brings more value to the platform. Linux on IBM Z supports a wealth of new products that are familiar to application developers, such as Python, Scala, Spark, MongoDB, PostgreSQL, and MariaDB. Access to data that was unavailable, without the need for Extract Transform and Load (ETL) allows for the development of intelligent transactions and intuitive business processes.
As your business technology needs evolve to compete in today’s digital economy, IBM stands ready to help with intelligent, robust, and comprehensive technology solutions. The IBM approach integrates server, software, and storage solutions to ensure that each member of the stack is designed and optimized to work together. The new IBM z14™ leads that approach by delivering the power and speed users demand, the security users and regulators require, and the operational efficiency that maximizes your bottom line.
 
Terminology: The remainder of this book uses the designation CPC to refer to the central processor complex.
1.2 z14 server highlights
This section reviews some of the most important features and functions of z14 (Driver 32) servers:
1.2.1 Processor and memory
IBM continues its technology leadership with the z14 server. The z14 server is built by using the IBM modular multi-drawer design that supports 1 - 4 processor drawers per CPC. Each processor drawer contains five or six Central Processor (CP) single-chip modules (SCMs) and one Storage Controller (SC) SCM. Both SCMs are redesigned by using 14 nm FINFET SOI technology.3 Each CP SCM has 10 processor units (PUs, or cores). In addition to SCMs, CPC drawers host memory DIMMs, connectors for I/O, oscillator interface, Flexible Service Processors (FSPs), and cooling manifolds.
The superscalar processor implements second-generation SMT4 now enabled for System Assist Processors (SAPs). It also implements redesigned caches and translation lookaside buffer (TLB), optimized pipeline, and better branch prediction. Also featured is an expanded instruction set with Vector Packed Decimal Facility, Guarded Storage Facility, Vector Facility enhancements, Semaphore Assist Facility, Order Preserving Compression, and Entropy Encoding for Co-processor Compression for better performance in several different areas.
Depending on the model, the z14 server can support 256 GB - 32 TB of usable memory, with up to 8 TB of usable memory per CPC drawer. In addition, a fixed amount of 192 GB is reserved for the hardware system area (HSA) and is not part of customer-purchased memory. Memory is implemented as a redundant array of independent memory (RAIM) and uses extra physical memory as spare memory. The RAIM function accounts for 20% of the physical installed memory in each CPC drawer.
New with z14, Virtual Flash Memory (VFM) feature is offered from the main memory capacity in 1.5 TB units and replaces the Flash Express adapters, which were available on the zEC12 and z13. VFM provides much simpler management and better performance by eliminating the I/O the adapters in the PCIe drawers. VFM does not require any application changes when moving from IBM Flash Express.
1.2.2 Capacity and performance
The z14 server provides increased processing and enhanced I/O capabilities over its predecessor, the z13 system. This capacity is achieved by increasing the performance of the individual PUs, increasing the number of PUs per system, redesigning the system cache, increasing the amount of memory, and introducing new and I/O technologies.
The increased performance and the total system capacity available (with possible energy savings) allow consolidating diverse applications on a single platform with significant financial savings. The introduction of new technologies and an expanded and enhanced instruction set ensure that the z14 server is a high-performance, reliable, and rich-security platform. The z14 server is designed to maximize the use of resources and allows you to integrate and consolidate applications and data across the enterprise IT infrastructure.
z14 servers are offered in five models, with 1 - 170 configurable PUs. Models M01, M02, M03, and M04 have up to 41 PUs per CPC drawer. The high-capacity model (the M05) has four processor (CPC) drawers with 49 PU per drawer. Model M05 is estimated to provide up to 35% more total system capacity than the z13 Model NE1, with the same amount of memory and power requirements. With up to 32 TB of main storage and enhanced SMT, the performance of the z14 processors deliver considerable improvement. Uniprocessor performance also increased significantly. A z14 Model 701 offers average performance improvements of 10%5 over the z13 Model 701.
The IFL zIIP processor units on the z14 server can be configured to run two simultaneous threads per clock cycle in a single processor (SMT). This feature increases the capacity of these processors with 25% in average6 over processors that are running single thread. SMT is also enabled by default on SAPs.
The z14 server expands the subcapacity settings, offering three subcapacity levels (in models 4xx, 5xx and 6xx) for up to 33 processors that are characterized as CPs (compared to up to 30 for z13). This configuration gives a total of 269 distinct capacity settings. The z14 servers deliver scalability and granularity to meet the needs of medium-sized enterprises, while also satisfying the requirements of large enterprises that have demanding, mission-critical transaction and data processing requirements.
This comparison is based on the Large System Performance Reference (LSPR) mixed workload analysis. For more information about performance and workload variation on z14 servers, see Chapter 12, “Performance” on page 447.
z14 servers continue to offer all the specialty engines7 that are available on z13.
Workload variability
Consult the LSPR when considering performance on z14 servers. The range of performance ratings across the individual LSPR workloads is likely to have a large spread. More performance variation of individual logical partitions (LPARs) is available when an increased number of partitions and more PUs are available. For more information, see Chapter 12, “Performance” on page 447.
For detailed performance information, see the LSPR website.
For more information about millions of service units (MSUs) ratings, see the IBM Z Software Contracts website.
Capacity on demand
Capacity on demand (CoD) enhancements enable clients to have more flexibility in managing and administering their temporary capacity requirements. The z14 server supports the same architectural approach for CoD offerings as the z13 (temporary or permanent). Within the z14 server, one or more flexible configuration definitions can be available to solve multiple temporary situations, and multiple capacity configurations can be active simultaneously.
Up to 200 staged records can be created to handle many scenarios. Up to eight of these records can be installed on the server at any time. After the records are installed, the activation of the records can be done manually, or the z/OS Capacity Provisioning Manager can automatically start the activation when Workload Manager (WLM) policy thresholds are reached. Tokens are available that can be purchased for On/Off CoD before or after workload execution (pre- or post-paid).
LPAR capping
IBM Processor Resource/Systems Manager™ (IBM PR/SM™) offers different options to limit the amount of capacity that is assigned to and used by an LPAR or a group of LPARs. By using the Hardware Management Console (HMC), a user can define an absolute or a relative capping value for LPARs that are running on the system.
1.2.3 Virtualization
This section describes built-in virtualization capabilities of z14 supporting operating systems, hypervisors, and available virtual appliances.
z14 servers support z/Architecture mode only, which can be initialized in LPAR mode (also known as PR/SM) or Dynamic Partition Manager (DPM) mode.
PR/SM mode
PR/SM is Licensed Internal Code (LIC) that manages and virtualizes all the installed and enabled system resources as a single large symmetric multiprocessor (SMP) system. This virtualization enables full sharing of the installed resources with high security and efficiency.
PR/SM supports configuring up to 85 LPARs, each of which includes logical processors, memory, and I/O resources. Resources of these LPARs are assigned from the installed CPC drawers and features. For more information about PR/SM functions, see 3.7, “Logical partitioning” on page 125.
LPAR configurations can be dynamically adjusted to optimize the virtual servers’ workloads. z14 servers provide improvements to the PR/SM HiperDispatch function. HiperDispatch provides alignment of logical processors to physical processors that ultimately improves cache utilization, minimizes inter-CPC drawer communication, and optimizes operating system work dispatching, which combined results in increased throughput. For more information, see “HiperDispatch” on page 93.
HiperSockets
z14 servers support defining up to 32 IBM HiperSockets™. HiperSockets provide for memory-to-memory communication across LPARs without the need for any I/O adapters and have virtual LAN (VLAN) capability.
Dynamic Partition Manager mode
DPM is an administrative mode (front end to PR/SM) that was introduced for Linux only systems for IBM z14, IBM z13, IBM z13s®, and IBM LinuxONE™ servers. A system can be configured in DPM mode or in PR/SM mode (POR is required to switch modes). DPM supports the following functions:
•Create, provision, and manage partitions (processor, memory, and adapters)
•Monitor and troubleshoot the environment
LPAR modes on z14
The following PR/SM LPAR modes with corresponding operating systems and firmware appliances are supported:
•General:
 – z/OS
 – IBM z/VM®
 – IBM z/VSE®
 – z/TPF
 – Linux on IBM Z
•Coupling Facility: Coupling Facility Control Code (CFCC)
•Linux only:
 – Linux on IBM Z
 – z/VM
•z/VM
•Secure Service Container:
 – VNA (z/VSE Network Appliance)
 – IBM High Security Business Network (HSBN)8
The following LPAR modes are available for DPM:
•z/VM
•Linux on IBM Z (also used for KVM deployments)
•Secure Service Container
IBM Z servers also offer other virtual appliance-based solutions and support other the following hypervisors and containerization:
•IBM GDPS® Virtual Appliance
•KVM for IBM Z
•Docker Enterprise Edition for Linux on IBM Systems9
Coupling Facility mode logical partition
Parallel Sysplex is a synergy between hardware and software; which is a highly advanced technology for clustering that is designed to enable the aggregate capacity of multiple z/OS systems to be applied against common workloads. To use this technology, a special LIC is used, which is called CFCC. To activate the CFCC, a special logical partition must be defined. Only PUs that are characterized as CPs or Internal Coupling Facilities (ICFs) can be used for Coupling Facility (CF) partitions. For a production CF workload, it is recommended to use dedicated ICFs.
The z/VM-mode LPAR
z14 servers support an LPAR mode, called z/VM-mode, that is exclusively for running z/VM as the first-level operating system. The z/VM-mode requires z/VM V6R4 or later, and allows z/VM to use a wider variety of specialty processors in a single LPAR, which increases flexibility and simplifying system management.
For example, in a z/VM-mode LPAR, z/VM can manage Linux on IBM Z guests that are running on IFL processors while also managing z/VSE and z/OS guests on CPs. It also allows z/OS to fully use zIIPs.
Secure Service Container
IBM Secure Service Container (SSC) is an enabling technology for building virtual appliances (exploiters). It provides the base infrastructure to build and host virtual appliances on IBM Z.
SSC can be used to create isolated partitions for protecting data and applications automatically, which helps keep them safe from insider threats and external cyber criminals. SSC offers the following benefits:
•Streamline the IBM Z Application experience so it is comparable to installing an application on a mobile device.
•Deploy an appliance in minutes instead of days.
•Protect the workload from being accessed by a sysadmin or external attacker.
IBM Z/VSE Network Appliance
The z/VSE Network Appliance builds on the z/VSE Linux Fast Path (LFP) function and provides Internet Protocol network access without requiring a TCP/IP stack in z/VSE. The appliance uses the SSC infrastructure that was introduced on z13 and z13s servers. Compared to a TCP/IP stack in z/VSE, this network appliance can support higher TCP/IP traffic throughput while reducing the processing resource consumption in z/VSE.
The z/VSE Network Appliance is an extension of the z/VSE - z/VM IP Assist (IBM VIA®) function that was introduced on z114 and z196 servers. VIA provides network access for TCP/IP socket applications that run on z/VSE as a z/VM guest. With the new z/VSE Network Appliance, this function is available for z/VSE systems that are running in an LPAR. The z/VSE Network Appliance is provided as a downloadable package that can then be deployed with the SSC Installer and Loader.
The VIA function is available for z/VSE systems that run as z/VM guests. The z/VSE Network Appliance is available for z/VSE systems that run without z/VM in LPARs. Both functions provide network access for TCP/IP socket applications that use the LFP without the requirement of TCP/IP stack on the z/VSE system and installing Linux on IBM Z.
 
IBM zAware: With Announcement Letter 916-201 dated November 1, 2016, IBM changed how IBM System z Advanced Workload Analysis Reporter (IBM zAware) is delivered. IBM zAware was available as a firmware feature on zEC12, zBC12, z13. Also, z13s is now offered as a software feature with IBM Operations Analytics for Z.
IBM Operations Analytics for Z brings new capabilities and functions to the product, or applies maintenance, based on user schedules, that is not tied to IBM Z firmware updates. Integration with IBM Operational Analytics for Z Problem Insights dashboard eliminates the need for tedious searching through volumes of operational data, which puts key operational issues at your fingertips. New functions, such as proactive outage avoidance with email alerts, improve the users’ ability to respond to identified anomalies.
GDPS Virtual Appliance
The GDPS Virtual Appliance solution implements GDPS/PPRC Multiplatform Resilience for IBM Z (xDR). xDR coordinates near-continuous availability and a disaster recovery (DR) solution through the following features:
•Disk error detection
•Heartbeat for smoke tests
•Re-IPL in place
•Coordinated site takeover
•Coordinated IBM HyperSwap®
•Single point of control
1.2.4 I/O subsystem and I/O features
The z14 server supports PCIe and InfiniBand I/O infrastructure. PCIe features are installed in PCIe I/O drawers. Up to five PCIe I/O drawers per z14 server are supported, which provides space for up to 160 PCIe I/O features. I/O drawers10 are not supported on z14 servers and cannot be carried forward during an upgrade from a z13 or zEC12 server.
 
Coupling Connectivity (Statement of Direction1 fulfillment): IBM z14 (Machine Type 3906) is the last server to support InfiniBand coupling features (HCA3-O and HCA3-O LR adapters).
Enterprises should migrate from HCA3-O and HCA3-O LR adapters to ICA SR or Coupling Express Long Reach (CE LR) adapters on z14, z13, and z13s. For high-speed short-range coupling connectivity, enterprises should migrate to the Integrated Coupling Adapter (ICA-SR).
For long-range coupling connectivity, enterprises should migrate to the new CE LR coupling adapter. For long-range coupling connectivity requiring a DWDM, enterprises must determine their wanted DWDM vendor’s plan to qualify the planned replacement long-range coupling link.
1 All statements regarding IBM plans, directions, and intent are subject to change or withdrawal without notice. Any reliance on these statements of general direction is at the relying party’s sole risk and will not create liability or obligation for IBM.
For a four CPC drawer system, up to 40 PCIe and 16 InfiniBand fanout slots can be configured for data communications between the CPC drawers and the I/O infrastructure, and for coupling. The multiple channel subsystem (CSS) architecture allows up to six CSSs, each with 256 channels.
For I/O constraint relief, four subchannel sets are available per CSS, which allows access to many logical volumes. The fourth subchannel set allows extending the amount of addressable external storage for Parallel Access Volumes (PAVs), Peer-to-Peer Remote Copy (PPRC) secondary devices, and IBM FlashCopy® devices. z14 supports Initial Program Load (IPL) from subchannel set 1 (SS1), subchannel set 2 (SS2), or subchannel set 3 (SS3), and subchannel set 0 (SS0). For more information, see “Initial program load from an alternative subchannel set” on page 200.
The system I/O buses use the Peripheral Component Interconnect® Express (PCIe) technology and the InfiniBand technology, which are also used in coupling links.
z14 connectivity supports the following I/O or special purpose features:
 – Fibre Channel connection (IBM FICON):
 • FICON Express16S+ 10 KM long wavelength (LX) and short wavelength (SX)
 • FICON Express16S 10 KM LX and SX (carry forward only)
 • FICON Express8S 10 KM LX and SX (carry forward only)
 – Open Systems Adapter (OSA):
 • OSA-Express7S 25GbE SR
 • OSA-Express6S 10 GbE long reach (LR) and short reach (SR)
 • OSA-Express6S GbE LX and SX
 • OSA-Express6S 1000BASE-T Ethernet
 • OSA-Express5S 10 GbE LR and SR (carry forward only)
 • OSA-Express5S GbE LX and SX (carry forward only)
 • OSA-Express5S 1000BASE-T Ethernet (carry forward only)
 • OSA-Express4S 1000BASE-T Ethernet (carry forward only)
 – IBM HiperSockets
 – Shared Memory Communication - Remote Direct Memory Access (SMC-R):
 • 25GbE RoCE (RDMA over Converged Ethernet) Express2
 • 10GbE RoCE Express2
 • 10GbE RoCE Express (carry forward only)
 – Shared Memory Communication - Direct Memory Access (SMC-D) through Internal Shared Memory (ISM)
 – Internal Coupling (IC) links
 – Integrated Coupling Adapter Short Reach (ICA SR)
 – CE LR
 – HCA3-O, 12x Parallel Sysplex InfiniBand (IFB) coupling links
 – HCA3-O, 1x Parallel Sysplex InfiniBand (IFB) coupling links
 – Crypto Express6S
 – Crypto Express5S (carry forward only)
 – Regional Crypto Enablement
•IBM zEnterprise® Data Compression (zEDC) Express features, which are installed in the PCIe I/O drawers (new build and carry forward).
1.2.5 Reliability, availability, and serviceability design
System reliability, availability, and serviceability (RAS) is an area of continuous IBM focus and a defining IBM Z platform characteristic. The RAS objective is to reduce, or eliminate if possible, all sources of planned and unplanned outages while providing adequate service information if an issue occurs. Adequate service information is required to determine the cause of an issue without the need to reproduce the context of an event.
IBM Z servers are designed to enable highest availability and lowest downtime. These facts are recognized by various IT analysts, such as ITIC11 and IDC12. Comprehensive, multi-layered strategy includes the following features:
•Error Prevention
•Error Detection and Correction
•Error Recovery
With a properly configured z14 server, further reduction of outages can be attained through First Failure Data Capture (FFDC), which is designed to reduce service times and avoid subsequent errors, and improve nondisruptive replace, repair, and upgrade functions for memory, drawers, and I/O adapters. In addition, z14 servers extended nondisruptive capability to download and install LIC updates.
IBM z14™ RAS features provide unique high-availability and nondisruptive operational capabilities that differentiate the Z servers in the marketplace. z14 RAS enhancements are made on many components of the CPC (processor chip, memory subsystem, I/O, and service) in areas, such as error checking, error protection, failure handling, error checking, faster repair capabilities, sparing, and cooling.
The ability to cluster multiple systems in a Parallel Sysplex takes the commercial strengths of the z/OS platform to higher levels of system management, scalable growth, and continuous availability.
1.3 z14 server technical overview
This section briefly reviews the following major elements of z14 servers:
1.3.1 Models
The IBM z14 server has a machine type of 3906. Five models are offered: M01, M02, M03, M04, and M05. The model name indicates the number of CPC drawers for models M01 - M04. Model M05 also has four CPC drawers, but with more PU per drawer than models M01 - M04. A PU is the generic term for the IBM z/Architecture processor unit (processor core) on the CP SCM.
On z14 servers, some PUs are part of the system base; that is, they are not part of the PUs that can be purchased by clients. They include the following characteristics:
•System assist processor (SAP) that is used by the channel subsystem. The number of predefined SAPs depends on the z14 model.
•One integrated firmware processor (IFP). The IFP is used in support of select features, such as zEDC and RoCE Express.
•Two spare PUs that can transparently assume any characterization during a permanent failure of another PU.
The PUs that clients can purchase can assume any of the following characteristics:
•CP for general-purpose use.
•Integrated Facility for Linux (IFL) for the use of Linux on Z.
•IBM Z Integrated Information Processor (zIIP13) is designed to help free-up general computing capacity and lower overall total cost of computing for select data and transaction processing workloads.
 
zIIPs: At least one CP must be purchased with, or before, a zIIP can be purchased. Clients can purchase up to two zIIPs for each purchased CP (assigned or unassigned) on the system (2:1). However, for migrations from zEC12 with zAAPs, the ratio (CP:zIIP) can go up to 4:1.
•Internal Coupling Facility (ICF) is used by the CFCC.
•More SAPs are used by the channel subsystem.
A PU that is not characterized cannot be used, but is available as a spare. The following rules apply:
•In the five-model structure, at least one CP, ICF, or IFL must be purchased and activated for any model.
•PUs can be purchased in single PU increments and are orderable by feature code.
•The total number of PUs purchased cannot exceed the total number that are available for that model.
•The number of installed zIIPs cannot exceed twice the number of installed CPs.
The multi-CPC drawer system design provides the capability to concurrently increase the capacity of the system in the following ways:
•Add capacity by concurrently activating more CPs, IFLs, ICFs, or zIIPs on a CPC drawer.
•Add a CPC drawer concurrently and activate more CPs, IFLs, ICFs, or zIIPs.
•Add a CPC drawer to provide more memory, or one or more adapters to support a larger number of I/O features.
1.3.2 Model upgrade paths
Upgrades from Models M01, M02, and M03 to M02, M03, and M04 are concurrent. Any z13 or zEC12 model can be upgraded to z14 models M01, M02, M03, M04, and M05 disruptively. Upgrades from z14 M01, M02, M03, or M04 to z14 model M05 are not supported. M05 is offered only as factory build. Figure 1-1 shows the supported upgrade paths.
 
Consideration: An air-cooled z14 server cannot be converted to a water-cooled z14 server, and vice versa.
Figure 1-1 z14 upgrades
z13 upgrade to z14
When a z13 is upgraded to a z14, the z13 driver level must be at least 27. Upgrading from z13 to z14 server is disruptive.
zEC12 upgrade to z14
When a zEC12 is upgraded to a z14, the zEC12 must be at least at Driver level 15. Upgrading from zEC12 to z14 servers is disruptive. zBX Model 003 cannot be upgraded to zBX Model 004 during an upgrade from zEC12 to z14.
The following processes are not supported:
•Downgrades within the z14 models
•Upgrade from a z13s or zBC12 to z14 servers
•Upgrades from z196 or earlier systems
•Attachment of a zBX Model 002 or model 003 to a z14 server
1.3.3 Frames
z14 servers have two frames that are bolted together and are known as the A frame and the Z frame. The frames contain the following components:
•Up to four CPC drawers in Frame A
•Up to five PCIe I/O drawers (up to one in Frame A and up to four in Frame Z) that hold I/O features and special purpose features
•Power supplies in Frame Z
•Optional Internal Battery Feature (IBF)
•Cooling units for either air or water cooling
•Two System Control Hubs (SCHs) to interconnect the CPC components through Ethernet
•Two 1U rack-mounted Support Elements (mounted in A frame) with their keyboards, pointing devices, and displays mounted on a tray in the Z frame
1.3.4 CPC drawer
Up to four CPC drawers are installed in frame A of z14 server. Each CPC drawer houses the SCMs, memory, and I/O interconnects.
Single Chip Module technology
z14 servers are built on the superscalar microprocessor architecture of its predecessor and provide various enhancements over the z13. Each CPC drawer has two logical CP clusters and one SC SCM. Two CPC drawer sizes are available in z14, depending on the number of CP SCMs. The z14 model M05 has six CP SCMs that include 49 active cores. All other z14 models have five CP SCMs that include 41 active cores. The CP SCM has 10 cores by design, with 7, 8, 9, or 10 active cores, which can be characterized as CPs, IFLs, ICFs, zIIPs, SAPs, or IFPs.
The SCM provides a significant increase in system scalability and an extra opportunity for server consolidation. All CPC drawers are fully interconnected by using high-speed communication links through the L4 cache (in the SC SCM). This configuration allows the z14 server to be controlled by the PR/SM facility as a memory-coherent and cache-coherent SMP system.
The PU configuration includes two designated spare PUs per CPC and a variable number of SAPs. The SAPs scale with the number of CPC drawers that are installed in the server. For example, five standard SAPs are available with one CPC drawer that is installed, and up to 23 when four CPC drawer are installed. In addition, one PU is used as an IFP and is not available for client use. The remaining PUs can be characterized as CPs, IFL processors, zIIPs, ICF processors, or extra SAPs. For z14 the SAPs have Simultaneous Multi-Threading that is enabled by default (cannot be changed).
The SCMs are cooled by a cold plate that is connected to an internal water cooling loop. In an air-cooled system, the radiator units (RUs) exchange the heat from the internal water loop with air. The RU has N+1 availability for pumps and blowers.
The z14 server offers also a water-cooling option for increased system and data center energy efficiency. The water cooling units (WCUs) are fully redundant in an N+1 arrangement.
Processor features
The processor core operates at 5.2 GHz. Depending on the z14 model, 41 - 196 active PUs are available on 1 - 4 CPC drawers.
Each core on the CP SCM includes an enhanced dedicated coprocessor for data compression and cryptographic functions, which are known as the Central Processor Assist for Cryptographic Functions (CPACF)14. Having standard clear key cryptographic coprocessors that are integrated with the processor provides high-speed cryptography for protecting data.
CPACF on z14 is improved to support pervasive encryption, which provides an envelope of protection around the data that is on IBM Z servers. CPACF encryption rates for like modes and data sizes on z14 are up to six times faster15 than z13. Enhancements include the support of SHA-3 and SHAKE hashing algorithms, True Random-number generation (TRNG), and the addition of a new hardware instruction to perform Galois Counter Mode (GCM) encryption.
Hardware data compression can play a significant role in improving performance and saving costs over performing compression in software. z14 is the fourth IBM Z generation having CMPSC, the on-chip compression co-processor. A new compression ratio with Entropy Encoding that uses Huffman coding and Order Preserving Compression in z14 results in fewer CPU cycles to enable further compression of data including Db2 indexes, which improves memory, transfer, and disk efficiency.
The zEDC Express feature complements the functionality of the coprocessor (CPACF). Their functions are not interchangeable.
The micro-architecture of the core was improved in several ways to increase parallelism and pipeline efficiency. z14 cores have 2x more on-chip cache, compared to z13 per core, to minimize memory waits while maximizing the throughput of concurrent workloads, which makes it perfectly optimized for data serving.
z14 includes a new translation lookaside buffer (TLB2) design with four hardware-implemented translation engines that reduces latency when compared with one pico-coded engine on z13. Pipeline optimization presents enhancements, including improved instruction delivery, faster branch wake-up, reduced execution latency, improved Operand Store Compare (OSC) prediction.
Processor RAS
In the unlikely case of a permanent core failure, each core can be individually replaced by one of the available spares. Core sparing is transparent to the operating system and applications.
Vector and Floating point Unit Enhancements
Each core has two Vector and Floating point Units (VFU) that contain subunits, such as single instruction multiple data (SIMD), binary floating-point units (BFUs), and decimal floating-point units (DFUs) that are enhanced with z14 in different ways.
z14 has a new decimal architecture with Vector Enhancements Facility and Vector Packed Decimal Facility for Data Access Accelerator. Vector Packed Decimal Facility introduces a set of instructions that perform operation on decimal type data that uses vector registers to improve performance. z14 offers up to 2x throughput of vector Binary Floating Point double precision operations and RSA/ECC acceleration (Long Multiply Support).
By using the Vector Packed Decimal Facility that allows packed decimal operations to be performed in registers rather than memory (by using new fast mathematical computations), compilers, such as Enterprise COBOL for z/OS, V6.2, Enterprise PL/I for z/OS, V5.2, z/OS V2.3 XL C/C++, the COBOL optimizer, Automatic Binary Optimizer for z/OS, V1.3, and Java, are optimized on z14.
Much of today’s commercial computing uses decimal floating point calculus, so on-core hardware decimal floating point units meet the requirements of business and user applications. This ability provides greater floating point execution throughput with improved performance and precision.
Hot line speculation avoidance
The new mechanism allows full out-of-order and branch speculation for any instruction that hits in L1/L2 caches, and allows for out-of-order TLB-miss handling. For known hot cache lines, the implementation does not allow sending requests to L3 speculatively and Load Store Unit (LSU) rejects the request until the instruction is next to complete.
Guarded Storage Facility
Also known as less-pausing garbage collection, Guarded Storage Facility is a new architecture that was introduced with z14 to enable enterprise scale Java applications to run without periodic pause for garbage collection on larger heaps. This facility improves Java performance by for reducing program pauses during Java Garbage Collection.
Instruction Execution Protection
Instruction Execution Protection (IEP) is a new hardware function that was introduced with z14 that enables software, such as IBM Language Environment®, to mark certain memory regions (for example, a heap or stack) as non-executable to improve the security of programs that are running on IBM Z servers against stack-overflow or similar attacks.
Simultaneous multithreading
z/Architecture introduced SMT support with z13, that allows simultaneous running of two threads (SMT) in the same zIIP or IFL core, which dynamically shares processor resources, such as execution units and caches. SMT in z13 allowed a more efficient use of the core and increased capacity because while one of the threads is waiting for a storage access (cache miss), the other thread that is running simultaneously in the core can use the shared resources rather than remain idle.
Adjusted with the growth in the core cache and TLB2, second-generation SMT on z14 improves thread balancing, supports multiple outstanding translations, optimizes hang avoidance mechanisms, and delivers improved virtualization performance to benefit Linux on Z. z14 provides economies of scale with next generation multithreading (SMT) for Linux and zIIP-eligible workloads. In z14, the I/O System Assist Processor (SAP) has SMT enabled by default.
Single instruction multiple data instruction set
Introduced with the IBM z13, the SIMD unit and instructions use the superscalar core to process operands in parallel, which enables more processor throughput.
z14 introduces a new decimal architecture and new SIMD (vector) instruction set, which are designed to boost performance for traditional workloads by using COBOL and new applications, such as analytics. The SIMD unit in z14 now supports 32-bit floating point operations. The use of enhanced mathematical libraries, such as OpenBLAS, provides performance improvements for analytical workloads.
Transactional execution facility
The IBM z14 includes a group of instructions that are run (in sequence) as a transaction; that is, if any of the instructions in the group fails, the entire group of instructions is run again from the beginning, which is known as transactional execution facility.
Out-of-order execution
As with its predecessor z13, z14 has an enhanced superscalar microprocessor with Out-of-Order execution to achieve faster throughput. With Out-of-Order, instructions might not run in the original program order, although results are presented in the original order. For example, Out-of-Order allows a few instructions to complete while another instruction is waiting. Up to six instructions can be decoded per system cycle, and up to 10 instructions can be in execution.
Concurrent processor unit conversions
The z14 supports concurrent conversion between various PU types, which provides the flexibility to meet the requirements of changing business environments. CPs, IFLs, zIIPs, ICFs, and optional SAPs can be converted to CPs, IFLs, zIIPs, ICFs, and optional SAPs.
Memory subsystem and topology
The z14 servers use double data rate fourth-generation (DDR4) dual inline memory module (DIMM) technology. For this purpose, IBM developed a chip that controls communication with the PU, which is an SC chip, and drives address and control from DIMM-to-DIMM. The DIMM is available in 32 GB, 64 GB, 128 GB, 256 GB, and 512 GB capacities.
Memory topology provides the following benefits:
•A RAIM for protection at the dynamic random access memory (DRAM), DIMM, and memory channel levels.
•A maximum of 32 TB of user configurable memory with a maximum of 40 TB of physical memory (with a maximum of 16 TB configurable to a single LPAR).
•One memory port for each CP SCM, and up to five independent memory ports per CPC drawer.
•Increased bandwidth between memory and I/O.
•Asymmetrical memory size and DRAM technology across CPC drawers.
•Large memory pages (1 MB and 2 GB).
•Key storage.
•Storage protection key array that is kept in physical memory.
•Storage protection (memory) key that is also kept in every L2 and L3 cache directory entry.
•A larger (192 GB) fixed-size HSA that eliminates planning for HSA.
PCIe fanout hot-plug
The PCIe fanout provides the path for data between memory and the PCIe features through the PCIe 16 GBps bus and cables. The PCIe fanout is hot-pluggable. During an outage, a redundant I/O interconnect allows a PCIe fanout to be concurrently repaired without loss of access to its associated I/O domains. Up to 10 PCIe fanouts are available per CPC drawer. The PCIe fanout can also be used for the ICA SR. If redundancy in coupling link connectivity is ensured, the PCIe fanout can be concurrently repaired.
Host channel adapter fanout hot-plug
The HCA fanout also is used for the InfiniBand coupling links (HCA3-O and HCA3-O LR), which provide the path for data between memory and the remote sysplex members and is connected to through 6 GBps or 5 Gbps IFB cables. If redundancy in coupling link connectivity is ensured, the HCA fanout can be concurrently repaired.
1.3.5 I/O connectivity: PCIe Generation 3
The z14 server offers new and improved I/O features and uses PCIe Gen 3 for I/O connectivity. This section briefly reviews the most relevant I/O capabilities.
The z14 uses PCIe Generation 3 to implement the following features:
•PCIe Generation 3 (Gen3) fanouts that provide 16 GBps connections to the PCIe I/O features in the PCIe I/O drawers
•PCIe Gen3 fanouts that provide 8 GBps coupling link connections through the new IBM ICA SR
1.3.6 I/O subsystem
The z14 I/O subsystem is similar to the subsystem on z13 and includes the PCIe Gen3 infrastructure. This infrastructure is designed to reduce processor usage and I/O latency, and provide increased throughput.
z14 servers offer PCIe I/O drawers that host PCIe features. I/O drawers that were used in previous IBM Z servers are not supported on z14.
PCIe I/O drawer
The PCIe I/O drawer, together with the PCIe features, offers finer granularity and capacity over previous I/O infrastructures. It can be concurrently added and removed in the field, which eases planning. Only PCIe cards (features) are supported, in any combination. Up to five PCIe I/O drawers can be installed on a z14 server.
Native PCIe and Integrated Firmware Processor
Native PCIe support was introduced with the zEDC and RoCE Express features, which are managed differently from the traditional PCIe I/O (FICON Express and OSA-Express) features. The device drivers for the native features are provided in the operating system. The diagnostic tests for the adapter layer functions of the native PCIe features are managed by LIC that is running as a resource group, which runs on the Integrated Firmware Processor (IFP).
With z14, the number of resource groups is increased from two to four to add granularity, which helps mitigate the effect of the disruptive Resource Group Microcode Change Level (MCL) installations. This firmware management enhancement contributes to the RAS of the server.
During the ordering process of the native PCIe features, features of the same type are evenly spread across the four resource groups (RG1, RG2, RG3, and RG4) for availability and serviceability reasons. Resource groups are automatically activated when these features are present in the CPC.
In addition to the zEDC and 10GbE RoCE Express features, the z14 introduces the following native PCIe I/O features:
•Coupling Express Long Reach (CE LR)
•zHyperLink Express
•25GbE and 10GbE RoCE Express2
1.3.7 I/O and special purpose features in the PCIe I/O drawer
The z14 server (new build) supports the following PCIe features that are installed in the PCIe I/O drawers:
 – FICON Express16S+ Short Wave (SX)
 – FICON Express16S+ Long Wave (LX) 10 km (6.2 miles)
 – zHyperLink Express
 – OSA-Express7S 25GbE Short Reach (SR)
 – OSA-Express6S 10GbE Long Reach (LR)
 – OSA-Express6S 10GbE Short Reach (SR)
 – OSA-Express6S GbE LX
 – OSA-Express6S GbE SX
 – OSA-Express6S 1000BASE-T
 – 25GbE RoCE Express2
 – 10GbE RoCE Express2
 – Crypto Express6S
 – Regional Crypto Enablement
When carried forward on an upgrade, the z14 server also supports the following features in the PCIe I/O drawers:
 – FICON Express 16S SX
 – FICON Express 16S LX
 – FICON Express 8S SX
 – FICON Express 8S LX
 – OSA-Express5S 10 GbE Long Reach (LR)
 – OSA-Express5S 10 GbE Short Reach (SR)
 – OSA-Express5S GbE LX
 – OSA-Express5S GbE SX
 – OSA-Express5S 1000BASE-T
 – OSA-Express 4S 1000BASE-T
 – 10GbE RoCE Express
 – Crypto Express5S
 – Regional Crypto Enablement
Although they are used for coupling connectivity, the IBM Integrated Coupling Adapter (ICA SR) and the InfiniBand coupling links HCA3-O and HCA3-O LR are other z14 supported features that are not listed here because they are attached directly to the CPC drawer.
1.3.8 Storage connectivity
z14 supports the new IBM zHyperLink Express and FICON channels for storage connectivity.
IBM zHyperLink Express
zHyperLink Express is a new short-distance mainframe attach link that is designed to increase the scalability of IBM Z transaction processing and lower I/O latency than High-Performance FICON (HPF) by for bringing data close to processing power. It is the new mainframe I/O channel link technology since FICON.
zHyperLink Express feature directly connects the z14 Central Processor Complex (CPC) to the I/O Bay of the DS8880 (R8.3). This short distance (up to 150 m) direct connection is intended to reduce I/O latency and improve storage I/O throughput.
The improved performance of zHyperLink Express allows the z14 PU to make a synchronous request for the data that is in the DS8880 cache. This feature eliminates the undispatch of the running request, the queuing delays to resume the request, and the PU cache disruption.
The IBM zHyperLink Express is a two-port feature in the PCIe I/O drawer. Up to 16 features with up to 32 zHyperLink Express ports are supported in a z14 CPC. The zHyperLink Express feature uses PCIe Gen3 technology, with x16 lanes that are bifurcated into x8 lanes for storage connectivity. It is designed to support a link data rate of 8 GigaBytes per second (GBps)16.
The point-to-point link is established by 24x fiber optic cable with Multi-fiber Termination Push-on (MTP) connectors. For more information, see “zHyperLink Express (FC 0431)” on page 173.
FICON channels
Up to 160 features with up to 320 FICON Express16S+ channels are supported on a new build z14. FICON Express 16S+ and FICON Express 16S (carry forward only) support 4, 8, or 16 Gbps. FICON Express8S features (carry forward only) support link data rates of 2, 4, or 8 Gbps.
FICON Express16S+ offers increased performance compared to FICON Express16S with its new IBM I/O ASIC that supports up to 3x the I/O start rate of previous FICON/FCP solution. Another distinction for FICON Express16S+ is that both ports of a feature must be defined as the same CHPID type (no mix of FC and FCP CHPID for the same feature).
The FICON features on z14 support the following protocols:
•FICON (CHPID type FC) and High-Performance FICON for Z (zHPF). zHPF offers improved performance for data access, which is important to online transaction processing (OLTP) applications.
•FICON channel-to-channel (CHPID type CTC).
•Fibre Channel Protocol (CHPID type FCP).
FICON also offers the following capabilities:
•Modified Indirect Data Address Word (MIDAW) facility: Provides more capacity over native FICON channels for programs that process data sets that use striping and compression, such as Db2, VSAM, partitioned data set extended (PDSE), hierarchical file system (HFS), and z/OS file system (zFS). It does so by reducing channel, director, and control unit processor usage.
•Enhanced problem determination, analysis, and manageability of the storage area network (SAN) by providing registration information to the fabric name server for FICON and FCP.
•An Extended Link Service command, Read Diagnostic Parameters (RDP) is used to obtain extra diagnostic data from the Small Form Factor Pluggable optics that are throughout the SAN fabric to improve the accuracy of identifying a failing component.
1.3.9 Network connectivity
The IBM z14 supports the following technologies for network connectivity:
•Shared Memory Communications - Direct Memory Access over Internal Shared Memory (SMC-D)
Open Systems Adapter
z14 allows any mix of the supported OSA Ethernet features that are listed in 1.3.7, “I/O and special purpose features in the PCIe I/O drawer” on page 18. OSA-Express6 features are a technology refresh of the OSA-Express5S features. Up to 48 OSA-Express6S features, with a maximum of 96 ports, are supported. The maximum number of combined OSA-Express features cannot exceed 48.
OSA-Express features provide important benefits for TCP/IP traffic by reducing latency and improving throughput for standard and jumbo frames. Data router function that is present in all OSA-Express features enables performance enhancements.
With OSA-Express7S, OSA-Express6S, OSA-Express5S, and OSA-Express4S, the functions that were performed in firmware are performed in the hardware. Extra logic in the IBM application-specific integrated circuit (ASIC) that is included with these features handle packet construction, inspection, and routing, which allows packets to flow between host memory and the LAN at line speed without firmware intervention.
On z14, an OSA feature that is configured as an integrated console controller CHPID type (OSC) supports the configuration and enablement of secure connections by using the Transport Layer Security (TLS) protocol versions 1.0, 1.1, and 1.2.
 
Important: Fulfilling the related Statement of Direction, z14 removed the support for configuring OSN CHPID types.
For more information about the OSA features, see 4.7, “Connectivity” on page 163.
HiperSockets
The HiperSockets function (also known as internal queued direct input/output or internal QDIO or iQDIO) is an integrated function of the z14 server that provides users with attachments to up to 32 high-speed virtual LANs with minimal system and network processor usage.
For communications between LPARs in the same z14 server, HiperSockets eliminate the need to use I/O subsystem features to traverse an external network. Connection to HiperSockets offers significant value in server consolidation by connecting many virtual servers.
HiperSockets can be customized to accommodate varying traffic sizes. Because the HiperSockets function does not use an external network, it can free system and network resources, eliminating attachment costs while improving availability and performance.
HiperSockets can also be used for Dynamic cross-system coupling, which is a z/OS Communications Server feature that creates trusted, internal links to other stacks within a Parallel Sysplex.
Shared Memory Communication - Remote Direct Memory Access
zEC12 GA2 was the first IBM Z server generation to support Remote Direct Memory Access over Converged Ethernet (RoCE) technology. This technology is designed to provide fast, reduced CPU consumption and memory-to-memory communications between two IBM Z CPCs.
RoCE Express features reduce CPU consumption for applications that use the TCP/IP stack (sockets communication), such as IBM WebSphere Application Server that accesses a Db2 database. It is transparent to applications and also might help to reduce network latency with memory-to-memory transfers that use SMC-R in supported z/OS releases.
IBM Z server generations continue to enhance the RoCE architecture. Although the z13 enabled ports of the 10GbE RoCE Express feature and sharing among 31 LPARs running z/OS, z14 refreshed the technology with the new 10GbE RoCE Express2, which supports 4x the number of LPAR and performance improvements. RoCE Express2 increases virtualization by supporting 63 Virtual Functions (VFs) per port for up to 126 VFs per PCHID (physical channel ID).
The 10GbE RoCE Express2 and 10GbE RoCE Express features use SR optics and support the use of a multimode fiber optic cable that ends with an LC Duplex connector. Both support point-to-point and switched connections with an enterprise-class 10 GbE switch. A maximum of eight RoCE Express features can be installed in PCIe I/O drawers of z14.
The new 25GbE RoCE Express2 also features SR optics and supports the use of 50 micron multimode fiber optic that ends with an LC duplex connector. It supports point-to point and switched connection with 25GbE capable switch (supports only 25 Gbps, no down negotiation to 10 Gbps).
Shared Memory Communications - Direct Memory Access
SMC-D enables low processor usage and low latency communications within a CPC that uses a Direct Memory Access connection over ISM. SMC-D implementation is similar to SMC-R over RoCE; SMC-D over ISM extends the benefits of SMC-R to operating system instances that are running on the same CPC without requiring physical resources (RoCE adapters, PCI bandwidth, ports, I/O slots, network resources, and 25/10 GbE switches).
Introduced with z13 GA2 and z13s, SMC-D enables high-bandwidth LPAR-to-LPAR TCP/IP traffic (sockets communication) by using the direct memory access software protocols over virtual Internal Shared Memory PCIe devices (vPCIe). SMC-D maintains the socket-API transparency aspect of SMC-R so that applications that use TCP/IP communications can benefit immediately without requiring any application software or IP topology changes.
z14 continues to support SMC-D with its lightweight design that improves throughput, latency, and CPU consumption and complements HiperSockets, OSA, or RoCE without sacrificing quality of service.
SMC-D requires an OSA or a HiperSockets connection to establish the initial TCP communications and can coexist with them. SMC-D uses a virtual PCIe adapter and is configured as a physical PCIe device. Up to 32 ISM adapters are available, each with a unique Physical Network ID per CPC.
 
Notes: SMC-D does not support coupling facilities (z/OS to z/OS only).
SMC-R and SMC-D do not currently support multiple IP subnets.
1.3.10 Coupling and Server Time Protocol connectivity
IBM z14 support for Parallel Sysplex includes the CF (running the CFCC) and coupling links.
Coupling links support
Coupling connectivity in support of Parallel Sysplex environments is provided on the z14 server by the following features:
•Internal Coupling (IC) links that are operating at memory speed.
 – HCA3-O, 12x Parallel Sysplex InfiniBand (IFB) coupling links.
 – HCA3-O, 1x Parallel Sysplex InfiniBand (IFB) coupling links.
 
Note1: z14 is the last high-end IBM Z server to support Parallel Sysplex InfiniBand (PSIFB) coupling links; z13s is the last midrange IBM Z server to support them. IBM Z enterprises should plan to migrate off from PSIFB links.
1 All statements regarding IBM plans, directions, and intent are subject to change or withdrawal without notice. Any reliance on these statements of general direction is at the relying party’s sole risk and will not create liability or obligation for IBM.
All coupling link types can be used to carry STP messages.
Integrated Coupling Adapter Short Reach
The Integrated Coupling Adapter Short Reach (ICA SR) feature (introduced with IBM z13) is a two-port fanout that is used for short distance coupling connectivity. It uses PCIe Gen3 technology, with x16 lanes that are bifurcated into x8 lanes for coupling.
The ICA SR is designed to drive distances up to 150 m and support a link data rate of 8 GBps. The ICA SR fanout takes one PCIe I/O fanout slot in the z14 CPC drawer. It is used for coupling connectivity between z14, z13, and z13s CPCs, and cannot be connected to HCA3-O or HCA3-O LR coupling fanouts. The ICA SR is compatible with another ICA SR only.
Coupling Express Long Reach
z14 introduces the new Coupling Express Long Range (CE LR), which is a two-port feature that is used for long-distance coupling connectivity and defined as coupling channel type, CL5. Unlike the 1x IFB InfiniBand coupling links, which were plugged into the CPC drawer, the CE LR link is plugged in a PCIe I/O drawer slot, which uses industry standard I/O technology.
Compared to the HCAO-3 LR feature, which has 4-port and 4-link increments, the CE LR link allows for more granularity when scaling up or completing maintenance. Link performance is similar to the InfiniBand 1x coupling link and uses identical Single Mode fiber. The CE LR link provides point-to-point coupling connectivity at distances of 10 km unrepeated and 100 km with a qualified dense wavelength division multiplexing (DWDM) device.
IFB coupling links
The following Parallel Sysplex coupling links use IFB:
•12x InfiniBand coupling links (HCA3-O) are used for local connections up to 150 meters (492 feet). The 12x IFB link (HCA3-O) connects between z14, z13, z13s, zEC12, and zBC12 servers. It cannot be connected to an ICA SR.
•1x InfiniBand coupling links (HCA3-O LR) are used for extended distance connections between z14, z13, z13s, zEC12, and zBC12 servers. The HCA3-O LR 1X InfiniBand links have a bandwidth of 5 Gbps, use single mode fiber, and cannot be connected to a CE LR. The HCA3-O LR link provides point-to-point coupling connectivity at distances of 10 km (6.21 miles) unrepeated and 100 km (62.1 miles) with a qualified dense wavelength division multiplexing (DWDM) device.
CFCC Level 23
CFCC level 23 is delivered on the z14 with driver level 36. CFCC Level 23 introduces the following enhancements:
•Asynchronous cross-invalidate (XI) of CF cache structures. Requires PTF support for z/OS and explicit data manager support (IBM DB2® V12 with PTFs).
•Coupling Facility hang detect enhancements- provides a significant reduction in failure scope and client disruption (CF-level to structure-level), with no loss of FFDC collection capability.
•Coupling Facility ECR granular latching
z14 servers with CFCC Level 23 require z/OS V1R13 or later, and z/VM V6R4 or later for virtual guest coupling.
CFCC Level 22
CFCC level 22 is delivered on the z14 with driver level 32. CFCC Level 22 introduces the following enhancements:
•Support for up to 170 ICF processors per z14. The maximum number of logical processors in a Coupling Facility Partition remains 16.
•Support for new Coupling Express LR links.
•CF Processor Scalability: CF work management and dispatching changes for z14 allow improved efficiency and scalability for coupling facility images.
•CF List Notification Enhancements: Significant enhancements were made to CF notifications that inform users about the status of shared objects within a CF.
•Coupling Link Constraint Relief: z14 provides more physical coupling link connectivity compared to z13.
•CF Encryption: z/OS 2.3 supports end-to-end encryption for CF data in flight and data at rest in CF structures (as a part of the Pervasive Encryption solution). Host-based CPACF encryption is used for high performance and low latency.
z14 servers with CFCC Level 22 require z/OS V1R13 or later, and z/VM V6R3 or later for virtual guest coupling.
Although the CF LPARs are running on different server generations, different levels of CFCC can coexist in the same sysplex, which enables upgrade from one CFCC level to the next. CF LPARs that are running on the same server share a single CFCC level.
A CF running on a z14 server (CFCC level 22) can coexist in a sysplex with CFCC levels 21 and 19. For more information about determining the CF LPAR size by using the CFSizer tool, see the System z Coupling Facility Structure Sizer Tool page of the IBM Systems support website.
Server Time Protocol facility
Time synchronization for Parallel Sysplex Server Time Protocol (STP) is designed to ensure events that occur in different servers are properly sequenced in time. STP is designed for servers that are configured in a Parallel Sysplex or a basic sysplex (without a CF), and servers that are not in a sysplex but need time synchronization.
STP is a server-wide facility that is implemented in the LIC, which presents a single view of time to PR/SM. Any IBM Z CPC (including CPCs that are running as stand-alone CFs) can be enabled for STP by installing the STP feature.
STP uses a message-based protocol in which timekeeping information is passed over externally defined coupling links between servers. The STP design introduced a concept called Coordinated Timing Network (CTN), which is a collection of servers and CFs that are time-synchronized to a time value called Coordinated Server Time (CST).
Network Time Protocol (NTP) client support is available to the STP code on the z14, z13, z13s, zEC12, and zBC12 servers. By using this function, these servers can be configured to use an NTP server as an External Time Source (ETS). This implementation fulfills the need for a single time source across the heterogeneous platforms in the enterprise, including IBM Z servers and others systems that are running Linux, UNIX, and Microsoft Windows operating systems.
The time accuracy of an STP-only CTN can be improved by using an NTP server with the pulse per second (PPS) output signal as ETS. This type of ETS is available from various vendors that offer network timing solutions.
HMC can be configured as an NTP client or an NTP server. To ensure secure connectivity, HMC NTP broadband authentication can be enabled on z14, z13, and zEC12 servers.
STP enhancements in z14
With HMC 2.14.1, Coordinated Timing Network split or merge are supported. CTN split defines a system-assisted method for dynamic splitting of a CTN, while CTN merge automates the time synchronization process and the STP role assignment in the merged CTN. IBM z14 introduces another synchronization level for STP: STP stratum level 4. With the extra stratum level, STP can synchronize systems up to three levels away from the CTS.
Older systems allowed synchronization up to only level 3, or up to two levels from the CTS. This extra stratum level is not intended for long-term use; rather, it is intended for short-term use during configuration changes for large timing networks to avoid some of the cost and complexity that is caused by being constrained to a three-level STP stratum configuration.
z14 also introduces a new Graphical User Display for the STP network and configuration. The new user interface was revamped for a quick, intuitive view and management of the various pieces of the CTN, including the status of the components of the timing network. The new level of HMC allows the management of older systems by using the same new interface.
 
Attention: As with its predecessor z13, a z14 server cannot be connected to a Sysplex Timer and cannot be a member in a Mixed CTN. An STP-only CTN is required for the z14 and z13 servers.
If a current configuration consists of a Mixed CTN or a Sysplex Timer (M/T 9037), the configuration must be changed to an STP-only CTN before z14 integration. The z14 server can coexist only with IBM Z CPCs that do not have the external time reference (ETR) port capability.
Enhanced Console Assisted Recovery
Console Assisted Recovery (CAR) support is designed to help a Backup Time Server (BTS) determine whether the Primary Time Server (PTS) is still available if coupling traffic ceases. The CAR process is started by the BTS if communication fails between the primary and backup time servers. The BTS queries the state of the PTS/CTS SE by using the SE and HMC of the BTS. If the PTS is down, the BTS starts the takeover process.
With the new Enhanced Console Assisted Recovery (ECAR), the process of BTS takeover is faster. When the PTS encounters a checkstop condition, the CPC informs the SE and HMC of the condition. The PTS SE recognizes the pending checkstop condition and an ECAR request is sent directly from the HMC to the BTS SE to start the takeover.
The new ECAR support is faster than the original support almost no delay occurs between the system checkstop and the start of CAR processing. ECAR is available on z14, z13 GA2, and z13s servers only. In a mixed environment with previous generation machines, ECAR supported servers are defined as the PTS and CTS.
1.3.11 Cryptography
A strong synergy exists between cryptography and security. Cryptography provides the primitives to support security functions. Similarly, security functions help to ensure authorized use of key material and cryptographic functions.
Cryptography on IBM Z is built on the platform with integrity. IBM Z platform offers hardware-based cryptography features that are used by the following environments and functions:
•Java
•Db2/IMS encryption tool
•Db2 built in encryption z/OS Communication Server
•IPsec/IKE/AT-TLS
•z/OS System SSL
•z/OS
•z/OS Encryption Facility
•Linux on Z
CP Assist for Cryptographic Functions
Supporting clear and protected key encryption, CP Assist for Cryptographic Function (CPACF) offers the full complement of the Advanced Encryption Standard (AES) algorithm and Secure Hash Algorithm (SHA) with the Data Encryption Standard (DES) algorithm. Support for CPACF is available through a group of instructions that are known as the Message-Security Assist (MSA).
z/OS Integrated Cryptographic Service Facility (ICSF) callable services and the z90crypt device driver that is running on Linux on Z also start CPACF functions. ICSF is a base element of z/OS. It uses the available cryptographic functions, CPACF, or PCIe cryptographic features to balance the workload and help address the bandwidth requirements of your applications.
With z14, CPACF is enhanced to support pervasive encryption to provide faster encryption and decryption than previous servers. For every Processor Unit that is defined as a CP or an IFL, it offers the following enhancements over z13:
•Reduced overhead on short data (hashing and encryption)
•Up to 4x throughput for AES
•Special instructions for elliptic curve cryptography (ECC)/RSA
•New hashing algorithms (for example, SHA-3)
•Support for authenticated encryption (combined encryption and hashing; for example, AES-GCM)
•True random number generator (for example, for session keys)
The z13 CPACF provides (supported by z14 also) the following features:
•For data privacy and confidentially: DES, Triple Data Encryption Standard (TDES), and AES for 128-bit, 192-bit, and 256-bit keys.
•For data integrity: Secure Hash Algorithm-1 (SHA-1) 160-bit, and SHA-2 for 224-, 256-, 384-, and 512-bit support. SHA-1 and SHA-2 are shipped enabled on all z14s and do not require the no-charge enablement feature.
•For key generation: Pseudo Random Number Generation (PRNG), Random Number Generation Long (RNGL) (1 - 8192 bytes), and Random Number Generation Long (RNG) with up to 4096-bit key RSA support for message authentication.
CPACF must be explicitly enabled by using a no-charge enablement feature (FC 3863). This requirement excludes the SHAs, which are enabled by default with each server.
The enhancements to CPACF are exclusive to the IBM Z servers and are supported by z/OS, z/VM, z/VSE, z/TPF, and Linux on Z.
Crypto Express6S
Crypto Express6S represents the newest generation of cryptographic features. Cryptographic performance improvements with new Crypto Express6S (FC #0893) allow more data to be securely transferred across the internet. Crypto Express6S is designed to complement the cryptographic capabilities of the CPACF. It is an optional feature of the z14 server generation.
The Crypto Express6S feature is designed to provide granularity for increased flexibility with one PCIe adapter per feature. Although installed in the PCIe I/O drawer, Crypto Express6S features do not perform I/O operations. That is, no data is moved between the CPC and any externally attached devices. For availability reasons, a minimum of two features is required.
z14 servers allow sharing of a cryptographic coprocessor across 85 domains (the maximum number of LPARs on the system for z14 is 85).
The Crypto Express6S is a state-of-the-art, tamper-sensing, and tamper-responding programmable cryptographic feature that provides a secure cryptographic environment. Each adapter contains a tamper-resistant hardware security module (HSM). The HSM can be configured as a Secure IBM CCA coprocessor, as a Secure IBM Enterprise PKCS #11 (EP11) coprocessor, or as an accelerator. Consider the following points:
•A Secure IBM CCA coprocessor is for secure key encrypted transactions that use CCA callable services (default).
•A Secure IBM Enterprise PKCS #11 (EP11) coprocessor implements an industry standardized set of services that adhere to the PKCS #11 specification v2.20 and more recent amendments. This new cryptographic coprocessor mode introduced the PKCS #11 secure key function.
•An accelerator for public key and private key cryptographic operations is used with Secure Sockets Layer/Transport Layer Security (SSL/TLS) acceleration.
The Crypto Express6S is designed to meet the following cryptographic standards, among others:
•FIPS 140-2 Level 417
•Common Criteria EP11 EAL4
•ANSI 9.97
•Payment Card Industry (PCI) HSM
•German Banking Industry Commission (GBIC), (formerly DK, Deutsche Kreditwirtschaft)
Federal Information Processing Standard (FIPS) 140-2 certification is supported only when Crypto Express6S is configured as a CCA or an EP11 coprocessor.
Crypto Express6S supports several ciphers and standards that are described next. For more information about cryptographic algorithms and standards, see Chapter 6, “Cryptographic features” on page 207.
Regional Crypto Enablement
Starting with z13 GA2, IBM enabled geo-specific Cryptographic Support that is supplied by IBM approved vendors. China is the first geography to use this support to meet the cryptography requirements of Chinese clients that are required to comply with the People’s Bank of China Financial IC Card Specifications (PBOC 3.0) for payment card processing.
Regional Crypto Enablement (RCE) is a framework that is used to enable the integration of IBM certified third-party cryptographic hardware for regional or industry encryption requirements. It also supports the use of cryptography algorithms and equipment from selected providers with IBM Z in specific countries. Support for the use of international algorithms (AES, DES, RSA, and ECC) with regional crypto devices (supporting regional algorithms, such as SMx) is added to the ICSF PKCS#11 services.
When ordered, the RCE support reserves the I/O slots for the IBM approved vendor-supplied cryptographic cards. Clients must contact the IBM approved vendor directly for purchasing information.
Trusted Key Entry workstation
The Trusted Key Entry (TKE) feature is an integrated solution that is composed of workstation firmware, hardware, and software to manage cryptographic keys in a secure environment. The TKE is network-connected or isolated, in which case smart cards are used.
The TKE workstation offers a security-rich solution for basic local and remote key management. It provides authorized personnel with a method for key identification, exchange, separation, update, and backup, and a secure hardware-based key loading mechanism for operational and master keys. TKE also provides secure management of host cryptographic module and host capabilities.
Support for an optional smart card reader that is attached to the TKE workstation allows the use of smart cards that contain an embedded microprocessor and associated memory for data storage. Access to and the use of confidential data on the smart cards are protected by a user-defined personal identification number (PIN).
TKE workstation and the most recent TKE 9.1 LIC are optional features on the z14. TKE workstation is offered in two types: TKE Tower (FC #0086) and TKE Rack Mount (FC #0085). TKE 9.x18 requires the crypto adapter FC 4768. You can use an older TKE version to collect data from previous generations of cryptographic modules and apply the data to Crypto Express6S coprocessors.
TKE 9.x is required if you choose to use the TKE to manage a Crypto Express6S. TKE 9.1 is also required to manage the new CCA mode PCI-HSM settings that are available on the Crypto Express6S. A TKE is required to manage any Crypto Express feature that is running in IBM Enterprise PKCS #11 (EP11) mode. If EP11 is to be defined, smart cards that are used require FIPS certification.
For more information about the cryptographic features, see Chapter 6, “Cryptographic features” on page 207.
For more information about the most current ICSF updates that are available, see the Web Deliverables download website.
1.3.12 zEDC Express
zEDC Express, an optional feature that is available on z14, z13, z13s, zEC12, and zBC12 servers, provides hardware-based acceleration for data compression and decompression with lower CPU consumption. The IBM zEnterprise Data Compression (zEDC) acceleration capability in z/OS and the zEDC Express feature are designed to help improve cross-platform data exchange, reduce CPU consumption, and save disk space.
Although installed in the PCIe I/O drawer, zEDC Express features do not perform I/O operations. That is, no data is moved between the CPC and externally attached devices. One PCIe adapter or compression coprocessor is available per feature. The zEDC Express feature can be shared by up to 15 LPARs. Up to 16 features can be installed on z14.
For more information about the IBM System z Batch Network Analyzer (zBNA) tool, which reports on potential zEDC usage for QSAM/BSAM data sets, see the IBM System z Batch Network Analyzer (zBNA) Tool page.
1.4 Reliability, availability, and serviceability
The z14 RAS strategy employs a building block approach, which is developed to meet the client’s stringent requirements for achieving continuous reliable operation. Those building blocks are error prevention, error detection, recovery, problem determination, service structure, change management, measurement, and analysis.
The initial focus is on preventing failures from occurring. This goal is accomplished by using Hi-Rel (highest reliability) components that use screening, sorting, burn-in, and run-in, and by taking advantage of technology integration.
For LIC and hardware design, failures are reduced through rigorous design rules; design walk-through; peer reviews; element, subsystem, and system simulation; and extensive engineering and manufacturing testing.
The RAS strategy is focused on a recovery design to mask errors and make them transparent to client operations. An extensive hardware recovery design is implemented to detect and correct memory array faults. In cases where transparency cannot be achieved, you can restart the server with the maximum capacity possible.
The IBM z14™ processor builds upon the RAS of the z13 family with the following RAS improvements:
•The level 3 cache added powerful symbol ECC, which makes it resistant to more failures (the z13 hardened the level 4 cache and the main memory was hardened with RAIM and ECC before that).
•The main memory added preemptive DRAM marking to isolate and recover failures faster.
•Small array error handling is improved in the processor cores.
•Error thresholding was added to the processor core to isolate “sick but not dead” failure scenarios.
•The number of Resource Groups for supporting native PCIe features increased to four from two to reduce the effect of firmware updates and failures.
•OSA-Express6S adds TCP checksum on large send offload.
1.5 Hardware Management Consoles and Support Elements
The HMCs and SEs are appliances that together provide platform management for IBM Z. The HMC is a workstation that is designed to provide a single point of control for managing local or remote hardware elements.
HMC is offered as a Tower (FC #0082) and a Rack Mount (FC #0083) feature. Rack Mount HMC can be placed in a customer-supplied 19-inch rack and occupies 1U rack space. z14 includes driver level 32 and HMC application Version 2.14.0.
1.6 Operating systems
The IBM z14 server is supported by a large set of software products and programs, including independent software vendor (ISV) applications. (This section lists only the supported operating systems.) Use of various features might require the latest releases. For more information, see Chapter 7, “Operating system support” on page 243.
1.6.1 Supported operating systems
The following operating systems with required maintenance applied are supported for z14 servers:
•z/OS:
 – Version 2 Release 3
 – Version 2 Release 2
 – Version 2 Release 1
 – Version 1 Release 13 (compatibility support only, with extended support agreement)
•z/VM:
 – Version 7 Release 1
 – Version 6 Release 4
•z/VSE
 – Version 6 Release 2
 – Version 6 Release 1
 – Version 5 Release 2
 –  
•z/TPF Version 1 Release 1
•Linux on IBM Z distributions19:
 – SUSE Linux Enterprise Server (SLES): SLES 12 and SLES 11.
 – Red Hat Enterprise Linux (RHEL): RHEL 7 and RHEL 6.
 – Ubuntu 16.04 LTS (or higher)
•KVM for IBM Z
The KVM hypervisor is supported with the following minimum Linux distributions:
 – SLES 12 SP2 with service.
 – RHEL 7.5 with kernel-alt package (kernel 4.14).
 – Ubuntu 16.04 LTS with service and Ubuntu 18.04 LTS with service.
For more information about supported Linux on Z distribution levels, see the Tested platforms for Linux page of the IBM Z website.
For more information about features and functions that are supported on z14 by operating system, see Chapter 7, “Operating system support” on page 243.
z/VM support
z/VM 7.1 (Available as of Sept. 2018) increases the level of engagement with the z/VM user community. z/VM 7.1 includes the following new features:
•Single System Image and Live Guest Relocation included in the base (no extra charge).
•Enhances the dump process to reduce the time that is required to create and process dumps.
•Upgrades to a new Architecture Level Set (requires an IBM zEnterprise EC12 or BC12, or later).
•Provides the base for more functionality to be delivered as service after general availability.
•Enhances the dynamic configuration capabilities of a running z/VM system with Dynamic Memory Downgrade* support. For more information, see this web page.
•Includes SPE20s shipped for z/VM 6.4, including Virtual Switch Enhanced Load Balancing, DS8K z-Thin Provisioning, and Encrypted Paging.
To support new functionality that was announced October 2018, z/VM requires fixes for the following APARs:
•PI99085
•VM66130
•VM65598
•VM66179
•VM66180
With the PTF for APAR VM65942, z/VM V6.4 provides support for z14.
z/VM 6.4 has support for the following functions on z14:
•z/Architecture support
•ESA/390-compatibility mode for guests
•New hardware facilities:
 – Miscellaneous-Instruction-Extensions Facility 2
 – Vector Enhancements Facility 1
 – Vector Packed Decimal Facility
 – Message-Security-Assist Extension 6, 7, and 8
•Up to 2 TB per z/VM LPAR
•Improved memory management support
•Support for IBM DPM, enabling z/VM 6.4 to be configured through new DPM interface rather than traditional PR/SM
•Guest support to use:
 – Large Page (1 MB pages)
 – Single Instruction Multiple Data (SIMD)
•Encrypted paging support
•Crypto Clear Key ECC operations
•Dynamic I/O support
•Guest support for:
 – RoCE Express2 and RoCE Express - SMC-R
 – SMC-D
 – Instruction Execution Protection Facility
 – Pause-less garbage collection
For more information about the features and functions that are supported on z14 by operating system, see Chapter 7, “Operating system support” on page 243.
z/OS support
z/OS uses many of the following new functions and features of z14 (depending on version and release; PTFs might be required to support new functions):
•Up to 170 processors per LPAR or up to 128 physical processors per LPAR in SMT mode (SMT for zIIP)
•Up to 16 TB of real memory per LPAR (dependent on z/OS version)
•Two-way simultaneous multithreading (SMT) optimization and support of SAPs (SAP SMT enabled by default) in addition to zIIP engines
•XL C/C++ ARCH(12) and TUNE(12)complier options
•Use of faster CPACF
Pervasive Encryption:
 – Coupling Facility Encryption
 – Dataset and network encryption
•HiperDispatch Enhancements
•z14 Hardware Instrumentation Services (HIS)
•Entropy-Encoding Compression Enhancements
•Guarded Storage Facility (GSF)
•Instruction Execution Protection (IEP)
•IBM Virtual Flash Memory (VFM)
•Improved memory management in Real Storage Manager (RSM)
•CF use of VFM
 – CFCC Level 23 and 22
•Coupling Express Long Reach (CE LR) CHPID type CL5
•zHyperLink Express
•FICON Express16S+: OSA Express7S and 6S
•RoCE-Express2 (25GbE and 10GbE)
•Cryptography:
 – Crypto Express6S:
 • Next Generation Coprocessor support
 • Support for Coprocessor in PCI-HSM Compliance Mode
 • Designed for up to 85 domains
 – TKE 9.x workstation
For more information about the features and functions that are supported on z14 by operating system, see Chapter 7, “Operating system support” on page 243.
1.6.2 IBM compilers
The following IBM compilers for Z servers can use z14 servers:
•Enterprise COBOL for z/OS
•Enterprise PL/I for z/OS
•Automatic Binary Optimizer
•z/OS XL C/C++
•XL C/C++ for Linux on Z
The compilers increase the return on your investment in IBM Z hardware by maximizing application performance by using the compilers’ advanced optimization technology for z/Architecture. Through their support of web services, XML, and Java, they allow for the modernization of assets in web-based applications. They also support the latest IBM middleware products (CICS, Db2, and IMS), which allows applications to use their latest capabilities.
To fully use the capabilities of z14 servers, you must compile it by using the minimum level of each compiler. To obtain the best performance, you must specify an architecture level of 12 by using the ARCH(12) option.
1 For more information, see the Technology in Action page of the IBM website.
2 This data accounts for 80% of all data that is generated today and is expected to grow to over 93% by 2020.
3 FINFET is the industry solution; SOI is the IBM solution for SER.
4 Simultaneous multithreading is two threads per core.
5 Observed performance increases vary depending on the workload types.
6 Observed performance increases vary depending on the workload types.
7 Workloads qualified to run on IBM System z® Application Assist Processor (zAAP) engines that were available on IBM Z server generations z990 - zEC12, run on a zIIP processor, which reduces the complexity of the IBM z/Architecture®.
8 IBM HSBN is a cloud service plan that is available on IBM Bluemix® for Blockchain.
9 For more information, see the IBM to Deliver Docker Enterprise Edition for Linux on IBM Systems topic of the BIM News releases website.
10 I/O drawers were introduced with the IBM z10™ BC.
12 For more information, see Quantifying the Business Value of IBM Z.
13 IBM zEnterprise Application Assist Processors (zAAPs) are not available on z14 servers. The zAAP workload is run on zIIPs.
14 Feature code (FC) 3863 must be ordered to enable CPACF. This feature code is available for no extra fee.
15 Based on preliminary internal IBM lab measurements on a stand-alone, dedicated system in a controlled environment and compared to the z13. Results might vary.
16 The link data rates do not represent the performance of the links. The actual performance is dependent upon many factors, including latency through the adapters, cable lengths, and the type of workload.
17 Federal Information Processing Standard (FIPS) 140-2 Security Requirements for Cryptographic Modules
18 TKE 9.0 LIC or TKE 9.1 LIC have the same hardware requirements. TKE 9.0 LIC can be upgraded to 9.1 LIC.
19 Customers should monitor for new distribution releases supported.
20 Small Program Enhancements, part of the continuous delivery model, see http://www.vm.ibm.com/newfunction/
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset