Chapter 1

Terascale to Petascale: The Past 17 Years in High Performance Computing

Jeffrey S. Vetter

Oak Ridge National Laboratory and Georgia Institute of Technology

Bibliography

High Performance Computing (HPC) is used to solve a number of complex questions in computational and data-intensive sciences. These questions include the simulation and modeling of physical phenomena, such as climate change, energy production, drug design, global security, and materials design; the analysis of large data sets, such as those in genome sequencing, astronomical observation, and cybersecurity; and, the intricate design of engineered products, such as airplanes and automobiles.

It is clear and well-documented that HPC is used to generate insight that would not otherwise be possible. Simulations can augment or replace expensive, hazardous, or impossible experiments. Furthermore, in the realm of simulation, HPC has the potential to suggest new experiments that escape the parameters of observability.

Although much of the excitement about HPC focuses on the largest architectures and on specific benchmarks, such as TOP500, there is a much deeper and broader commitment from the international scientific and engineering community than is first apparent. In fact, it is easy to lose track of history in terms of the broad uses of HPC and the communities that design, deploy, and operate HPC systems and facilities. Many of these sponsors and organizations have spent decades developing scientific simulation methods and software which serves as the foundation of HPC today. This community has worked closely with countless vendors to foster the sustained development and deployment of HPC systems internationally.

In this second volume of Contemporary High Performance Computing [4], we continue to document international HPC ecosystems including the sponsors and sites that host them. We have selected contributions from international HPC sites, which represent a combination of sites, systems, vendors, applications, and sponsors. Rather than focus on simply the architecture or the application, we focus on HPC ecosystems that have made this dramatic progress possible. Though the very word ecosystem can be a broad, all-encompassing term, it aptly describes HPC. That is, HPC is far more than one sponsor, one site, one application, one software system, or one architecture. Indeed, it is a community of interacting entities in this environment that sustains the community over time. Contributors were asked to address the following topics in their chapters:

1.  Sponsor and site history

2.  Highlights of applications, workloads, and benchmarks

3.  Systems overview

4.  Hardware architecture

5.  System software

6.  Programming systems

7.  Storage, visualization, and analytics

8.  Data center/facility

9.  Site HPC statistics

By any measure, the past 17 years have witnessed dramatic increases in both the use and scale of HPC. Thinking back, it was 1997, only 17 years ago, when the ASCI Red system at Sandia National Laboratories in Albuquerque, New Mexico, broke the 1 Terascale barrier on TOP500 Linpack using 7,264 Pentium P6 processors over a proprietary interconnection network. In 2013, China’s Tihane-2 system accelerated with Xeon Phi surpassed 33.8 Petaflops on the same benchmark: an increase of about 33,000 times in 17 years! It is impressive, indeed, when considering the well-known fact that the performance of commodity microprocessors has slowed due to power and thermal constraints [1, 2, 3].

Although much of the focus is on the #1 system on TOP500, and its architecture, it is important to promote the fact that there are dozens, in fact, hundreds of systems around the world in daily use for solving HPC problems. As Table 1.1 illustrates, we have seven chapters from contributing authors around the world describing their ecosystem. The architectures of these systems span the range from a fully customized processor and interconnection network to a complete commodity solution. A number of new sites have installed the new Cray XC30. Two sites have commodity clusters, where one is accelerated by Xeon Phi. Finally, Japan’s K system is a specialized version of the SPARC64 processor, and the Tofu interconnection network.

In contrast to the hardware diversity we see in these systems, they share a tremendous amount of software as listed in Table 1.2. In fact, much of this software is open-source software that organizations can download, port, and install on any system. In fact, although HPC systems share many hardware components with servers in enterprise and data centers, the HPC software stack is a distinguishing component of HPC. Generally speaking, the HPC software stack has multiple levels: system software, development environments, system management software, and scientific data management and visualization systems. Nearest to the hardware, system software typically includes operating systems, runtime systems, and low-level I/O software, like file systems. Next, the category of programming systems is a broad area that facilitates design and development of applications. In our framework, it includes programming models, compilers, scientific frameworks and libraries, and correctness and performance tools. Then, system management software coordinates, schedules, and monitors the system and the applications running on that system. Finally, scientific data management and visualization software provides users with domain specific tools for generating, managing, and exploring data for their science. This data may include output from simulations per se or empirically measured data from sensors in the real world, which can also be used to validate and calibrate simulation software.

TABLE 1.1: Significant systems in HPC.

TABLE 1.2: HPC software summary.

Nevertheless, over the past 17 years, HPC software has had to adapt and respond to several challenges. First, the concurrency in applications and systems has grown over three orders of magnitude. The primary programming model, MPI, has had to grow and change to allow this scale. Second, the increase in concurrency has per core driven lower the memory and I/O capacity, and the memory, I/O, and interconnect bandwidth. Third, in the last five years, heterogeneity and architectural diversity have placed a new emphasis on application and software portability across programming environments.

Bibliography

[1]  Jack Dongarra, Pete Beckman, Terry Moore, Patrick Aerts, Giovanni Aloisio, Jean-Claude Andre, David Barkai, Jean-Yves Berthou, Taisuke Boku, Bertrand Braunschweig, Franck Cappello, Barbara Chapman, Xuebin Chi, Alok Choudhary, Sudip Dosanjh, Thom Dunning, Sandro Fiore, Al Geist, Bill Gropp, Robert Harrison, Mark Hereld, Michael Heroux, Adolfy Hoisie, Koh Hotta, Zhong Jin, Yutaka Ishikawa, Fred Johnson, Sanjay Kale, Richard Kenway, David Keyes, Bill Kramer, Jesus Labarta, Alain Lichnewsky, Thomas Lippert, Bob Lucas, Barney Maccabe, Satoshi Matsuoka, Paul Messina, Peter Michielse, Bernd Mohr, Matthias S. Mueller, Wolfgang E. Nagel, Hiroshi Nakashima, Michael E Papka, Dan Reed, Mitsuhisa Sato, Ed Seidel, John Shalf, David Skinner, Marc Snir, Thomas Sterling, Rick Stevens, Fred Streitz, Bob Sugar, Shinji Sumimoto, William Tang, John Taylor, Rajeev Thakur, Anne Trefethen, Mateo Valero, Aad van der Steen, Jeffrey Vetter, Peg Williams, Robert Wisniewski, and Kathy Yelick. The international exascale software project roadmap. International Journal of High Performance Computing Applications, 25(1):3–60, 2011.

[2]  Peter Kogge, Keren Bergman, Shekhar Borkar, Dan Campbell, William Carlson, William Dally, Monty Denneau, Paul Franzon, William Harrod, Kerry Hill, Jon Hiller, Sherman Karp, Stephen Keckler, Dean Klein, Robert Lucas, Mark Richards, Al Scarpelli, Steven Scott, Allan Snavely, Thomas Sterling, R. Stanley Williams, and Katherine Yelick. Exascale computing study: Technology challenges in achieving exascale systems. Technical report, DARPA Information Processing Techniques Office, 2008.

[3]  Horst Simon, Thomas Zacharia, and Rick Stevens. Modeling and simulation at the exascale for energy and the environment. Technical report, US Department of Energy, Office of Science, 2007.

[4]  Jeffrey S. Vetter, editor. Contemporary High Performance Computing: From Petascale Toward Exascale, volume 1 of CRC Computational Science Series. Taylor & Francis, Boca Raton, 1 edition, 2013.