Supporting Portable, Reproducible Computational Science

Heart-broad help for, and R&D about, containers assists researchers compute with simplicity at TACC and elsewhere.

Researchers who use supercomputers for science typically never restrict by themselves to a single program. They transfer their assignments to whatever means are accessible, normally working with lots of different programs concurrently, in their lab, on their campus cluster, at highly developed computing centers like TACC, and in the cloud.

It is not a lack of loyalty, it’s just a fact of exploration existence — chances are shifting and hungry scientists uncover what they want to get their exploration achieved.

The programs researchers use are not automatically the same however. They may possibly include different hardware with different architectures, and different compilers or libraries.

This opportunistic computing paradigm makes a lot of extra function for computational scientists and program administrators – adapting outdated codes to function on new programs or putting in software package offers numerous situations. As running programs evolve, supporting code that was created on deprecated environments turns into a reproducibility problem.

In new yrs, a new remedy has emerged. Generically identified as “containers,” it involves a kind of isolation, wherever a researcher’s code is packaged jointly with all the software package dependencies in this sort of a way that it can run at lots of internet sites without the need of requiring recompilations. By incorporating an application’s lots of dependencies into self-sustainable pictures, containers keep away from a lot of problems.

Popularized by Docker in 2013, containers have been promptly acknowledged in each the commercial and academic scientific computing arenas. TACC was an early adopter and commenced enabling containerized science in 2016 – very first as a result of Docker, and far more recently as a result of Singularity, which was launched in 2015 by a staff of researchers at Lawrence Berkeley Countrywide Laboratory, and is particularly effectively-suited to large-efficiency computer systems, which include tens of thousands of tightly connected processors.

Containerized Science

Amid the people of containers on TACC programs are Thomas Hughes — professor of Aerospace Engineering and Engineering Mechanics at The College of Texas at Austin and a member of the Countrywide Academies of Science and Engineering — and David Kamensky — a former member of Hughes’ staff, now an assistant professor at the College of California, San Diego. The pair use containers to produce predictive designs of coronary artery move and study turbulence.

“The motive we started working with containers was to run the numerical PDE [partial differential equation] software package FEniCS on Stampede2,” explained Kamensky. “FEniCS is a difficult software package with lots of dependencies, and it can be tough to set up.”

When they desired to conduct an isogeometric investigation on leading of FEniCS, they transformed a Docker image preserved by the FEniCS undertaking staff into a Singularity image and ended up working with far more than one,000 node hrs on Stampede2.

Collaborating with John Evans at the College of Colorado, Boulder (CU Boulder) on a turbulence modeling study, they have been equipped to simply swap from Stampede2 to CU Boulder’s cyberinfrastructure mainly because of containerization.

“I never see it as realistic for supercomputer centers to maintain and debug all the different software package to fulfill each scientist’s desires,” Kamensky explained. “With Singularity, they only want to maintain a single piece of software package.”

Sharon Glotzer, a professor of chemical engineering at the College of Michigan and a member of each the Countrywide Academy of Sciences and the Countrywide Academy of Engineering, also takes advantage of Singularity on TACC and quite a few other centers to study how the developing blocks of make a difference transition from fluid to stable to improved understand how to style and design new elements.

In specific, her team takes advantage of molecular simulations to study the assembly behavior of substantial quantities of really hard particles into designs working with HOOMD-Blue, a basic-purpose particle simulation toolkit.

“We make use of compute means on XSEDE — together with Stampede2, Comet, and Bridges — Summit at the Oak Ridge Leadership Computing Facility, and area clusters,” explained Joshua Anderson, a exploration space expert who builds and maintain the container pictures for Glotzer’s team. “Singularity containers make it possible for us to use the same software package natural environment across all of these programs so that we can simply transition our workflows amongst them.”

Transitioning amongst programs is not constantly trivial. Going workflows that use Information Passing Interfaces (MPI) to harness the parallel computing electrical power of supercomputers is still the most important problem researchers deal with for working with containers on different clusters.

“Each requires its personal appropriate MPI and user-space driver stack within the container,” Anderson explained. To handle this, Anderson builds specialized pictures for each and every program primarily based on the same container recipe. In 2019, their staff utilized far more than five,three hundred node hrs on Stampede2 and lots of far more on other programs.

“Each requires its personal appropriate MPI and user-space driver stack within the container,” Anderson explained. To handle this, Anderson builds specialized pictures for each and every program primarily based on the same container recipe. In 2019, their staff utilized far more than five,three hundred node hrs on Stampede2 and innumerable far more on other programs.

Michael Gurnis, professor of Geophysics and director of the Seismological Laboratory at Caltech, takes advantage of containers on Stampede2 to produce computational designs of subduction and mantle move, conduct substantial parallel computation of grid lookups, and take a look at the parameter space with very first order impression on the evolution of the Earth working with the geodynamic code, Underworld2.

Configuring Underworld2 as a docker image authorized Gurnis’s staff to circumvent the installation of dependent offers, configure the natural environment to their desires, and simply run the code on Stampede2.

Ian Wang, a Exploration Affiliate in the HPC Performance & Architectures Group at TACC, labored with the developer of the Underworld software package to containerize the device. “I consider this is the very first application that runs at incredibly substantial scale in just Singularity containers working with MPI,” Wang explained. “The people and the developers in fact helped me discover a bug in Singularity that only seems at substantial scale MPI runs.”

Gurnis’s staff has utilized eleven,000 node hrs so much in containers and expects to continue on. “With the pulled image and Singularity, we can circumvent the aggravating installation of related offers and configuration of natural environment. Containerization helps make it straightforward to set up and run a substantial code on Stampede2,” explained Yida Li, a graduate scholar in Gurnis’ team.

Containers Enable Community Computing Endeavours

Biology and bioinformatics are two of the leading communities that have adopted containerization. The disciplines are relative newcomers to the HPC world, and the enhancement of new codes and instruments in the discipline has been quick and furious. This has led to some problems.

“A new study discovered that, in bioinformatics, of the software package printed in past ten yrs, 50 percent couldn’t be mounted,” explained Greg Zynda, a bioinformatician and member of the Daily life Sciences Computing team at TACC. “They have been designed for legacy running programs and can’t be run on today’s supercomputers. We’re hoping to address that problem working with containers.”

Zynda has led an work at TACC to make 16,000 bio-containers accessible on TACC’s supercomputers, getting rid of the want for existence science researchers to bundle and maintain each and every and each piece of software package in the discipline.

Two of TACC’s major collaborative software package solutions assignments — the DARPA Synergistic Discovery and Style (SD2E) undertaking and Cyverse — also leverage containers thoroughly.

SD2E takes advantage of automation and equipment understanding to accelerate discovery in areas wherever the fundamental product is not effectively understood. The undertaking provides jointly exploration labs and organizations from about the U.S. with complementary capabilities, and streamlines their interactions working with cyberinfrastructure at TACC, together with: a centralized facts repository and facts catalog for sharing, provenance, and discovery the Tapis APIs to help automated, party-pushed facts investigation working with each cloud and HPC back again-finish hardware and a developer ecosystem of command-line tooling, variation command, and continual integration.

“The platform is particularly highly effective and adaptable, and pretty much each component takes advantage of Docker containers as a core developing block,” explained John Fonner, who manages Emerging Systems in TACC’s Daily life Sciences Computing team. “The investigation instruments, persistent solutions, and even the APIs by themselves are pretty much fully composed of containers.”

Cyverse is an NSF-funded undertaking that assists existence scientists use cyberinfrastructure to handle huge datasets and complicated analyses, so enabling facts-pushed discovery. CyVerse includes a facts storage facility, an interactive, website-primarily based, analytics platform, and cloud infrastructure to for computation, investigation, and storage — a lot of it create and preserved at TACC.

“Docker containers are the most important way Cyverse researchers combine custom applications into the platform,” explained Fonner. “This includes non-interactive cloud and HPC applications, as effectively as interactive ‘VICE’ applications this sort of as Jupyter Notebooks.”

Nevertheless normally taken care of as a panacea, there are trade-offs to working with containers, Zynda states. Containers are not constantly as optimized for efficiency as they could be, and when they are produced, they are static and not straightforward to adjust.

“But when a container is printed and can address a problem, it’s possible it doesn’t want to adjust,” he explained.

TACC doesn’t only empower the use of containers, they are energetic in acquiring instruments to make containers easier to use. A person case in point of this is Rolling Gantry Crane (RGC) — named right after the equipment that offload containers from ships in harbors. RGC integrates containers into TACC’s environmental module program (Lmod) to empower familiar interactions, fundamentally producing the container program clear.

TACC also trains researchers to use containers as a result of frequent workshops, webinars, and integration into TACC’s Institutes.

“We thinks that software package containers are an important component of reproducible computing,” Fonner explained. “Containers are supported on all our HPC clusters and in our cloud infrastructure. Internally, we use Docker greatly in our standard enhancement tactics, and we deploy pictures to compute means working with each Docker and Singularity interchangeably. In a shorter time, containers have turn out to be a central component of how we help science at TACC.”

Written by Aaron Dubrow

Source: TACC