HPC Story: Interview with K. Hoste on Easybuild: Part 1

Can you introduce yourself and your role in EasyBuild?

I obtained a PhD in computer science in 2010 and I joined the HPC team at Ghent University (UGent) shortly after. My focus is on user support & training, including installing of scientific software. EasyBuild started as an in-house project, and was thrown in my lap in early 2011. We later released it to the public (April 2012) and since then, I slowly grew into becoming a lead developer, release manager, and … community cat herder.

Can you explain to our reader what EasyBuild is and what it does? What are the benefits for the users?

EasyBuild is a tool to make the installation of scientific software on supercomputing systems (which is usually done by compiling the software from source code) easier.

K. Hoste

Using EasyBuild, you can maintain a consistent central software stack [1] without compromising on the performance of the installations. Over the years, a worldwide community has grown around EasyBuild, both in terms of users and developers, and hence it has become an “expert system” for software installation on HPC systems.

Users are thus presented with a well-performing consistent software stack, where they don’t have to worry about the technical details of how the software was built or installed. That way, they can focus on the research, rather than spending time trying to get things to work (which may result in ending up with a software stack that isn’t performing as well as it could).

Users can also use EasyBuild themselves to manage their own stack on top of what is provided centrally, for example, to install software they develop themselves, or to install other software that was modified or configured differently from the central installations. Indeed, using EasyBuild does not require admin privileges!

And for system administrators?

It greatly alleviates the burden of providing a rich central software stack to end users, which can easily consist of thousands of different software packages (even ignoring different versions!). Without a tool like EasyBuild, this would be unfeasible unless you have a dedicated team of people who can work on this full-time. EasyBuild also allows people to collaborate with other HPC teams around the world, and help out each other regarding the installation of scientific software.

How did the story begin?

EasyBuild started as an in-house project in the HPC-UGent team in 2009. Shortly after the first central HPC cluster at UGent was installed, it became clear that dealing with the incoming software installation requests was going to be a problem without appropriate tooling, especially since the infrastructure was bound to grow in the future (which means that the same software will need to be installed again for different hardware).

Other HPC teams seemed to throw manpower at this problem, which was not available in the HPC-UGent team at the time (since it basically was 2 system administrators + 1 manager back then), so some crude Python scripting was done to facilitate the task of installing scientific software from source, and the project was named “EasyBuild”. Over time, the capabilities and the set of supported software packages grew, and the code base was re-designed a couple of times with the help of student interns.

In 2012, we felt confident enough about the usefulness and design of EasyBuild to make it available publicly as open-source software. Our main intention was to gather feedback from others in the HPC community since we still felt that we had to be overlooking existing tooling to tackle this ubiquitous problem. But it turned out that we were the first team crazy enough to start developing a tool to facilitate the installation of scientific software on HPC systems! So other HPC teams from the University of Luxembourg, The Cyprus Institute, the University of Basel in Switzerland, and the Jülich Supercomputing Centre in Germany started using it, and also contributed back enhancements and bug fixes. Before we knew it, there was a worldwide EasyBuild community of 100’s of HPC sites that were adopting EasyBuild to manage their central stack of scientific software installations…

What were the main challenges?

The quick growth of the EasyBuild community and the influx of questions, problem reports, and contributions that resulted from it was a challenge. Keeping up with all of that proved to be very difficult since we didn’t have a lot of time that we could dedicate to it. In 2015, we got over 1,000 pull requests [2], and we realized that we had to get help from people in the EasyBuild community to process the incoming contributions, answer questions, and provide help where needed. We found 5 people, early adopters who had been using EasyBuild for several years, who were willing to become EasyBuild maintainers. This gradually grew out to the group of about 20 EasyBuild maintainers we currently have [3]. I’m very grateful for all the work that these people are doing, mostly on top of their own work, since without their help it would be impossible to keep up with incoming contributions…

Next to the volume of contributions, the lack of structure in them also was a problem. In addition, many people who wanted to contribute to EasyBuild were not familiar with Git or GitHub, which was a required skill. To mitigate this, we implemented support in EasyBuild itself to open pull requests straight from the EasyBuild command line. This not only automates away the bulk of the actions you need to perform as a contributor (like creating a branch in a fork of the Git repository, adding files, pushing the branch to GitHub, opening a pull request using that branch, etc.), it also results in more structured contributions which make it significantly easier to review and process them. We also implemented support for testing contributions before they are merged into EasyBuild, and to report back detailed test reports in the pull requests which show that installing that particular software worked as expected on your system.

Slide explaining the procedure to open a new pull request. If you are interested, please also check the documentation.

These features were quickly adopted by the EasyBuild community, to the point where we are now getting 2,500 pull requests per year for so-called easyconfig files (“build recipes”), with over 90% of those being created by EasyBuild (as opposed to manually). So, although this automated contribution workflow has resulted in significantly more contributions, the group of EasyBuild maintainers can mostly keep up thanks to the well-structured contributions and the ease with which they can be tested.

 
Graph obtained from the easyconfigs Github repository, showing the evolution of the number of pull request created manually (darker yellow) and with the command line (lighter yellow).

Among the many difficulties you stepped on, which one was the most difficult to overcome?

The biggest difficulty we deal with on an almost daily basis is the, let’s call them “creative”, installation procedures of some software projects. Although we’ve been playing this game for over a decade now, we still run into nasty surprises regularly. We know that most scientific researchers did not get any formal training in software development, which often shows in the way that they make their software available. Even large companies like Google (which develops popular scientific software packages like TensorFlow or AlphaFold) produce software applications that are difficult to install in the context of an HPC system.

In their defense, we do not make it easy on ourselves either by wanting to install everything from source code rather than using the pre-compiled binaries they often provide as well. We feel this is a necessary evil though. Indeed, when compiling from source code, you can make sure that the installations you obtain are properly optimized for the hardware on which it will be run. This can result in dramatic performance improvements! It's only by working together with HPC sites around the world on EasyBuild that we can still keep this up, since we see a growing variety of software packages that scientific researchers want to use, especially in bioinformatics and machine learning/AI.

I vented my frustration on this with a sarcastic talk at the FOSDEM’18 open source conference, which was really well received people who have experience in fighting software installations; see https://www.youtube.com/watch?v=NSemlYagjIU.

What would you recommend to the scientific researcher to ease the installation of their software?

If possible, people that develop software should try to use the standard tools of the programming language they used (e.g., pip, CMake, …), and publish the software through standard channels (PyPI for Python, CRAN for R, …). They might want to stick to the standards and follow commonly accepted guidelines and best practices. The installation script, if there is one, should not try to do complex things, such as downloading dependencies in the background: this is guaranteed to break at some point!

They should also provide (short) tests: this is especially useful for system administrators, which, knowing nothing about the software and usually lacking domain knowledge, should be able to check if what they installed works. EasyBuild runs tests by default as a part of the installation procedure if they are available!

Read the second part of the interview here.

• [1] For a given software, its “software stack” include the program itself, but also all its dependencies (code libraries, other programs, etc.) and the compiler that was used to compile all that. Everything needs to be consistent in order to provide a good user experience when using the software.
• [2] Contributions for new software packages, software version updates, etc.
• [3] See https://docs.easybuild.io/en/latest/Maintainers.html.