HPC kitchen [containers/environments]: Moving code around

When you go to a different kitchen to cook, everything is different and it slows you down. This is a worse problem for computers, which don't have a brain and thus the recipe has to be exact. As you get farther and farther into scientific coding, you'll see how hard of a difficulty this "installing software and makin g it work" is. Environments and containers solve this problem.

More notes:

This metaphor is by no means perfect - containers are much more complex than a box containing some objects. Still, the main idea of "create your standard tools from a manifest, reproducible" still stands.

Operating system containers: Apptainer (formerly Singularity) for HPC kind of work. Docker and similar are often used for services and has problems running on clusters.

For Python virtual environments handle Python dependencies and Conda environments handle Python + compiled dependencies well. Conda is made for the complexities of scientific code (and can also be used for non-Python code too).

00:05 Cooking in another kitchen: everything is different
00:50 Working with no dependencies (why this is a bad idea)
01:25 Containers let you transport code dependencies
02:02 Creating a container from a manifest/definition.
03:15 Types of containers
03:20 Operating system containers (e.g. Singularity, Apptainer, Docker)
03:50 Software dependenciy containers (e.g. Python virtual environments, Conda environments)
05:10 Updating containers
06:05 Why recreating from scratch is so important
07:00 What's next?

-----

Playlist:    • HPC kitchen: cooking metaphors to scientif...