16. HPC Cluster Essentials: Tools, Techniques, and Best Practices [HPC in Julia]

We start this video by covering the basics of what a High Performance Computing cluster is - its architecture and how users interact with it. For the remainder of the video, we comprehensively look at the entire lifecycle of using a HPC to run your code - including connecting, editing files, submitting jobs and downloading code. We'll see how to connect to the HPC using VS code for a smooth and user-friendly experience.

Towards the end of the video, we also show how to run more advanced multiprocessing jobs, using either Distributed.jl or MPI and we end with a discussion of some tips and best practices to keep in mind when you are using a HPC.

This module was designed as an MPAGS (Midlands Physics Alliance Graduate School) module and aimed at postgraduates and early career researchers.

Resources:
OpenSSH for Windows - https://geekrewind.com/how-to-install...
Job Arrays - https://slurm.schedmd.com/job_array.html (and https://blog.ronin.cloud/slurm-job-ar...)

Timecodes:
00:00:00 Introduction
00:00:34 Cluster architecture
00:03:20 User access to a cluster
00:05:20 How to connect to a cluster
00:09:15 Navigating a bash terminal
00:13:57 SSH Passwordless Login
00:17:21 VS Code Remote SSH
00:19:27 Transferring files via Git
00:21:12 VS Code environment
00:21:59 SLURM scheduler
00:23:53 Loading software
00:26:48 Job scripts
00:32:45 Interactive Jobs
00:36:22 Connecting directly to a compute node in VS Code
00:39:28 Transferring files to and from a cluster
00:42:53 Using multiple nodes (Distributed.jl)
00:47:55 Running MPI code
00:52:51 Tips @ Tricks
00:58:28 Outro

Useful links:
Git/GitHub for Researchers:    • MLiS1 Introduction to Git by Jamie Mair  
Course Information Website: https://jamiemair.github.io/mpags-hig...
Julia Documentation: https://docs.julialang.org/en/v1/
MPAGS Registration: https://warwick.ac.uk/fac/sci/physics...
Course Notes: https://jamiemair.co.uk/courses/hpc