Course: Navigating and Working in HPC Environments | DiRAC Training

Why use a cluster?

Frequently, research problems that use computing can outgrow the desktop or laptop computer where they started:

A cosmology student wants to cross-validate their model. This involves running the model thousands of times, but each run takes an hour. Running on their laptop will take over a month!
An astronomy researcher has been using small datasets of sequence data, but soon will be receiving a new type of sequencing data that is 10 times as large. It’s already challenging to open the datasets on their computer, analysing these larger datasets will probably crash it.
An astrophysicist is using a fluid dynamics package that has an option to run in parallel. So far, they haven’t used this option on their desktop, but in going from 2D to 3D simulations, simulation time has more than tripled and it might be useful to take advantage of that feature.

In all these cases, what is needed is access to more computers that can be used at the same time. Luckily, large-scale computing systems (shared computing resources with lots of computers) are available at many universities, labs, or through national networks. Compared to personal computers, these facilities usually have more memory, more storage, and more central processing units (CPUs), which operate at higher speeds, and also have faster connections with other computing systems. They are frequently called “clusters”, “supercomputers” or resources for “high performance computing” or HPC.

Using a cluster often has the following advantages for researchers:

Speed. With many more CPU cores, often with higher performance specs than a typical laptop or desktop, HPC systems can offer significant speedup.
Volume. Many HPC systems have both the processing memory (RAM) and disk storage to handle very large amounts of data. Terabytes of RAM and petabytes of storage are available for research projects.
Efficiency. Many HPC systems operate a pool of resources that are drawn on by many users. In most cases when the pool is large and diverse enough the resources on the system are used almost constantly.
Cost. Bulk purchasing and government funding mean that the cost to the research community for using these systems is significantly less than it would be otherwise.
Convenience. Maybe your calculations just take a long time to run or are otherwise inconvenient to run on your personal computer. There’s no need to tie up your own computer for hours when you can use a purpose-built facility instead.

This is how a large-scale computing system like a cluster can help solve problems like those listed at the start of the lesson.

Thinking ahead

How do you think using a large-scale computing system will be different from using your laptop? Think about some differences you may already know about and some differences/difficulties you imagine you may run into.

The command line

Using HPC systems often involves the use of a shell through a command-line interface (CLI) and either specialised software or programming techniques. The shell is a program with the special role of running other programs rather than performing calculations or similar tasks itself; it is called “the shell” because it encloses the operating system in order to hide some of its complexity and make it simpler to interact with. The user types commands that are sent to the shell, which then figures out what commands to run and orders the computer to execute them. The most popular Unix shell is Bash, the Bourne Again SHell (so-called because it’s derived from a shell written by Stephen Bourne). Bash is the default shell on most modern implementations of Unix and in most packages that provide Unix-like tools for Windows.

Interacting with the shell is done via a CLI on most HPC systems. In the earliest days of computing, the only way to interact with early computers was to rewire them. From the 1950s to the 1980s most people used line printers. These devices only allowed input and output of the letters, numbers, and punctuation found on a standard keyboard, so programming languages and software interfaces had to be designed around that constraint and text-based interfaces were the way to do this. A typing-based interface is often called a command-line interface to distinguish it from a graphical user interface, or GUI, which most people now use. The heart of a CLI is a read-evaluate-print loop, or REPL: when the user types a command and then presses the Enter (or Return) key, the computer reads it, executes it, and prints its output. The user then types another command, and so on until the user logs off.

Learning to use Bash or any other shell sometimes feels more like programming than like using a mouse. Commands are terse (often only a couple of characters long), their names are frequently cryptic, and their outputs are lines of text rather than something visual like a graph. However, using a command-line interface can be extremely powerful and learning how to use one will allow you to reap the benefits described above.

The rest of this lesson

The only way to use HPC resources like DiRAC is by learning to use the command line. This introduction to HPC systems has two parts:

We will learn to use the UNIX command line (by using Bash).
We will use our new Bash skills to connect to and operate a supercomputer.

The skills we learn here have other uses beyond just HPC: Bash and UNIX skills are used everywhere, be it for web development, running software, or operating servers. It’s become so essential that Microsoft now ships it as part of Windows! Knowing how to use Bash and HPC systems will allow you to operate virtually any modern device. With all of this in mind, let’s connect to a cluster and get started!

Foundation HPC Skills

Navigating and Working in HPC Environments

0% complete

0% complete
Resume

Foundation in Bash

Develop the essential skills needed to work effectively in computational research environments through this course.

Navigating and Working in HPC Environments provides a practical introduction to Bash, equipping you with the essential skills to interact with and manage Linux-based systems effectively.

Show More Show Less
- Overview Page
- Enrol on course Page
Show More

The Course
- Select activity Why use a cluster?
  
  Why use a cluster? Page
  
  Students must
  
  Mark as done
  
  Teaching: 25 min
  Exercises: 5 min
  
  Questions
  
  Why would I be interested in High Performance Computing (HPC)?
  
  What can I expect to learn from this course?
  
  Objectives
  
  Be able to describe what an HPC system is.
  
  Identify how an HPC system could benefit you.
- Select activity Connecting to the remote HPC system
  
  Connecting to the remote HPC system Page
  
  Teaching: 25 min
  Exercises: 10 min
  
  Questions
  
  How do I open a terminal?
  
  How do I connect to a remote computer?
  
  What is an SSH key?
  
  Objectives
  
  Connect to a remote HPC system.
  
  Understand the importance of keeping SSH keys, and any other credentials, secure.
  
  Describe the major benefit of 2-factor authentication.
  
  Not available unless: You belong to any group
- Select activity Moving around and looking at things
  
  Moving around and looking at things Page
  
  Teaching: 15 min
  Exercises: 5 min
  
  Questions
  
  How do I navigate and look around the system?
  
  Objectives
  
  Learn how to navigate around directories and look at their contents
  
  Explain the difference between a file and a directory.
  
  Translate an absolute path into a relative path and vice versa.
  
  Identify the actual command, flags, and filenames in a command-line call.
  
  Demonstrate the use of tab completion, and explain its advantages.
  
  Not available unless: You belong to any group
- Select activity Writing and reading files
  
  Writing and reading files Page
  
  Teaching: 30 min
  Exercises: 15 min
  
  Questions
  
  How do I create/edit text files?
  
  How do I move/copy/delete files?
  
  Objectives
  
  Learn to use the nano text editor.
  
  Understand how to move, create, and delete files.
  
  Learn to copy files to/from a remote DiRAC resource.
  
  Use and create archival collections of files for efficient file transfer.
  
  Not available unless: You belong to any group
- Select activity Wildcards and pipes
  
  Wildcards and pipes Page
  
  Teaching: 45 min
  Exercises: 10 min
  
  Questions
  
  How can I run a command on multiple files at once?
  
  Is there an easy way of saving a command’s output?
  
  Objectives
  
  Redirect a command’s output to a file.
  
  Process a file instead of keyboard input using redirection.
  
  Construct command pipelines with two or more stages.
  
  Explain what usually happens if a program or pipeline isn’t given any input to process.
  
  Not available unless: You belong to any group
- Select activity Scripts, variables, and loops
  
  Scripts, variables, and loops Page
  
  Teaching: 45 min
  Exercises: 10 min
  
  Questions
  
  How do I turn a set of commands into a program?
  
  Objectives
  
  Write a shell script
  
  Understand and manipulate UNIX permissions
  
  Understand shell variables and how to use them
  
  Write a simple “for” loop.
  
  Not available unless: You belong to any group
- Select activity Using Bash scripts in pipes
  
  Using Bash scripts in pipes Page
  
  Teaching: 25 min
  Exercises: 10 min
  
  Questions
  
  Can I use scripts I write in pipes as well?
  
  Objectives
  
  Use a Bash script we’ve written within a pipe.
  
  Create a Bash script that reads input from other commands within a pipe.
  
  Not available unless: You belong to any group
Acknowledgements
- Select activity Contributors
  
  Contributors Page