HyperQueue: ergonomic HPC task executor written in Rust

Hi! Me and my colleagues are working on an HPC (High Performance Computing)-focused tool for executing workflows/pipelines/task graphs in an ergonomic and efficient manner. It is called HyperQueue, it is open-source (https://github.com/It4innovations/hyperqueue) and it is of course written in Rust :) (previous incarnations of the tool were written in C++, which was a nightmare to maintain). It leverages async Rust heavily, not just for I/O (we use Tokio), but also for communicating between a number of independent "processes" (actors) running on a single thread.

Here's a bit of context:

When you want to execute something on an HPC cluster, you typically have to go through a job manager, such as PBS/Torque or SLURM, which forces you to split your computation into individual jobs that go through a queue. Nowadays, it is quite common to define HPC applications using task graphs, a set of (interconnected) tasks that are composed together to perform some complex computation. But in many cases, you cannot easily take the tasks from your problem and just create a single job per task, for various reasons. The number of allowed jobs is usually quite strictly limited and each job has significant overhead, which is bad especially if the task computation is short. Current HPC nodes are also quite powerful, and usually a single task cannot fully utilize the whole node, which is problematic because job managers often allow only a single job to execute on a node at the same time, usually for security reasons.

This leads to a situation where the user has to manually split/group tasks into a smaller number of jobs. This is both a lot of effort and it can cause inefficiencies, because the tasks are split statically before the start of the computation, without taking into account dynamic properties like their execution duration, resource requests, current load of the cluster etc.

HyperQueue takes a different approach. It disentangles the definition of a computation (task graph) with the provision of computational resources (PBS/SLURM jobs). Therefore you can simply submit your task graph into HQ and it will take care of distributing and load balancing its tasks amongst all available computing nodes (workers) of the cluster. It can also automatically provide the computational resources on your behalf by automatically submitting jobs into PBS/Slurm.

For more details, you can check a poster about HyperQueue that was presented at the SuperComputing'21 conference.

As an example, let's say that you want to execute a program on a HPC cluster a thousand times, with each instance calculating something slightly different (e.g. a different parametrization of a simulation) based on a numeric index. You can do this with HQ simply like this:

$ hq submit --array 1-1000 -- my-program

This would execute my-program, with each invocation receiving a different value of an env. variable (HQ_TASK_ID=1, HQ_TASK_ID=2, etc.). The executions would be distributed amongst all available nodes on a computing cluster, and the user doesn't have to care about splitting the computation into individual jobs.

HQ contains a lot of other goodies, such as support for general resource requirements (you state that a task needs 2x GPU and 3x "resource X", the scheduler will find it a worker that currently has those resources available), NUMA-aware CPU scheduling, low overhead per task (<1ms), output streaming over the network to avoid filesystem congestion, Python API built with pyo3 that can be used to define dependencies between individual tasks and create tasks out of Python functions (á la Dask), etc.

It is also designed for effortless deployment, as it can be sometimes quite painful to deploy stuff on HPC clusters. HQ is distributed as a single, statically linked executable, without external dependencies, and doesn't require root.

If there are any HPC/cluster users here that want to try a fresh approach for task computation on HPC, we'd be glad for your feedback. We're big fans of Rust, and we're trying to increase its usage in HPC. If you know any other Rust HPC tools, let us know in the comments!