Slurm HPC Cluster on AWS for AI/ML Workflows

Research computing departments at academic institutions face growing demand for compute resources to support large-scale AI and ML workflows. Because most groups manage their own on-premises infrastructure, it can be difficult to respond as conference deadlines and the fast-moving AI landscape drive sharp spikes in demand. As a result, hardware may not be available when it's most needed, and maintenance costs persist even when resources are not being used.

Cloud providers allow for virtually infinite scalability, while at the same time not charging for resources that aren't used. AWS's ParallelCluster, for example, allows for HPC configurations that reserve resources only when they are requested, minimizing idle cost without a noticeable change in experience from the user's perspective.

This repository constructs cloud architecture for a compute environment that supports the same workflows researchers are used to running on-premises, including distributed AI/ML training workflows.

The entire stack is defined as code, with Terraform provisioning infrastructure, Packer building a custom compute image, and Ansible configuring nodes at boot.

Design priorities

• Performance: The cluster is built with high-bandwidth RDMA interconnects as well as I/O-optimized flash SSD storage.
• Cost efficiency: Compute scales to zero when idle, so the cluster only costs money while jobs are actually running.
• Reproducibility: Every layer is version-controlled and rebuildable.

Tech stack: AWS ParallelCluster · Slurm · Terraform · Packer · Ansible · FSx for Lustre & OpenZFS · Apptainer · Amazon RDS · EFA

How to Deploy

• Install terraform and packer using Hashicorp's instructions.

Install nvm and nodejs:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.5/install.sh | bash
source ~/.zshrc
nvm install --lts

Create Python virtual environment to run pcluster commands:

python -m venv .venv
source .venv/bin/activate
pip install --upgrade "aws-parallelcluster==3.15.1"

Initialize packer and terraform, then deploy:

terraform init
cd packer
sudo chmod g+w ~/.config
packer init .
packer build .
cd ..
terraform apply

Architecture

A single VPC hosts the cluster. A public subnet holds the internet-facing head node (the Slurm controller and SSH login node); a private subnet holds the compute fleet, reachable only from the head node. Terraform wires together the network, storage, database, and IAM, then hands a rendered config to ParallelCluster, which manages the Slurm control plane and the elastic compute nodes.

Compute

Two Slurm partitions cover common academic ML workloads:

• cpu: c5n.9xlarge for preprocessing, simulation, and multi-node MPI jobs.
• gpu: g4dn.12xlarge (4× NVIDIA T4) for training and inference.

Both partitions run with MinCount = 0, so nodes are launched on demand when jobs are queued and terminated when idle. Nodes launch into a placement group with EFA (Elastic Fabric Adapter) enabled to give distributed jobs low-latency, high-bandwidth interconnect.

Slurm's accounting database runs on a managed RDS MySQL instance, giving persistent sacct history and usage tracking that survives cluster teardown. The DB password is generated by Terraform, stored in AWS Secrets Manager, and read by the head node through a scoped IAM policy.

Storage

Two filesystems back the cluster, each matched to its access pattern:

• /home — FSx for OpenZFS (NFS): Durable, general-purpose home directories with LZ4 compression, automatic daily backups, and per-user storage quotas enforced at the filesystem level.
• /data — FSx for Lustre: A high-throughput parallel filesystem for training datasets and job scratch, where many nodes read the same data at once. It is linked to an S3 bucket via a Data Repository Association, so objects in S3 appear transparently under /data and new results are automatically exported back to S3. This is not for backup purposes but instead allows for lazy loading so that data is only pulled onto the fast storage when needed. Lustre is extremely performant but can degrade with higher file counts.

Networking

• Public subnet: Head node only, with an Internet Gateway for SSH access and administration.
• Private subnet: Compute nodes, with no direct inbound access. Outbound internet (package installs, pulling images) routes through a NAT Gateway.
• S3 Gateway VPC Endpoint: Compute nodes reach S3 directly via AWS rather than paying NAT data-processing charges for large dataset transfers.

Software

• Custom AMI (Packer): Compute nodes boot from a pre-baked Ubuntu 22.04 image containing Apptainer (rootless containers for reproducible ML environments), cluster tooling, and benchmarking tools. Baking these into the image keeps node startup fast and deterministic instead of installing packages on every scale-up. The build publishes its AMI ID to SSM Parameter Store, which Terraform reads to wire the image into the cluster.
• Node configuration (Ansible): ParallelCluster's OnNodeConfigured custom actions pull playbooks from S3 and run them as each node boots. The playbooks provision consistent Linux users/UIDs across the cluster, set per-user Lustre quotas, and register users into Slurm accounting. Keeping this in Ansible (rather than the AMI) means user and policy changes don't require rebuilding the image.

Next Steps

• Integrate software modules via Lmod so that researchers can use software that is compartmentalized and does not conflict with other software.
• Create a streamlined workflow for adding new users.