Infrastructure Design

Building an HPC Environment with AWS ParallelCluster - Automated Slurm Cluster Provisioning and Scaling

Automatically provision HPC clusters with CloudFormation and manage jobs with the Slurm scheduler. Also covers cost optimization with Spot Instances.

約 3 分で読めます

Overview of ParallelCluster

ParallelCluster is an open-source HPC cluster management tool provided by AWS. You define cluster configuration (instance types, node count, storage, networking) in a YAML configuration file and automatically provision it as a CloudFormation stack with the pcluster create-cluster command. The Slurm job scheduler is configured by default, allowing you to use existing Slurm job scripts as-is. It is used for large-scale parallel computing workloads including computational fluid dynamics (CFD), molecular dynamics, genome analysis, weather simulation, and financial risk calculation.

Auto Scaling and Cost Optimization

ParallelCluster's auto scaling works in conjunction with Slurm's job queue. When jobs are submitted, compute nodes automatically launch, and when nodes remain idle for a specified period after job completion, they automatically terminate. During periods with no jobs, the compute node count drops to zero, with only the head node incurring charges. Using Spot Instances can reduce HPC workload costs by up to 90%. Specifying multiple instance types in the capacity type configuration improves Spot availability. For checkpoint-capable applications, you can configure automatic job requeuing when Spot interruptions occur.

EFA and Shared Storage

EFA (Elastic Fabric Adapter) is a high-speed network interface for HPC workloads, providing 100 Gbps inter-node communication. It is effective for MPI (Message Passing Interface) based parallel computing where inter-node data exchange becomes a bottleneck. Simply enabling EFA in the ParallelCluster configuration automatically provisions EFA on compute nodes. For shared storage, you can choose from FSx for Lustre, EFS, and EBS. FSx for Lustre provides throughput of up to hundreds of GB/s, making it ideal for parallel reads of large datasets. Integration with S3 automatically imports data from S3 into the Lustre file system and exports computation results back to S3. For a comprehensive guide to ParallelCluster architecture, technical books (Amazon) are a useful reference.

Summary

ParallelCluster is a tool that automatically provisions Slurm-based HPC clusters on AWS. Auto scaling provides resource management aligned with job demand, Spot Instances reduce costs, and EFA delivers high-speed inter-node communication. It is ideal for migrating from on-premises HPC clusters or handling burst computing demands.

Related Services

AWS ParallelClusterAn open-source tool for building and managing HPC (High Performance Computing) clusters on AWSAmazon EC2A compute service that provides virtual servers in the cloudAmazon FSxFully managed file systems for Windows File Server, Lustre, NetApp ONTAP, and OpenZFS

Related Articles

Large-Scale Batch Processing with AWS Batch - Job Queue Design and Cost OptimizationLearn how to design job queue priorities, choose between Fargate and EC2 compute environments, and build complex computational pipelines using array jobs and job dependencies.GPU-Based Machine Learning Training with AWS Batch - Cost-Efficient Large-Scale TrainingRun GPU training with your existing Docker containers, and cut costs by up to 90% using Spot Instances and checkpointing. Includes guidance on when to choose Batch over SageMaker.AWS CodeDeploy EC2/On-Premises Deployment - Designing AppSpec and Lifecycle HooksDeclaratively define deployment target paths and lifecycle hooks using AppSpec files. Learn about fleet management with tag-based deployment groups and Auto Scaling integration.

More on This Topic

Why Auto Scaling Scales Out Fast but Scales In Cautiously - The Design Intent Behind Asymmetric Decision LogicThis article explains why EC2 Auto Scaling executes scale-out immediately while applying a cooldown period for scale-in, the flapping prevention mechanism, and the internal logic of target tracking scaling.Demand-Driven Infrastructure with AWS Auto Scaling - Designing and Optimizing Scaling PoliciesLearn how to use target tracking, predictive, and scheduled scaling policies effectively, and optimize costs with mixed instances policies that leverage Spot Instances.AWS Fault Domain Design - How the Three-Layer Structure of AZs, Regions, and Partitions Protects AvailabilityLearn why AWS infrastructure is designed with three layers of fault domains - AZs (fault isolation), Regions (geographic separation), and Partitions (political separation) - and how far failures propagate at each layer, with real-world examples.Distributed Systems Principles Learned from AWS Outages - How Past Major Incidents Reshaped ArchitectureUsing AWS's published incident reports as case studies - including the S3 outage (2017), Kinesis outage (2020), and the unique nature of us-east-1 - this article explains design principles such as Shuffle Sharding, Static Stability, and Cell-based Architecture.Why AWS Builds Regions Where It Does - The Hidden Criteria Behind Data Center Site SelectionWe explain the criteria AWS considers when deciding region locations, including power supply, geopolitical risk, data sovereignty legislation, network connectivity, and natural disaster risk, with concrete examples from specific regions.Why AWS Availability Zone IDs Differ Per Account - The Design Intent Behind AZ MappingExplains how us-east-1a maps to different physical AZs per account, why AZ IDs (use1-az1) were introduced, the design intent of even capacity distribution, and considerations for cross-account AZ specification.Batch Computing Infrastructure - Large-Scale Parallel Processing with AWS BatchLearn how to build large-scale batch processing with AWS Batch. Covers job queue design, auto-scaling compute environments, cost optimization with Spot Instances, and building batch infrastructure ideal for scientific computing and large-scale data processing.Automating Batch Computing with AWS Batch - Designing Job Queues and Compute EnvironmentsLearn about job scheduling with AWS Batch, choosing between Fargate and EC2 compute environments, and leveraging Spot Instances for cost optimization.

Similar Articles and Services

Batch Computing Infrastructure - Large-Scale Parallel Processing with AWS BatchLearn how to build large-scale batch processing with AWS Batch. Covers job queue design, auto-scaling compute environments, cost optimization with Spot Instances, and building batch infrastructure ideal for scientific computing and large-scale data processing.AWS BatchA fully managed service that automatically schedules hundreds of thousands of batch processing jobs and runs them on optimal computing resourcesLarge-Scale Batch Processing with AWS Batch - Job Queue Design and Cost OptimizationLearn how to design job queue priorities, choose between Fargate and EC2 compute environments, and build complex computational pipelines using array jobs and job dependencies.Automating Batch Computing with AWS Batch - Designing Job Queues and Compute EnvironmentsLearn about job scheduling with AWS Batch, choosing between Fargate and EC2 compute environments, and leveraging Spot Instances for cost optimization.