Infrastructure Design

Batch Computing Infrastructure - Large-Scale Parallel Processing with AWS Batch

Learn how to build large-scale batch processing with AWS Batch. Covers job queue design, auto-scaling compute environments, cost optimization with Spot Instances, and building batch infrastructure ideal for scientific computing and large-scale data processing.

約 7 分で読めます

Batch Processing Challenges and Where AWS Batch Fits

Large-scale batch processing is essential in many fields including scientific computing, financial risk analysis, media transcoding, machine learning training, and genome analysis. Building batch processing infrastructure on-premises involves challenges such as procuring HPC clusters, building and operating job schedulers (PBS, Slurm, Grid Engine), capacity planning for peak loads, and wasted resources during idle periods. AWS Batch is a fully managed batch processing service that solves these challenges. It centrally manages job definition, queuing, scheduling, and automatic provisioning and scaling of compute resources. It automatically launches EC2 instances or Fargate tasks based on the number and requirements of jobs, and releases resources when processing completes, eliminating idle costs.

Designing Job Definitions and Compute Environments

AWS Batch job definitions declaratively describe the container image to execute, vCPU and memory requirements, environment variables, mount points, and retry strategies. Docker container-based job execution lets you run the same container validated in your local environment directly on AWS Batch, eliminating issues caused by environment differences. Compute environments come in two types: managed and unmanaged. In managed environments, AWS Batch automatically handles instance launching, termination, and scaling. For instance type specification, you can either specify particular types or use optimal to let AWS Batch automatically select the best type. You can specify P4d or G5 instances for machine learning jobs requiring GPUs, or R6i instances for memory-intensive processing, enabling optimal resource allocation for each workload. Choosing a Fargate-type compute environment eliminates even EC2 instance management, achieving fully serverless batch processing. Below is an example of registering a job definition with the AWS CLI. ```bash aws batch register-job-definition \ --job-definition-name my-batch-job \ --type container \ --container-properties '{ "image": "123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/my-app:latest", "vcpus": 4, "memory": 8192, "retryStrategy": {"attempts": 3} }' ```

Cost Optimization with Spot Instances

The combination of AWS Batch and EC2 Spot Instances is a powerful way to reduce batch processing costs by up to 90%. Spot Instances let you use EC2 surplus capacity at significantly discounted prices, making them ideal for interruption-tolerant batch processing. When you enable Spot Instances in an AWS Batch managed compute environment, it automatically switches instance types based on Spot price fluctuations, securing the most cost-efficient resources. If a Spot Instance is interrupted, AWS Batch automatically reschedules the job on another instance and continues processing based on the retry strategy. Selecting SPOT_CAPACITY_OPTIMIZED as the allocation strategy prioritizes capacity from instance pools with lower interruption probability, improving job stability. Hybrid configurations mixing On-Demand and Spot Instances are also possible, enabling cost optimization strategies where critical jobs run on On-Demand and others run on Spot.

Job Dependencies and Workflow Construction

AWS Batch can define dependencies between jobs, enabling construction of complex workflows. It supports flexible execution order control including sequential execution where one job starts after another completes, and fan-in/fan-out patterns where an aggregation job runs after multiple jobs all complete. The array job feature lets you submit thousands of jobs with different parameters from a single job definition, ideal for parameter sweeps and large-scale data partitioning. Integration with Step Functions lets you visually design and manage more complex workflows that include AWS Batch jobs. For example, you can build an end-to-end pipeline where EventBridge detects data uploaded to S3, Step Functions executes a preprocessing Lambda function, AWS Batch performs large-scale parallel computation, results are stored in DynamoDB, and notifications are sent via SNS. To comprehensively learn batch large-scale computing architecture, refer to technical books (Amazon).

Technical Background and Design Philosophy of Batch Computing

The design philosophy of batch computing is elastic resource management - securing large amounts of compute resources only when needed and releasing them immediately after processing completes. AWS Batch embodies this philosophy, achieving efficient resource sharing through the separation architecture of job queues and compute environments. By setting different priorities on multiple job queues that share the same compute environment, you can build a system where high-priority jobs secure resources before low-priority ones. Fair share scheduling policies also enable equitable distribution of compute resources among multiple teams or projects. Internally, AWS Batch uses Amazon ECS as its container orchestration foundation, benefiting from ECS's mature container management capabilities. The multi-node parallel job feature supports MPI (Message Passing Interface)-based parallel computation spanning multiple EC2 instances, accommodating HPC workloads. This feature enables running large-scale scientific computations requiring inter-node communication, such as weather simulations and computational fluid dynamics, on AWS Batch.

Summary - Choosing a Batch Computing Platform

AWS Batch is a service for running large-scale batch processing in a fully managed manner, comprehensively providing job scheduling, automatic compute resource scaling, and cost optimization through Spot Instances. Docker container-based job execution maintains environment consistency, while array jobs and dependency definitions enable complex workflow construction. When considering large-scale parallel processing for scientific computing, data processing, or media conversion, a batch infrastructure centered on AWS Batch is the optimal choice.

Related Services

AWS BatchA managed service for efficiently scheduling and running batch computing jobsAmazon EC2A compute service that provides virtual servers in the cloud

Related Articles

GPU-Based Machine Learning Training with AWS Batch - Cost-Efficient Large-Scale TrainingRun GPU training with your existing Docker containers, and cut costs by up to 90% using Spot Instances and checkpointing. Includes guidance on when to choose Batch over SageMaker.Automating Batch Computing with AWS Batch - Designing Job Queues and Compute EnvironmentsLearn about job scheduling with AWS Batch, choosing between Fargate and EC2 compute environments, and leveraging Spot Instances for cost optimization.Large-Scale Batch Processing with AWS Batch - Job Queue Design and Cost OptimizationLearn how to design job queue priorities, choose between Fargate and EC2 compute environments, and build complex computational pipelines using array jobs and job dependencies.

More on This Topic

Why Auto Scaling Scales Out Fast but Scales In Cautiously - The Design Intent Behind Asymmetric Decision LogicThis article explains why EC2 Auto Scaling executes scale-out immediately while applying a cooldown period for scale-in, the flapping prevention mechanism, and the internal logic of target tracking scaling.Demand-Driven Infrastructure with AWS Auto Scaling - Designing and Optimizing Scaling PoliciesLearn how to use target tracking, predictive, and scheduled scaling policies effectively, and optimize costs with mixed instances policies that leverage Spot Instances.AWS Fault Domain Design - How the Three-Layer Structure of AZs, Regions, and Partitions Protects AvailabilityLearn why AWS infrastructure is designed with three layers of fault domains - AZs (fault isolation), Regions (geographic separation), and Partitions (political separation) - and how far failures propagate at each layer, with real-world examples.Distributed Systems Principles Learned from AWS Outages - How Past Major Incidents Reshaped ArchitectureUsing AWS's published incident reports as case studies - including the S3 outage (2017), Kinesis outage (2020), and the unique nature of us-east-1 - this article explains design principles such as Shuffle Sharding, Static Stability, and Cell-based Architecture.Why AWS Builds Regions Where It Does - The Hidden Criteria Behind Data Center Site SelectionWe explain the criteria AWS considers when deciding region locations, including power supply, geopolitical risk, data sovereignty legislation, network connectivity, and natural disaster risk, with concrete examples from specific regions.Why AWS Availability Zone IDs Differ Per Account - The Design Intent Behind AZ MappingExplains how us-east-1a maps to different physical AZs per account, why AZ IDs (use1-az1) were introduced, the design intent of even capacity distribution, and considerations for cross-account AZ specification.Automating Batch Computing with AWS Batch - Designing Job Queues and Compute EnvironmentsLearn about job scheduling with AWS Batch, choosing between Fargate and EC2 compute environments, and leveraging Spot Instances for cost optimization.Large-Scale Batch Processing with AWS Batch - Job Queue Design and Cost OptimizationLearn how to design job queue priorities, choose between Fargate and EC2 compute environments, and build complex computational pipelines using array jobs and job dependencies.

Similar Articles and Services

AWS BatchA fully managed service that automatically schedules hundreds of thousands of batch processing jobs and runs them on optimal computing resourcesAutomating Batch Computing with AWS Batch - Designing Job Queues and Compute EnvironmentsLearn about job scheduling with AWS Batch, choosing between Fargate and EC2 compute environments, and leveraging Spot Instances for cost optimization.Large-Scale Batch Processing with AWS Batch - Job Queue Design and Cost OptimizationLearn how to design job queue priorities, choose between Fargate and EC2 compute environments, and build complex computational pipelines using array jobs and job dependencies.GPU-Based Machine Learning Training with AWS Batch - Cost-Efficient Large-Scale TrainingRun GPU training with your existing Docker containers, and cut costs by up to 90% using Spot Instances and checkpointing. Includes guidance on when to choose Batch over SageMaker.