Amazon Elastic Inference
A service that attaches low-cost GPU accelerators to EC2 or SageMaker instances to optimize machine learning inference workloads
Overview
Amazon Elastic Inference attaches GPU-based inference accelerators to EC2 instances or SageMaker endpoints, reducing machine learning model inference costs by up to 75%. When using full GPU instances (p3, g4dn), GPU compute capacity is often underutilized for inference workloads. Elastic Inference achieves an optimal balance of cost and performance by adding only the necessary amount of GPU resources. It supports the major frameworks TensorFlow, PyTorch, and MXNet.
Accelerator Types and Attachment Methods
Elastic Inference provides three accelerator sizes: eia2.medium (1GB GPU memory), eia2.large (2GB), and eia2.xlarge (4GB). Accelerators are attached at EC2 instance launch time and connected via a private link within the VPC. A typical configuration adds an accelerator to a CPU instance (c5, m5, etc.) - for example, a c5.xlarge + eia2.medium combination achieves inference throughput equivalent to p3.2xlarge at approximately one-third the cost. Select the accelerator size based on model size and inference latency requirements, choosing a size whose GPU memory can accommodate the model. For SageMaker endpoints, simply specifying the accelerator type in the endpoint configuration automatically attaches it, offloading the GPU processing portion of inference requests to the accelerator.
Framework Integration and Inference Optimization
Elastic Inference integrates with TensorFlow Serving, PyTorch, and Apache MXNet inference runtimes. For TensorFlow, SavedModel format models can be used as-is with GPU acceleration applied without code changes. Operations in the inference graph that can be accelerated by GPU (convolutions, matrix multiplications, activation functions) are automatically executed on the accelerator, while other operations are processed on CPU - this split execution happens transparently. Combining with model quantization (FP32 to FP16 or INT8) can reduce accelerator memory usage while further improving inference throughput. For batch inference, batching strategies that send multiple requests to the GPU together maximize accelerator utilization efficiency.
Cost Optimization and Use Case-Specific Design
Elastic Inference's greatest value lies in cost optimization for inference workloads. While training requires massive GPU computation, inference involves less computation per request, and full GPU instances often see GPU utilization of only 10-30%. Elastic Inference eliminates this inefficiency by charging only for the minimum necessary GPU resources. For real-time inference (image classification, natural language processing, recommendations), an effective design selects the smallest accelerator size that meets latency requirements and adjusts instance count with Auto Scaling. Computer vision models (ResNet, YOLO) often work well with eia2.medium, while large language model inference requires eia2.xlarge. Monitor accelerator GPU utilization via CloudWatch metrics - consider sizing up if consistently above 80%, or sizing down if below 30%.