AWS First Steps
LanguageEnglish
New feature

Amazon SageMaker HyperPod now supports model caching for faster inference autoscaling and reduced cold starts

Amazon SageMaker HyperPod now generally offers model caching, solving cold starts for LLM inference with up to 60% faster scale-out and 97% reduction in image-pull time

Amazon SageMaker HyperPod now supports model caching, preloading model weights and container images onto cluster nodes to cut pod startup from minutes to seconds for inference optimization. This solves cold-start bottlenecks in large-scale LLM inference, with two features: weight caching stores model weights on local NVMe, avoiding network reads via S3 or FSx, while image caching preloads container images, skipping ECR downloads entirely. If a pod lands on a node without a warm cache, it automatically reloads from the original source, eliminating pod failure risks. Benchmarks for 57 GB to 145 GB models show about 60% faster scale-out, with image caching reducing image pull time by over two minutes (a 97% reduction), with greater benefits for larger models. Customers enable model caching by adding a modelCacheConfig section to their InferenceEndpointConfig or JumpStartModel resources via the HyperPod Inference Operator, which manages the full lifecycle without manual configuration or cleanup. Model caching is generally available in all regions where SageMaker HyperPod is offered.

Why it matters

Solves cold start bottlenecks in large-scale LLM inference, especially beneficial for large model deployments

Read the original AWS announcement

Related articles and terms

Amazon SageMakerA fully managed platform for building, training, and deploying machine learning models end-to-end, significantly boosting productivity for data scientists and ML engineersBuilding an ML Platform with Amazon SageMaker - From Model Development to DeploymentFrom development in Studio to managed spot training, MLOps with Pipelines, and data drift detection with Model Monitor, this article covers how to integrate the entire ML lifecycle.Amazon ECR Container Image Management - Lifecycle Policies and Image ScanningLearn how to automatically clean up old images with private repository lifecycle policies and detect vulnerabilities with image scanning.Amazon Elastic Inference[Discontinued] Service that attached fractional GPU accelerators to EC2 and SageMaker for inference. Inferentia (Inf1 / Inf2), GPU instances, and SageMaker AI inference are the replacementsUnderstanding Lambda Cold Starts and Choosing Among Three Optimization StrategiesThis article explains the mechanism behind Lambda cold starts from the Firecracker MicroVM lifecycle perspective, and compares optimization techniques across three axes: SnapStart, Provisioned Concurrency, and function design in terms of cost and constraints.