Amazon SageMaker HyperPod now supports model caching for faster inference autoscaling and reduced cold starts
Amazon SageMaker HyperPod now generally available with model caching, reducing cold starts for LLM inference by pre-loading model weights and container images, achieving up to 60% faster scale-out
Amazon SageMaker HyperPod now supports model caching, preloading model weights and container images onto cluster nodes to reduce pod startup from minutes to seconds. This resolves bottlenecks in large workloads like LLM inference where downloading container images and model weights was an issue. Model caching comprises two features: 'weight caching' storing model weights on local NVMe for fast access, and 'image caching' pre-downloading container images to skip ECR downloads. Benchmarks for models ranging from 57 GB to 145 GB show about a 60% speedup in scale-out, with image caching reducing image pull time by over two minutes (a 97% reduction). The benefits increase with larger model sizes while maintaining source path reliability. Customers can enable model caching by adding a modelCacheConfig section to InferenceEndpointConfig or JumpStartModel resources using the HyperPod Inference Operator, which manages the entire lifecycle without manual configuration or cleanup. Model caching is generally available in all regions where SageMaker HyperPod is offered.
Why it matters
For large-scale LLM inference workloads like chat assistants, agentic pipelines, RAG, and document analysis, model caching eliminates cold starts and significantly improves response times