New featureMedium

Amazon SageMaker HyperPod now supports disaggregated prefill and decode

Amazon SageMaker HyperPod supports Disaggregated Prefill and Decode (DPD), separating LLM inference phases to dedicated GPU pools for consistent latency and higher throughput

Amazon SageMaker HyperPod now supports Disaggregated Prefill and Decode (DPD), an inference optimization that separates the two phases of large language model (LLM) inference-prefill and decode-onto dedicated GPU pools and transfers the key-value (KV) cache between them over Elastic Fabric Adapter (EFA) using GPU-Direct RDMA. With DPD, customers run compute-bound prefill on one set of GPUs and memory-bandwidth-bound decode on another, so the two phases no longer contend for the same resources. This delivers more consistent per-token latency under sustained concurrency, higher goodput at strict latency SLOs, and the ability to scale prefill and decode capacity independently to match the input and output distribution of the workload. An intelligent router automatically directs long-context requests through the disaggregated path and sends shorter prompts directly to the decoder. Customers enable DPD by adding a `pdSpec` section to the same `InferenceEndpointConfig` custom resource they already use for inference endpoints on the HyperPod Inference Operator, and DPD is composable with the existing KV cache offloading and intelligent routing features on HyperPod. DPD is available for SageMaker HyperPod clusters using the EKS orchestrator on EFA-capable instance types in all AWS Regions where Amazon SageMaker HyperPod is available.

Read the original AWS announcement

Related articles and terms

Amazon Elastic Inference[Discontinued] Service that attached fractional GPU accelerators to EC2 and SageMaker for inference. Inferentia (Inf1 / Inf2), GPU instances, and SageMaker AI inference are the replacementsAmazon SageMakerA fully managed platform for building, training, and deploying machine learning models end-to-end, significantly boosting productivity for data scientists and ML engineersBuilding an ML Platform with Amazon SageMaker - From Model Development to DeploymentFrom development in Studio to managed spot training, MLOps with Pipelines, and data drift detection with Model Monitor, this article covers how to integrate the entire ML lifecycle.The Layered Architecture of AWS AI/ML Services - Flexibility Through the Three Tiers of SageMaker, Bedrock, and API ServicesThis article organizes AWS AI/ML services into three layers - SageMaker (full control), Bedrock (managed generative AI), and Rekognition/Comprehend/etc. (API-based) - and explains AWS's flexibility through comparisons with GCP Vertex AI and Azure OpenAI Service, including custom silicon integration.GPU-Based Machine Learning Training with AWS Batch - Cost-Efficient Large-Scale TrainingRun GPU training with your existing Docker containers, and cut costs by up to 90% using Spot Instances and checkpointing. Includes guidance on when to choose Batch over SageMaker.