Amazon SageMaker HyperPod Inference Gateway now available for scalable LLM inference
Amazon SageMaker HyperPod Inference Gateway is a Kubernetes-native, GPU-aware routing system that deploys as a single EKS managed add-on on existing SageMaker HyperPod infrastructure, reducing first-token latency by up to 82% and p99 TTFT by 97-98% in mixed-hardware and burst traffic scenarios.
Amazon SageMaker HyperPod Inference Gateway is a Kubernetes-native GPU-aware routing system deployed as an EKS managed add-on over existing SageMaker HyperPod infrastructure, requiring no application code changes. It replaces traditional round-robin load balancing with real-time inference signal-based routing, reducing first-token latency by up to 82% and p99 TTFT by 97-98% in mixed hardware and burst traffic scenarios. Composed of three core components-Envoy endpoints terminating HTTPS traffic and exposing a single private endpoint per cluster, body-based routers reading model names directly from each incoming request to route to the correct GPU pool enabling many models behind one endpoint URL with no client-side changes, and endpoint pickers real-time scoring each model server pod across six inference-level signals to select the best pod for each request-it works arbitrarily with OpenAI-compatible model servers like vLLM and SGLang, also requiring no application code changes. Currently, per-cluster routing is available in all AWS regions supporting SageMaker HyperPod inference add-ons, with upcoming features including centralized fleet gateway cross-cluster and cross-region routing, global rate limiting, and cost-tier-aware traffic shaping.
Why it matters
A new routing system for improving large language model (LLM) inference performance. It integrates with existing SageMaker HyperPod infrastructure, providing optimal request routing based on real-time inference signals without application code changes.