Amazon SageMaker AI inference now supports GPU partitioning for real-time endpoints

Amazon SageMaker AI inference now supports GPU partitioning, enabling multiple model execution on a single GPU. Using NVIDIA Multi-Instance GPU technology, a GPU can be divided into up to seven isolated partitions, each with dedicated memory and compute. Customers running small to medium models such as fine-tuned variants, embedding models, rankers, and classifiers no longer need to allocate a full accelerator per model, avoiding payment for unused GPU capacity. With GPU partitioning, users select how to split the GPU behind an endpoint and place each model in a suitably sized partition. Multiple models can share one accelerator, with hardware-enforced isolation ensuring that busy models do not affect the memory or throughput of neighboring models. This capability also allows access to new instance generations with capacities better matched to workload requirements. When creating an endpoint to host inference components, GPU partitioning can be configured while keeping application code unchanged by continuing to call each model by name. GPU partitioning is available for P4d, P5, G7, and G7e instances in US East (N. Virginia), US East (Ohio), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Tokyo), Europe (Frankfurt), and Europe (Ireland).

Why it matters

Beneficial for customers hosting small to mid-size models, improving cost efficiency and resource optimization