New featureMedium

Amazon SageMaker AI inference now supports G7 instances

Amazon SageMaker AI inference now supports G7 instances powered by NVIDIA RTX PRO 4500 Blackwell Server Edition GPUs, delivering up to 4.6x AI inference performance over previous-generation G6 instances

Amazon SageMaker AI inference now supports G7 instances powered by NVIDIA RTX PRO 4500 Blackwell Server Edition GPUs. These instances offer up to 4.6x AI inference performance compared to previous-generation G6 instances. G7 instances provide 32 GB of GPU memory per GPU with 5th Generation Tensor Cores, up to 700 Gbps of EFA-enabled networking (7x compared to G6), and up to 7.6 TB of local NVMe SSD storage for keeping large models close to compute. These capabilities make G7 instances well suited for serving models in the 7B-30B parameter range, image and video generation workloads, and multi-model inference endpoints that benefit from higher memory bandwidth and throughput. You can deploy models on G7 instances using the SageMaker AI Inference console, API, or SDK by specifying G7 instance types (such as ml.g7.xlarge through ml.g7.48xlarge) in your endpoint configuration.

Read the original AWS announcement

Related articles and terms

Amazon Elastic Inference[Discontinued] Service that attached fractional GPU accelerators to EC2 and SageMaker for inference. Inferentia (Inf1 / Inf2), GPU instances, and SageMaker AI inference are the replacementsAmazon SageMakerA fully managed platform for building, training, and deploying machine learning models end-to-end, significantly boosting productivity for data scientists and ML engineersBuilding an ML Platform with Amazon SageMaker - From Model Development to DeploymentFrom development in Studio to managed spot training, MLOps with Pipelines, and data drift detection with Model Monitor, this article covers how to integrate the entire ML lifecycle.The Layered Architecture of AWS AI/ML Services - Flexibility Through the Three Tiers of SageMaker, Bedrock, and API ServicesThis article organizes AWS AI/ML services into three layers - SageMaker (full control), Bedrock (managed generative AI), and Rekognition/Comprehend/etc. (API-based) - and explains AWS's flexibility through comparisons with GCP Vertex AI and Azure OpenAI Service, including custom silicon integration.AWS Graviton and Custom Silicon Strategy - How In-House Chips Are Reshaping Cloud EconomicsExplore how AWS's Arm-based Graviton processors and AI-focused custom silicon (Inferentia and Trainium) are transforming cloud cost structures and performance, compared with Azure and GCP approaches.