Region expansionMedium

SageMaker AI inference expands to G7e instances in new regions

Amazon SageMaker AI inference now offers G7e instances in Seoul, London, and Tokyo regions, supporting large LLM inference and high-memory workloads

Amazon SageMaker AI inference now offers EC2 G7e instances in Asia Pacific (Seoul), Europe (London), and Asia Pacific (Tokyo) regions. G7e instances feature up to eight NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs with 96GB of memory per GPU, 5th Generation Intel Xeon processors, and up to 1,600 Gbps of Elastic Fabric Adapter networking bandwidth. This delivers up to 2.3x inference performance compared to previous-generation G6e instances. With up to 768GB of total GPU memory on a single instance, G7e instances enable serving medium-to-large language models of up to 70 billion parameters with FP8 precision without multi-node configurations. These instances are well suited for LLM inference, image and video generation, spatial computing, and scientific computing workloads requiring high GPU memory capacity and bandwidth.

Read the original AWS announcement

Related articles and terms

Amazon Elastic Inference[Discontinued] Service that attached fractional GPU accelerators to EC2 and SageMaker for inference. Inferentia (Inf1 / Inf2), GPU instances, and SageMaker AI inference are the replacementsAmazon SageMakerA fully managed platform for building, training, and deploying machine learning models end-to-end, significantly boosting productivity for data scientists and ML engineersBuilding an ML Platform with Amazon SageMaker - From Model Development to DeploymentFrom development in Studio to managed spot training, MLOps with Pipelines, and data drift detection with Model Monitor, this article covers how to integrate the entire ML lifecycle.The Layered Architecture of AWS AI/ML Services - Flexibility Through the Three Tiers of SageMaker, Bedrock, and API ServicesThis article organizes AWS AI/ML services into three layers - SageMaker (full control), Bedrock (managed generative AI), and Rekognition/Comprehend/etc. (API-based) - and explains AWS's flexibility through comparisons with GCP Vertex AI and Azure OpenAI Service, including custom silicon integration.AWS Graviton and Custom Silicon Strategy - How In-House Chips Are Reshaping Cloud EconomicsExplore how AWS's Arm-based Graviton processors and AI-focused custom silicon (Inferentia and Trainium) are transforming cloud cost structures and performance, compared with Azure and GCP approaches.