Why AWS APIs Return XML - The Evolution from Query APIs to REST JSON
Explore the historical reasons why S3 and EC2 APIs return XML responses, the differences between Query APIs and REST APIs, the evolution of authentication from Signature V2 to V4, and the complexity that SDKs abstract away.
API Design in 2006 - When XML Was the Standard
When S3 and EC2 launched in 2006, XML was the standard data format in the world of web APIs. SOAP (Simple Object Access Protocol) was the mainstream approach for enterprise web services, and strict type definitions via XML Schema combined with service descriptions via WSDL (Web Services Description Language) were considered "proper" API design. JSON is a lightweight format proposed by Douglas Crockford in the early 2000s, but as of 2006 it was not yet widely adopted and was evaluated as "lightweight but lacking type safety." AWS's early APIs reflect the design philosophy of this era. The EC2 API uses a format called "Query API," where the action name and parameters are specified in HTTP GET request query parameters, and responses are returned in XML. For example, a request to list EC2 instances looks like https://ec2.amazonaws.com/?Action=DescribeInstances&Version=2016-11-15. The S3 API follows a REST style, using HTTP methods (GET, PUT, DELETE) and paths to operate on resources, but responses are in XML.
The Curse of Backward Compatibility - Why XML Cannot Be Retired
One of AWS's most important design principles is that "once an API is published, its compatibility is, as a rule, preserved over the long term." The EC2 Query API has been running continuously since its release in 2006, and the XML response format has not changed. Because millions of customers' applications depend on this API, changing the response format to JSON would be a massive breaking change. AWS solves this problem by adopting JSON for new services while maintaining the APIs of older services as-is. Services launched after 2012 (DynamoDB, Lambda, API Gateway, etc.) almost universally use JSON. However, not every service that uses JSON follows a REST style. DynamoDB's API is not REST that represents resources in the URL path; it is a JSON-RPC style (awsJson1_0 in Smithy's protocol naming) in which the operation to call is specified in a header against a single endpoint. The Content-Type is application/x-amz-json-1.0, and the operation you want to run is specified in the X-Amz-Target header. Lambda's API is REST style, using resource paths and HTTP methods, with a JSON body. Meanwhile, early services like EC2, S3, and SNS still return XML responses. That said, early services are not permanently fixed to XML. SQS added support for the JSON protocol in November 2023, and for new users on the latest AWS SDKs JSON became the default protocol (existing XML-based calls continue to work as before). AWS SDKs absorb these differences internally, so developers using SDKs never need to worry about the distinction between XML and JSON.
Signature V4 - Signing Every Request
Another distinctive feature of AWS APIs is that every request requires a cryptographic signature. The current standard, Signature Version 4 (SigV4), concatenates and hashes the HTTP method, URL, headers, and body of a request, then generates an HMAC-SHA256 signature using the secret access key. This signature is included in the Authorization header and verified on the AWS side. The SigV4 signing process consists of four steps. First, creating the Canonical Request: a normalized string of the HTTP method, URI, query string, and headers. Second, creating the String to Sign: concatenating the date, region, service name, and hash of the canonical request. Third, deriving the signing key: progressively deriving a signing key from the secret access key for each date, region, and service. Fourth, computing the signature: HMAC-SHA256 signing the string to sign with the signing key. Note that because SigV4 is a symmetric-key scheme that derives a signing key per date, region, and service, a signature you generate can only be verified for a single region. Requests that span multiple regions (such as Amazon S3 Multi-Region Access Points) require SigV4a, an extension that uses asymmetric signatures based on elliptic-curve cryptography, and the AWS SDKs and AWS CLI switch to SigV4a automatically for such calls. SigV4a does not derive keys per date or region; it signs with a key pair derived from the secret access key, and the AWS side verifies using only the public key. Implementing this complex process manually is impractical, and AWS SDKs handle it automatically. When calling AWS APIs with curl without using an SDK, you need to implement this signing process yourself, making debugging extremely difficult.
The Complexity SDKs Abstract Away
AWS SDKs are a massive abstraction layer that hides API complexity from developers. The things SDKs handle internally are extensive: request signing (SigV4), XML/JSON serialization and deserialization, retry logic (with exponential backoff), error handling (distinguishing transient from persistent errors), pagination (automatically fetching large result sets across multiple requests), region endpoint resolution, and automatic refresh of temporary credentials (IAM role AssumeRole). SDK v3 (JavaScript) and boto3 (Python) implement these processes as middleware pipelines, and developers can add custom middleware as well. The SDK's retry logic is particularly important. AWS APIs apply throttling (rate limiting), and sending too many requests in a short period causes calls to be rejected. The error code returned in that case differs by service. The EC2 API returns RequestLimitExceeded when the request count limit is exceeded, while many services use ThrottlingException. The HTTP status code also differs by service and operation, so an implementation that detects throttling by looking at a single status code alone should be avoided. SDKs automatically distinguish these throttling errors from transient server-side failures (5xx) and retry with exponential backoff plus jitter (random variation). The base wait time is on the order of milliseconds, and the maximum number of retries and the maximum wait time are also defined as SDK defaults, so developers do not need to implement retry logic themselves.
API Evolution - From Query APIs to GraphQL and Event-Driven
AWS API design has evolved significantly from the launch of S3 and EC2 in 2006 to the present. From the early Query API + XML, to REST + JSON, and on to GraphQL with AWS AppSync, which became generally available in 2018, and event-driven architectures with EventBridge, the API paradigms themselves have diversified. What has remained consistent in AWS API design is the stance that "once an API is published, its compatibility is, as a rule, preserved over the long term." New features are delivered as new API versions or additional parameters in a way that does not break existing calls, and old versions remain usable as they are for a long time. This maintenance of backward compatibility is fundamental to AWS's reliability. Systems that enterprises spent years building essentially do not suddenly stop working due to API changes. However, this is not a guarantee that an API will "never be retired." As a real example, the S3 SOAP API was no longer offered to new users and reached End of Life on August 31, 2025. Long-term compatibility is a principle, and exceptions with advance notice and a migration period do exist. On the other hand, this principle also creates technical debt. Design decisions from 2006 (XML responses, Query API format) are still maintained today, leading new developers to wonder "why such an old format?" The answer is "for backward compatibility," which is a manifestation of AWS's commitment to prioritizing customer trust.
References (Official AWS Resources)
The primary sources for this page are the official AWS website and documentation. Check the official pages below for the latest specifications and pricing.
If this page and the official documentation disagree, treat the official documentation as authoritative.