Leveraging Third-Party Data with AWS Data Exchange - Data Procurement and Subscription Management
Procure third-party data products via Marketplace and build automated delivery pipelines to S3. This article also covers how to productize and monetize your own data.
How Data Exchange Works
AWS Data Exchange offers over 3,500 data products and is a service for procuring and delivering third-party datasets on AWS. Data providers publish data products and data consumers subscribe to them, with integration into AWS Marketplace. Traditionally, procuring third-party data required individual contract negotiations, API integration development, and data format conversions, but Data Exchange standardizes all of these. Data is available in S3 file, API, Amazon Redshift table, and AWS Lake Formation table formats, and after subscribing, you can access it directly within your own AWS account.
Data Procurement and Automated Ingestion
Search for your desired dataset from the data categories in AWS Marketplace and subscribe. Many free datasets are available, so you can start with a trial. When a new revision (updated version) of a subscribed dataset is published, an event is sent to EventBridge. You can build an automated ingestion pipeline by receiving the event with a Lambda function and exporting the new revision data to S3. Data delivered to S3 can be queried directly with Athena, transformed with Glue ETL and loaded into Redshift, or used as training data for SageMaker.
Publishing and Monetizing Data Products
You can also publish your own data on Data Exchange and sell it through AWS Marketplace. Data products consist of datasets, revisions, and assets (actual files or APIs). Pricing options include subscription (monthly/annual) or per-revision pay-as-you-go. AWS Marketplace handles contract management, billing, and payment processing, so data providers can focus on data quality and updates. Before publishing, you need to perform data quality checks and privacy verification to ensure no personal information is included. To comprehensively learn Data Exchange design patterns, refer to technical books on Amazon.
Data Exchange Pricing
There is no charge for using Data Exchange itself; the cost is the price of the data products you subscribe to. Data product prices are set by providers and range from free datasets to premium data costing thousands of dollars per month. There are no additional charges for exporting data to S3, but S3 storage charges apply separately. When publishing products as a data provider, an AWS Marketplace fee is deducted as a percentage of revenue. A phased approach is recommended: start with free datasets to understand how Data Exchange works, then consider paid subscriptions for data with confirmed business value.
Summary
Data Exchange is a service that standardizes the procurement and delivery of third-party data. Integrated with AWS Marketplace, it delivers financial data, weather data, geospatial data, and more directly to S3. Subscribe to datasets to receive automatic updates and analyze them immediately with Athena or Redshift. You can also monetize your own data as a data provider.