Leveraging Third-Party Data with AWS Data Exchange - Data Procurement and Subscription Management
This article covers procuring third-party data products through Marketplace and building an automated delivery pipeline to S3. This article also introduces methods for productizing and monetizing your own data.
How Data Exchange Works
AWS Data Exchange is a service that offers 3,500 or more data products and procures / delivers third-party datasets on AWS. It works by having data providers publish data products and data consumers subscribe to them, and it is integrated with AWS Marketplace. Traditionally, procuring third-party data required individual contract negotiations, development of API integrations, and conversion of data formats, but Data Exchange standardizes these. Data is provided in the form of S3 files, APIs, Amazon Redshift tables, and AWS Lake Formation tables, and after subscribing you can access it directly within your own AWS account.
Data Procurement and Automated Ingestion
You search for the dataset you want in the data category of AWS Marketplace and enter into a subscription. Many free datasets are also offered, so you can start with a trial. When a new revision (updated version) of a dataset you subscribe to is published, an event is sent to EventBridge. You can build an automated ingestion pipeline in which a Lambda function receives the event and exports the data of the new revision to S3. Data delivered to S3 can be queried directly with Athena, transformed with Glue ETL and loaded into Redshift, or used as training data for SageMaker.
Publishing and Monetizing Data Products
It is also possible to publish data your company holds on Data Exchange and sell it through AWS Marketplace. A data product consists of datasets, revisions, and assets (the actual files or APIs). For pricing, you can choose between a subscription (monthly/annual) and pay-as-you-go billing per revision. Because AWS Marketplace handles contract management, billing, and payment processing on your behalf, data providers can focus on the quality and updating of their data. Before publishing, you need to perform data quality checks and privacy reviews and verify that no personal information is included.
Data Exchange Pricing
Data Exchange itself is free to use, and the cost is the price of the data products you subscribe to. Data product prices are set by providers, and a wide range is offered, from free datasets to premium data costing several thousand dollars per month. No additional charge is incurred for exporting data to S3, but S3 storage charges are required separately. When you publish products as a data provider, AWS Marketplace fees are deducted as a fixed percentage of sales. A phased approach is recommended: first understand how Data Exchange works with free datasets, and then consider paid subscriptions for data whose business value you have been able to confirm.
Delivery Formats of Data Products
Data products in Data Exchange come in several delivery formats. The format in which you receive data as files via S3 is suited to ingesting datasets that are updated regularly. The format in which you query an API each time is suitable when you want to reference the latest values in real time. There are also formats that can be used directly from a data warehouse. Depending on the format, the ingestion mechanism, the update frequency, and the way the data is built into applications change. Choosing products in a delivery format that is easy to handle, based on the nature of the data you want to use and its compatibility with your company's analytics platform, leads to smooth use.
Data Governance and Quality
When using data procured from outside, it is important to check its quality and terms of use. You determine whether the data's origin, update frequency, and coverage are sufficient for your company's purposes. License terms define the scope of permitted use and whether redistribution is allowed, so you understand and comply with the contract terms. You verify ingested data for missing values and outliers, and check its consistency before matching it against your company's data. For analyses where freshness matters, you keep track of the update timing and avoid making decisions based on stale data. Building quality management of external data into operations is the prerequisite for trustworthy analysis.
Integrating into the Analytics Platform
The value of Data Exchange is that you can smoothly bring procured data into your company's analytics. Products in file format can be configured to be delivered automatically to S3 when they are updated, and you can trigger downstream processing upon arrival. If you register ingested data in a data catalog, you can analyze across your company's data and external data from query engines such as Athena. Combining external information such as market data, demographics, and weather data with your company's data produces insights that cannot be obtained from either alone. Building a pipeline that automates everything from procurement to analysis maximizes the effect of using external data.
Considerations When Providing Your Own Data
With Data Exchange, your company can also become the side that provides the data it holds as products. When turning data into products, you appropriately design subscriber management and access control over who can access what scope. You process and anonymize the data so that the data you provide contains no personal or sensitive information, and you ensure compliance. A mechanism for continuously delivering data updates and clearly stated terms of use are also necessary. While you monetize data as an asset, providing it comes with responsibility, so putting in place a structure that balances maintaining quality with legal compliance is the condition for becoming a trusted provider.
Summary
Data Exchange is a service that standardizes the procurement and delivery of third-party data. It integrates with AWS Marketplace and delivers financial data, weather data, geospatial data, and more directly to S3. You receive automatic updates through dataset subscriptions and can analyze them immediately with Athena or Redshift. You can also use it to monetize your own data as a data provider.
References (Official AWS Resources)
The primary sources for this page are the official AWS website and documentation. Check the official pages below for the latest specifications and pricing.
- AWS Data Exchange official page
- Amazon S3 official page
- Amazon Athena official page
- AWS Documentation (official)
If this page and the official documentation disagree, treat the official documentation as authoritative.