Record Matching with AWS Entity Resolution - Customer Data Deduplication and Integration
Learn about record matching across multiple data sources with Entity Resolution and how to design matching workflows.
Entity Resolution Overview
Entity Resolution is a service that matches and links records distributed across multiple data sources to build a unified entity view, processing up to 20 million records per workflow. It automatically links records of the same customer scattered across CRM, e-commerce, and support systems. It provides two matching methods, rule-based and ML-based, handling name variations and address abbreviations. Input data can be specified as tables registered in AWS Glue Data Catalog or files on S3, and output is written as match result files to S3. The service is fully managed with no infrastructure to maintain, and processing scales automatically.
Matching Methods
Rule-based matching uses explicit rules such as exact matches on email addresses or phone numbers. Multiple rules can be combined with OR conditions, enabling flexible definitions like "email matches OR phone number and last name match." Within each rule, multiple fields are combined with AND conditions to fine-tune precision. ML-based matching provides flexible matching that accounts for name variations, address abbreviations, and phone number format differences. It uses models pre-trained by AWS, so customers do not need to prepare their own training data. A staged approach combining both methods optimizes the balance between accuracy and cost. An effective design processes high-confidence matches with rule-based matching first, then handles the remainder with ML.
Workflows and ID Mapping
Matching workflows take data sources (S3 or Glue tables) as input and output matching results to S3. Schema mapping maps input data columns to Entity Resolution standard fields (name, address, phone number, email address). ID mapping workflows integrate with third-party data providers (LiveRamp, TransUnion) to match your customer IDs against external ID graphs and generate unified IDs. Matching results include match IDs, confidence scores, and matched record pairs, which can be integrated into downstream analytics and marketing systems. To gain a deeper understanding of Entity Resolution analysis methods, specialized books (Amazon) can be helpful.
Use Cases
The primary use case for Entity Resolution is Customer Data Integration (CDI). It links customer records scattered across e-commerce purchase histories, customer support inquiries, and email marketing lists with a unified ID to build a 360-degree customer view. In advertising, it matches first-party data with ad platform audience data through ID mapping to enable cross-channel attribution analysis. In healthcare, it improves care continuity by deduplicating patient records distributed across multiple medical institutions. Financial institutions use it for KYC (Know Your Customer) processes to identify the same person across multiple databases. Combined with AWS Clean Rooms, you can build a "data clean room" pattern that performs joint matching analysis across organizations without directly sharing data.
Design Best Practices and Pitfalls
In schema mapping design, accurately mapping input data columns to Entity Resolution standard fields is key to precision. Splitting addresses into separate fields (state, city, street) yields higher accuracy than mapping a single combined field. A common pitfall is insufficient data cleansing before matching. Inconsistencies like full-width vs. half-width characters, mixed old and new character forms, and phone numbers with or without hyphens should be normalized beforehand. Records containing NULL values or empty strings significantly degrade matching accuracy and should be excluded or flagged in preprocessing. For ML-based matching, confidence score threshold settings are critical - too low increases false positives, too high increases false negatives. For large-scale data, initial full-match execution times can be lengthy, so splitting data across parallel workflows and merging results afterward is an effective design pattern.
Entity Resolution Pricing
Entity Resolution pricing is based on the number of records processed for matching. Rule-based matching costs approximately 0.25 USD per 1,000 records, and ML-based matching costs approximately 0.75 USD. ID mapping incurs additional per-provider charges. While the initial matching processes all records, leveraging incremental matching (new and updated records only) reduces the cost of periodic runs. Performing data cleansing (normalizing notation, pre-eliminating obvious duplicates) before matching reduces the number of processed records and optimizes costs. If you exceed the 20 million records per workflow limit, you need to split data sources and execute across multiple workflows.
Summary
Entity Resolution is a service that matches and integrates records from multiple data sources to build a unified customer view. A staged approach that processes high-confidence matches with rule-based matching and handles name variations and address abbreviations with ML-based matching is effective. ID mapping enables integration with external data providers, and incremental matching optimizes the cost of periodic runs. Proper schema mapping design and input data preprocessing are key to improving accuracy, and integration with Clean Rooms enables cross-organization data collaboration.