AWS DataSync
A service that automates and accelerates data transfers between on-premises and AWS or between AWS services, supporting scheduled transfers across NFS/SMB/HDFS/S3/EFS/FSx
Overview
AWS DataSync is a service that automates data transfers between on-premises storage systems and AWS storage services, or between AWS storage services. It supports transfers from NFS, SMB, HDFS, and object storage to S3, EFS, FSx for Windows File Server, FSx for Lustre, FSx for OpenZFS, and FSx for NetApp ONTAP. A purpose-built network protocol delivers transfer speeds up to 10x faster than open-source tools. Data verification during transfer, encryption, bandwidth throttling, and scheduled execution are included as standard features.
Agent Placement and Network Design
For data transfers from on-premises to AWS, you need to deploy a DataSync agent in your on-premises environment. The agent can run as a virtual machine on VMware ESXi, Microsoft Hyper-V, or KVM, or as an Amazon EC2 instance. A Basic mode agent VM requires at least 4 vCPUs and 32 GB RAM, and 64 GB RAM for task executions working with more than 20 million files or directories (as of October 2026). The agent communicates with DataSync through one of four service endpoint types (public, FIPS, VPC with PrivateLink, or FIPS VPC), and on-premises storage reaches a VPC endpoint over Direct Connect or VPN. You can set a bandwidth limit for each task execution and change it while the execution is running. For transfers between AWS storage services in the same account (e.g., S3 to EFS), no agent is required - DataSync handles the transfer in a fully managed fashion.
Transfer Task Design and Data Integrity Guarantees
A DataSync task consists of three elements: a source location, a destination location, and transfer settings. Transfer settings let you configure data filtering (include/exclude patterns), file metadata preservation (POSIX permissions, timestamps, ownership), and overwrite policies (transfer only changed files vs. all files). DataSync verifies data integrity with checksums during the transfer, and each task lets you choose the additional verification at the end of the transfer (only transferred data, all data, or none). For large-scale transfers of hundreds of millions of files, task execution may take hours to days, but if interrupted, the next run transfers only the delta - no need to start over. With scheduled execution, you can automate daily or weekly periodic syncs (the minimum interval is 1 hour).
Design Patterns by Migration Scenario
DataSync has three typical use cases. First, migrating on-premises NFS/SMB file servers to S3 or EFS. After the initial full copy, incremental syncs continue until cutover, at which point the application's connection target is switched. When file counts exceed tens of millions, splitting tasks by directory and running them in parallel can reduce transfer time. Second, continuous data synchronization in hybrid cloud environments. A representative pattern is periodically transferring logs or sensor data generated on-premises to S3 for analysis with Athena or EMR. Third, data transfers across AWS regions or accounts. For replicating data to an S3 bucket in another region for disaster recovery, DataSync is effective for file system data (EFS, FSx) that S3 cross-region replication cannot handle. Transfer Family (SFTP/FTPS) is used to exchange files with external partners, while DataSync focuses on bulk data transfer and automation between storage systems.
References (Official AWS Resources)
The primary sources for this page are the official AWS website and documentation. Check the official pages below for the latest specifications and pricing.
If this page and the official documentation disagree, treat the official documentation as authoritative.