Accelerating Data Transfer with AWS DataSync - Migration from On-Premises to S3 and EFS
Automate high-speed data transfer from on-premises to S3 and EFS. This guide covers agent deployment, task scheduling, and transfer data integrity verification.
DataSync Overview
DataSync is a service that automates and accelerates data transfer between on-premises and AWS. The transfer protocol uses proprietary optimizations. It supports transfers from on-premises NFS/SMB file servers to S3, EFS, FSx for Windows, FSx for Lustre, and FSx for OpenZFS. HDFS (Hadoop Distributed File System) data migration is also supported, making it suitable for migrating big data platforms to the cloud.
Agent and Task Design
Deploy a DataSync agent (a virtual machine on VMware ESXi, Hyper-V, or KVM) on-premises and connect it to the source storage. The agent VM has CPU and memory requirements. Additional agents can also be deployed for scale-out. In the task, specify the source location (on-premises NFS) and destination location (S3 bucket), and configure transfer options (file metadata preservation, exclude filters, bandwidth throttle). The metadata preservation option maintains POSIX permissions, timestamps, and UID/GID at the destination. Schedule execution to automate daily differential transfers and continuously synchronize data between on-premises and AWS. After transfer completion, data integrity is automatically verified, and any checksum mismatches are reported.
Inter-AWS Transfer and Scheduling
DataSync also supports data transfer between AWS services. You can execute transfers between S3 buckets, between EFS file systems, between FSx file systems, and from S3 to EFS without an agent. It supports cross-account and cross-region transfers, making it useful for data replication to DR environments. Task schedules automate periodic transfers, synchronizing data through daily or weekly incremental transfers. Transfer filters narrow down targets based on file name patterns or modification dates, avoiding unnecessary data transfers. EventBridge integration enables building automated workflows triggered by task completion or failure, such as launching Glue jobs, executing Lambda functions, or sending SNS notifications. You can monitor the status of task executions with CloudWatch.
Comparison with Other Data Transfer Methods
AWS offers multiple data transfer options, which you choose between based on data volume and requirements. DataSync is suited for continuous synchronization and migration over the network, with incremental transfer of only changed data and automatic integrity verification. AWS Transfer Family (SFTP/FTPS/FTP) is suited for partner integrations and file exchange with legacy systems, enabling cloud migration without changing existing SFTP workflows, but is not designed for bulk data migration. S3 Replication (CRR/SRR) provides object-level replication between buckets that copies automatically on write, but cannot be used for on-premises-to-AWS transfers. AWS Snow Family devices are no longer available for new customers to order (customers currently using Snowball Edge are not affected). New customers are directed to DataSync for online transfers.
Design Best Practices and Considerations
Key points to consider when deploying DataSync. First, bandwidth throttle configuration. When running DataSync on a network shared with production traffic, limit bandwidth during business hours and schedule full-speed transfers overnight to limit the impact on business traffic. Second, choosing the destination S3 storage class. DataSync can transfer data directly into a specific S3 storage class, and placing infrequently accessed archive data in S3 Glacier Instant Retrieval keeps storage costs down. Third, when transferring large numbers of small files (millions of files), per-task overhead increases, so splitting tasks by directory and running them in parallel makes it easier to raise throughput. Fourth, security and compliance. Data in transit is encrypted with TLS, but DataSync can't access S3 buckets encrypted with SSE-C. Using VPC endpoints (PrivateLink) ensures agent-to-AWS communication does not traverse the public internet. IAM policies provide fine-grained control over task execution permissions, and CloudTrail records API calls. DataSync is compliant with HIPAA, PCI DSS, and SOC 1/2/3, enabling use in regulated industries.
DataSync Pricing
DataSync pricing depends on factors such as the volume of data copied. Transferring from on-premises requires an agent execution environment (VM), but the agent itself incurs no additional charges. Storage costs at the destination (S3, EFS, FSx) are charged separately. Using differential transfer to transfer only changed files keeps data volume down. Setting bandwidth throttle does not affect pricing based on data volume.
Summary
DataSync is a service that accelerates and automates data transfer from on-premises to AWS and between AWS services. Differential transfer efficiently synchronizes only changed files. It supports diverse sources including HDFS and secure transfers via VPC endpoints. Bandwidth throttle lets you adapt transfers to your migration scenario.
References (Official AWS Resources)
The primary sources for this page are the official AWS website and documentation. Check the official pages below for the latest specifications and pricing.
- AWS DataSync official page
- Amazon S3 official page
- Amazon EFS official page
- AWS Documentation (official)
If this page and the official documentation disagree, treat the official documentation as authoritative.