Accelerating Data Transfer with AWS DataSync - Migration from On-Premises to S3 and EFS
Automate high-speed data transfer from on-premises to S3 and EFS. This guide covers agent deployment, task scheduling, and transfer data integrity verification.
DataSync Overview
DataSync is a service that automates and accelerates data transfer between on-premises and AWS, achieving throughput of up to 10 Gbps per task. It delivers transfer speeds up to 10 times faster than rsync or robocopy, maximizing network bandwidth utilization. The transfer protocol uses proprietary optimizations including TCP window scaling and parallel stream transfers to absorb WAN latency. It supports transfers from on-premises NFS/SMB file servers to S3, EFS, FSx for Windows, FSx for Lustre, and FSx for OpenZFS. HDFS (Hadoop Distributed File System) data migration is also supported, making it suitable for migrating big data platforms to the cloud.
Agent and Task Design
Deploy a DataSync agent (a virtual machine on VMware ESXi, Hyper-V, or KVM) on-premises and connect it to the source storage. The recommended resources for the agent VM are 4 or more vCPUs and 16 GB or more of RAM. A single agent can run multiple tasks in parallel, and additional agents can be deployed for scale-out. In the task, specify the source location (on-premises NFS) and destination location (S3 bucket), and configure transfer options (file metadata preservation, exclude filters, bandwidth throttle). The metadata preservation option maintains POSIX permissions, timestamps, and UID/GID at the destination, allowing file server access controls to be reflected at the migration target. Schedule execution to automate daily differential transfers and continuously synchronize data between on-premises and AWS. After transfer completion, data integrity is automatically verified, and any checksum mismatches are reported.
Inter-AWS Transfer and Scheduling
DataSync also supports data transfer between AWS services. You can execute transfers between S3 buckets, between EFS file systems, between FSx file systems, and from S3 to EFS without an agent. It supports cross-account and cross-region transfers, making it useful for data replication to DR environments. Task schedules automate periodic transfers, synchronizing data through daily or weekly incremental transfers. Transfer filters narrow down targets based on file name patterns or modification dates, avoiding unnecessary data transfers. EventBridge integration enables building automated workflows triggered by task completion or failure, such as launching Glue jobs, executing Lambda functions, or sending SNS notifications. Task execution reports can be output to CloudWatch Logs, enabling monitoring of transferred file counts, bytes, elapsed time, and error details. For understanding DataSync best practices, related books (Amazon) can be helpful.
Comparison with Other Data Transfer Methods
AWS offers multiple data transfer options, and choosing the right one based on data volume and requirements is important. DataSync is optimal for continuous synchronization and migration over the network, with incremental transfer of only changed data and automatic integrity verification. AWS Transfer Family (SFTP/FTPS/FTP) is suited for partner integrations and file exchange with legacy systems, enabling cloud migration without changing existing SFTP workflows, but is not designed for bulk data migration. S3 Replication (CRR/SRR) provides object-level replication between buckets that copies automatically on write, but cannot be used for on-premises-to-AWS transfers. AWS Snow Family (Snowball Edge, Snowcone) is used when network bandwidth is limited or for shipping PB-scale data on physical devices. A common migration pattern is transferring the initial bulk data with Snowball and then synchronizing subsequent changes with DataSync.
Design Best Practices and Considerations
Key points to consider when deploying DataSync. First, bandwidth throttle configuration. When running DataSync on a network shared with production traffic, limit bandwidth during business hours and schedule full-speed transfers overnight to avoid impacting business traffic. Second, choosing the destination S3 storage class. DataSync can specify the storage class for placed objects; infrequently accessed archive data can be placed directly in S3 Glacier Instant Retrieval to reduce costs. Third, when transferring large numbers of small files (millions of files), per-task overhead increases, so splitting tasks by directory and running them in parallel maximizes throughput. Fourth, security and compliance. Data in transit is encrypted with TLS 1.2, and destination S3 supports SSE-S3, SSE-KMS, or SSE-C server-side encryption. Using VPC endpoints (PrivateLink) ensures agent-to-AWS communication does not traverse the public internet. IAM policies provide fine-grained control over task execution permissions, and CloudTrail records all API calls. DataSync is compliant with HIPAA, PCI DSS, and SOC 1/2/3, enabling use in regulated industries.
DataSync Pricing
DataSync pricing is based on the volume of data copied, at approximately $0.0125 per GB. Transferring from on-premises requires an agent execution environment (VM), but the agent itself incurs no additional charges. Inter-AWS transfers follow the same per-data-volume pricing. Storage costs at the destination (S3, EFS, FSx) are charged separately. Optimize costs by using differential transfer to transfer only changed files and minimize data volume. Setting bandwidth throttle does not affect data volume pricing, so transfer speed and cost are not a tradeoff; cost is proportional only to the bytes transferred. For large initial transfers, consider combining with AWS Snowball.
Summary
DataSync is a service that accelerates and automates data transfer from on-premises to AWS and between AWS services. It maximizes WAN bandwidth utilization through CBT and parallel streams, efficiently synchronizing only changed files via differential transfer. It supports diverse sources including HDFS and secure transfers via VPC endpoints. Combining with Snow Family or leveraging bandwidth throttle provides flexibility for various migration scenarios.