Offline Data Transfer - Large-Scale Cloud Migration with AWS Snow Family
Learn about offline transfer of large-scale data using AWS Snow Family (Snowcone, Snowball Edge, Snowmobile) and hybrid data migration strategies combining Snow Family with AWS DataSync. Practical approaches for petabyte-scale data migration.
Challenges of Large-Scale Data Transfer and the Role of Snow Family
When migrating petabyte-scale data to the cloud, network-based transfers can take weeks to months due to bandwidth constraints. Even with a dedicated 1 Gbps line, transferring 1 PB of data takes approximately 100 days. AWS Snow Family is a physical device-based offline data transfer service that enables rapid cloud migration of large-scale data without depending on network bandwidth. Snowcone is a compact device with 8 TB (HDD) or 14 TB (SSD) capacity, ideal for data collection at remote sites and edge environments. Snowball Edge offers 80 TB (Storage Optimized) or 42 TB (Compute Optimized) capacity and provides edge computing capabilities in addition to data transfer. Snowmobile is a container truck with 100 PB capacity, designed for exabyte-scale data center migrations.
Choosing a Snow Family Device and the Transfer Workflow
The data transfer workflow begins by creating a job from the AWS Console and having the device shipped to you. You can create a job with the following CLI command: ```bash aws snowball create-job \ --job-type IMPORT \ --resources '{"S3Resources":[{"BucketArn":"arn:aws:s3:::my-migration-bucket"}]}' \ --snowball-type EDGE \ --shipping-option SECOND_DAY ``` Once you receive the device, connect it to your local network and copy data using the Snowball client or S3-compatible API. Snowball Edge also provides NFS mount points, enabling seamless data transfer from existing backup and file copy tools. After data copying is complete, ship the device back to AWS, and AWS imports the data into your specified S3 bucket. Data in transit is protected with 256-bit encryption, and the device's physical security is also ensured. You can scale out transfer throughput by using multiple devices in parallel. For a 10 PB data migration, using multiple Snowball Edge devices simultaneously can complete the migration in a few weeks.
Hybrid Migration Strategy Combining DataSync
AWS DataSync is a service that automates data transfer between on-premises storage and AWS storage services, and a hybrid migration strategy combining it with Snow Family is highly effective. The initial large-scale data migration is performed with Snow Family, followed by continuous incremental data synchronization via DataSync over the network. DataSync supports diverse protocols including NFS, SMB, HDFS, and object storage, and supports transfers to S3, EFS, and FSx for Windows File Server. Data integrity is guaranteed through in-transit data verification (checksum comparison). Bandwidth throttling settings allow you to control network load during business hours while running transfers in the background. DataSync excels in supporting a wide range of protocols including Linux-based storage environments. Scheduled execution automates periodic incremental synchronization, keeping data between on-premises and cloud consistently up to date.
Edge Computing and Data Processing
Snowball Edge Compute Optimized provides edge computing capabilities in addition to data transfer. You can run EC2-compatible instances and Lambda functions on the device, performing data preprocessing, filtering, and aggregation at the edge before transferring to the cloud. It can be used for IoT sensor data collection and preprocessing, video data transcoding, and machine learning model inference in environments with limited network connectivity. Snowcone supports AWS IoT Greengrass and can be permanently deployed at remote sites as an edge device. You can also build workflows that run a DataSync agent on Snowcone to automatically transfer collected data to S3 over the network. Snow Family integrates data transfer and edge computing in a single device, simplifying operations and reducing costs. For a comprehensive look at cloud migration strategies, technical books (Amazon) are a useful reference.
Security and Compliance
Snow Family devices are equipped with enterprise-grade security features. Data at rest is protected with 256-bit encryption (AES-256), and encryption keys are managed by AWS KMS. A Trusted Platform Module (TPM) detects physical tampering, automatically locking the device if unauthorized access is attempted. GPS tracking enables real-time monitoring of device shipping status. The E Ink display is used for automatic shipping label updates, reducing the effort required for return shipping. The devices comply with HIPAA, PCI DSS, FedRAMP, and other compliance requirements, making them suitable for data migration in regulated industries. After data import is complete, data on the device is fully erased following NIST 800-88 compliant media sanitization procedures. IAM policies provide fine-grained control over job creation and device access permissions, enabling operations that comply with organizational security policies.
Snow Family Pricing
Snowcone costs approximately $60 per day, and Snowball Edge Storage Optimized costs approximately $300 per day. Shipping fees are charged separately. The standard usage period includes 10 days, with additional days billed at a daily rate. Importing to S3 is free. For migrating 100 TB of data with Snowball Edge, the combined device and shipping costs are approximately $5,000-$6,000, which is less expensive than internet-based data transfer fees (approximately $9,000).
Summary - Choosing a Large-Scale Data Migration Strategy
AWS Snow Family is a physical device-based solution for migrating large-scale data to the cloud beyond network bandwidth constraints. Three form factors - Snowcone, Snowball Edge, and Snowmobile - support data migrations ranging from terabyte to exabyte scale. Combined with DataSync, you can implement a hybrid migration strategy that integrates initial large-scale migration with continuous incremental synchronization. 256-bit encryption, TPM-based tamper detection, and NIST 800-88 compliant data erasure ensure enterprise-grade security.