Amazon MWAA
A fully managed Apache Airflow service that orchestrates data pipelines using DAGs
Overview
Amazon MWAA (Managed Workflows for Apache Airflow) is a fully managed workflow orchestration service built on Apache Airflow. It eliminates the need to manage Airflow's scheduler, worker, and web server infrastructure, letting you focus on DAG (Directed Acyclic Graph) development and data pipeline operations. In addition to native integration with AWS services like Glue, EMR, Lambda, ECS, Redshift, and Athena, it supports third-party service integration through Airflow's provider packages.
Environment Classes and Worker Scaling
MWAA environments are available in classes such as mw1.micro, mw1.small, mw1.medium, mw1.large, mw1.xlarge, and mw1.2xlarge (as of October 2026). According to the user guide, the class determines the size of the Fargate containers where the Celery Executor runs and of the Aurora PostgreSQL metadata database. The same guide lists the DAG capacity and the default number of concurrent tasks for each class. Except for mw1.micro, workers support autoscaling - set minimum and maximum worker counts, and they automatically scale out and in based on the number of running and queued tasks. For predictable large batch processing, one approach is to raise the minimum worker count in advance. Environment class changes are made through an update operation. According to the user guide, forced updates, the default strategy, immediately stop all active workers and cause running tasks to fail, so schedule updates within a maintenance window.
DAG Deployment and S3 Sync Mechanics
MWAA DAG files are placed under a designated prefix in an S3 bucket. Python files uploaded to S3 are synced to the scheduler and worker containers every 30 seconds, automatically parsed and registered as DAGs. DAG file updates are similarly reflected just by uploading to S3, so deployment pipelines only need to incorporate S3 sync. One example workflow is running DAG syntax checks and unit tests in CodePipeline and CodeBuild, then deploying to S3 after tests pass. Placing a requirements.txt in S3 triggers pip install during environment updates, installing additional Python packages. However, requirements.txt changes require an environment update, so unlike DAG file updates, they don't take effect immediately. When using external libraries in DAGs, pin versions in requirements.txt and verify functionality after the environment update. A startup.sh script lets you define custom initialization processing at environment startup.
Plugin Management and VPC Network Design
MWAA custom plugins are uploaded to S3 as plugins.zip. Plugins can include custom Airflow operators, sensors, hooks, and web UI extensions. Packaging shared database connection logic or custom notification processing as plugins enables reuse across multiple DAGs. Plugin updates require an environment update, so like requirements.txt, apply them during a maintenance window. MWAA environments must be created within a VPC and require two private subnets in different Availability Zones. Web server access modes include public, private, and both public and private (as of October 2026). Public mode allows access to the Airflow UI over the internet, while private mode restricts access to within the VPC only. Worker communication with AWS services (S3, Glue, Redshift, etc.) requires VPC endpoints or a NAT gateway. From a cost optimization perspective, configuring VPC endpoints for frequently accessed S3 and CloudWatch Logs is one design option for reducing NAT gateway data processing charges.
References (Official AWS Resources)
The primary sources for this page are the official AWS website and documentation. Check the official pages below for the latest specifications and pricing.
- AWS Glue official page
- AWS Step Functions official page
- Amazon EMR official page
- AWS Documentation (official)
If this page and the official documentation disagree, treat the official documentation as authoritative.