Workflow Management - Running Apache Airflow as a Managed Service with Amazon MWAA
Learn about data pipeline orchestration with Amazon MWAA (Managed Workflows for Apache Airflow). This article covers setup, DAG management, and when to choose Step Functions instead.
Where MWAA Fits In and Airflow's Challenges
Apache Airflow is the de facto standard for data pipeline orchestration. It lets you define DAGs (Directed Acyclic Graphs) in Python, declaratively managing task dependencies, schedules, retries, and alerts. However, self-hosting Airflow requires building and operating a web server, scheduler, workers, and metadata database (PostgreSQL), with significant overhead for version upgrades and scaling. Amazon MWAA (Managed Workflows for Apache Airflow), released in 2020, is a fully managed Airflow service where AWS handles environment provisioning, patching, scaling, and availability. Users simply upload DAG files and a requirements.txt to an S3 bucket, and the Airflow environment is automatically configured. Because MWAA uses Python code to define DAGs, it integrates naturally with Git version control, code reviews, and unit testing, bringing software engineering practices to data pipelines.
Environment Setup and DAG Deployment
Creating an MWAA environment requires an S3 bucket for DAGs, a VPC (with two or more private subnets), and an execution role (IAM). Environment classes come in five tiers: mw1.small, mw1.medium, mw1.large, mw1.xlarge, and mw1.2xlarge, differing in concurrent task capacity and scheduler performance. mw1.small is suited for small validation environments (approximately $0.49 USD/hour), while mw1.large targets production workloads (approximately $1.96 USD/hour). ```python # DAG example: Glue job ->Athena query ->SNS notification from airflow import DAG from airflow.providers.amazon.aws.operators.glue import GlueJobOperator from airflow.providers.amazon.aws.operators.athena import AthenaOperator from airflow.providers.amazon.aws.operators.sns import SnsPublishOperator from datetime import datetime with DAG('daily_etl', start_date=datetime(2026, 1, 1), schedule='@daily') as dag: extract = GlueJobOperator(task_id='run_glue', job_name='raw-to-clean') query = AthenaOperator(task_id='run_query', query='SELECT count(*) FROM clean.events', database='clean', output_location='s3://results/') notify = SnsPublishOperator(task_id='notify', target_arn='arn:aws:sns:ap-northeast-1:123:alerts', message='ETL completed') extract >> query >> notify ``` When you upload a DAG file to S3, MWAA automatically detects it and registers it with the scheduler. Additional Python packages are specified in requirements.txt, and plugins are placed in S3 as plugins.zip.
AWS Service Integration and Operations
MWAA provides a rich set of integration Operators for AWS services through the apache-airflow-providers-amazon package. It covers nearly every service needed for data pipelines: Glue (ETL job execution), Athena (SQL queries), EMR (Spark jobs), Redshift (data warehouse queries), Lambda (function invocation), ECS (container task execution), SageMaker (ML training/inference), and Step Functions (state machine execution). Worker Auto Scaling automatically adjusts the number of workers based on task queue depth. You can set minimum and maximum worker counts, scaling out only during peaks to optimize costs. Monitoring is available through both CloudWatch metrics (DAG processing time, task success/failure counts, queue depth) and the Airflow Web UI. The web server can be configured for public access (with IAM authentication) or private access (VPC only). For a comprehensive study of data pipeline architecture, refer to technical books (Amazon).
Choosing Between MWAA and Step Functions
MWAA and Step Functions are both workflow orchestration services, but they differ in design philosophy and strengths. Step Functions is event-driven, responding to triggers from API Gateway or EventBridge with low latency (milliseconds). It defines state transitions in JSON (ASL) and is ideal for serverless workflows combining Lambda, ECS, Bedrock, and more. Pricing is per state transition (Standard: $0.025 per 1,000 transitions), making it well-suited for short-duration, high-frequency workflows. MWAA, on the other hand, excels at schedule-driven batch pipelines. It handles complex dependencies (DAGs with tens to hundreds of tasks), retry policies, SLA monitoring, and backfill (re-running for past dates) - requirements specific to data engineering. The ability to migrate existing Airflow assets (DAGs, plugins, custom Operators) as-is is also a major advantage. As a rule of thumb, choose Step Functions for real-time processing and API integrations, and MWAA for daily or hourly data batch processing.
MWAA Pricing
MWAA pricing consists of hourly charges for the environment class and workers. An mw1.small environment costs approximately $0.49/hour (about $353/month). Additional workers cost approximately $0.055/hour for mw1.small. Since the environment runs 24/7, the minimum monthly cost is approximately $353. When comparing with self-hosted Airflow (EC2 + RDS + Redis), evaluate the total cost of ownership including operational cost savings (patching, scaling, monitoring). If DAG execution frequency is low, Step Functions may be more cost-effective.
Summary - Guidelines for Using MWAA
Amazon MWAA is a managed Apache Airflow service that delivers data pipeline orchestration without operational overhead. Python-based DAG definitions integrate naturally with version control and testing, and rich Operator integrations with AWS services enable efficient construction of ETL, analytics, and ML pipelines. While Step Functions is suited for event-driven serverless workflows, MWAA is optimal for schedule-driven batch processing with complex dependencies. As a migration target for existing Airflow environments and a productivity tool for data engineering teams, MWAA is a compelling choice.