Workflow Management - Running Apache Airflow as a Managed Service with Amazon MWAA

Where MWAA Fits and the Challenges of Airflow

Apache Airflow is the de facto standard in data pipeline orchestration. You define DAGs (directed acyclic graphs) in Python and can declaratively manage dependencies between tasks / schedules / retries / alerts. However, running Airflow yourself requires building and operating a web server / scheduler / workers / metadata DB (PostgreSQL), and there was the challenge that the burden of version upgrades and scaling was heavy. Amazon MWAA (Managed Workflows for Apache Airflow) is a fully managed Airflow service released in 2020, and AWS manages environment building / patching / scaling / ensuring availability. Users simply upload DAG files and requirements.txt to an S3 bucket, and the Airflow environment is configured automatically. Because MWAA defines DAGs in Python code, it has high affinity with version control through Git, code reviews, and unit tests, and you can apply software engineering practices to data pipelines. MWAA also has two deployment options: environment-based deployment, which keeps the Airflow environment running at all times, and MWAA Serverless, which charges only for workflow execution time (as of August 2026).

Building the Environment and Deploying DAGs

Creating an MWAA environment requires an S3 bucket to store DAGs, a VPC (two or more private subnets), and an execution role (IAM). The environment classes you can choose in environment-based deployment come in 6 tiers, mw1.micro / mw1.small / mw1.medium / mw1.large / mw1.xlarge / mw1.2xlarge, and they differ in the number of tasks that can run concurrently and in scheduler performance. As of August 2026, the unit prices are the same in the US East (N. Virginia) Region and the Tokyo Region. The smallest configuration, mw1.micro, is $0.29/hour, mw1.small, used for small to medium-sized test environments, is $0.49/hour, and mw1.large, for production workloads, is $0.99/hour.

# Example DAG: Glue job -> Athena query -> SNS notification
from airflow import DAG
from airflow.providers.amazon.aws.operators.glue import GlueJobOperator
from airflow.providers.amazon.aws.operators.athena import AthenaOperator
from airflow.providers.amazon.aws.operators.sns import SnsPublishOperator
from datetime import datetime

with DAG('daily_etl', start_date=datetime(2026, 1, 1), schedule='@daily') as dag:
 extract = GlueJobOperator(task_id='run_glue', job_name='raw-to-clean')
 query = AthenaOperator(task_id='run_query', query='SELECT count(*) FROM clean.events',
 database='clean', output_location='s3://results/')
 notify = SnsPublishOperator(task_id='notify', target_arn='arn:aws:sns:ap-northeast-1:123:alerts',
 message='ETL completed')
 extract >> query >> notify

When you upload a DAG file to S3, MWAA automatically detects it and registers it with the scheduler. Additional Python packages are specified in requirements.txt, and plugins are placed in S3 as plugins.zip. The Airflow version is specified when the environment is created, and if you do not specify it, the environment is created with the latest version that MWAA supports. The versions listed in the official documentation as of August 2026 are v3.2.1 / v3.0.6 in the 3.x series and v2.11.2 / v2.11.0 / v2.10.3 / v2.10.1 / v2.9.2 / v2.8.1 / v2.7.2 in the 2.x series. The latest is v3.2.1 (Python 3.12). When bringing in existing DAGs, the packages that provide Operators and the way schedules are written differ between the 2.x and 3.x series, so you choose the version to match the assets you are migrating from. For a new build, choosing the 3.x series is the straightforward option. From v2.7.2 onward, a --constraint statement is required in requirements.txt, and if you do not write it, MWAA automatically applies a compatible constraints file. Version upgrades can be performed in units of minor versions, and support for older versions ends in turn, so if you also check the support period at the time of selection, later migration plans become easier to draw up.

Deployment Forms - Environment-Based and MWAA Serverless

MWAA has two deployment forms (as of August 2026). One is the environment-based form described above. In this form, you specify an environment class from mw1.micro to mw1.2xlarge, keep the Airflow environment (web server / scheduler / workers / metadata DB) running at all times, and automatically scale only the workers up and down according to demand. Because you can use the Airflow web UI as is and bring in existing Python DAGs and custom plugins, it is a straightforward choice as a migration target from self-hosted environments. The other is MWAA Serverless. It has no always-on environment, and only the time required to run workflows is billed, so no costs arise for idle time. Because a dedicated execution role and workers are assigned to each workflow, you can separate permissions finely, and there is no need to prepare multiple environments for the separation. Workflows are written in YAML definition files, and you can roll back to past definitions with the built-in versioning feature. Tasks can also run inside a VPC that you specify. However, because it is fixed to Apache Airflow v3 / Python 3.12, DAGs that depend on the 2.x series cannot be moved onto it as is. The supported Regions are not necessarily the same as for the environment-based form, so you check the Region list in the official documentation before selecting it. As a rule of thumb, Serverless suits workflows that run intermittently with long waiting times or cases where you want to separate permissions per workflow. The environment-based form suits cases with a strong dependence on the Airflow web UI or existing plugins and a scale at which DAGs keep running constantly.

Integration and Operations with AWS Services

Through the apache-airflow-providers-amazon package, MWAA provides a rich set of Operators for integrating with AWS services. They cover almost all the services needed in data pipelines, such as Glue (ETL job execution), Athena (SQL queries), EMR (Spark jobs), Redshift (data warehouse queries), Lambda (function invocation), ECS (container task execution), SageMaker (ML training/inference), and Step Functions (state machine execution). With worker Auto Scaling, the number of workers automatically increases and decreases according to the depth of the task queue. You can set a minimum and a maximum number of workers, and you can optimize costs by scaling out only at peak times. Monitoring can be done both with CloudWatch metrics (DAG processing time, task success/failure counts, queue depth) and with the Airflow web UI. For the web server, you can choose public access (with IAM authentication) or private access (within the VPC only).

Choosing Between MWAA and Step Functions

MWAA and Step Functions are both workflow orchestration services, but they differ in design philosophy and in the areas they excel at. Step Functions is event-driven and responds to triggers from API Gateway or EventBridge with low latency (on the order of milliseconds). You define state transitions in JSON (ASL), and it is ideal for serverless workflows that combine Lambda / ECS / Bedrock and others. Pricing is billed per state transition (Standard: $0.025/1,000 transitions), and it is suited to short / high-frequency workflows (as of August 2026 / US East (N. Virginia) Region). MWAA, on the other hand, has its strength in schedule-driven batch pipelines. It handles requirements specific to data engineering, such as complex dependencies (DAGs of tens to hundreds of tasks), retry policies, SLA monitoring, and backfill (re-running past dates). Another major advantage is that you can migrate existing Airflow assets (DAGs, plugins, custom Operators) as is. As a decision criterion, choosing Step Functions for real-time processing or API integration and MWAA for daily/hourly data batch processing is appropriate.

MWAA Pricing

In environment-based deployment, the hourly charge for the additional workers / schedulers / web servers and the storage charge for the metadata DB are added to the hourly charge for the environment class. The following are unit prices as of August 2026, and the US East (N. Virginia) Region and the Tokyo Region have the same prices. The environment classes are $0.29/hour for mw1.micro, $0.49/hour for mw1.small, $0.74/hour for mw1.medium, $0.99/hour for mw1.large, $1.98/hour for mw1.xlarge, and $3.96/hour for mw1.2xlarge. Additional workers are $0.055/hour for small, $0.11/hour for medium, and $0.22/hour for large, and standard storage for the metadata DB is $0.10/GB-month. Because the environment-based form keeps billing while the environment is running, calculating with 730 hours per month gives about $212/month for the environment class portion when you run just one environment of the smallest configuration, mw1.micro. For mw1.small, it is about $358/month (additional workers and storage are separate). If you can stop test environments at night / on weekends, this amount goes down in proportion to the running time. With MWAA Serverless, only workflow execution time is billed, and the unit price of AWS managed tasks is $0.08/hour in US East (N. Virginia) and $0.104/hour in Tokyo (likewise as of August 2026). Because there is no always-on portion, pipelines that run only a few times a day come out much cheaper than with the environment-based form. Conversely, in configurations that run many DAGs constantly, the cost per run is lower with the environment-based form, which assumes always-on operation. When comparing with self-hosted Airflow (EC2 + RDS + Redis), you evaluate by TCO that includes not only the difference in instance charges but also operational effort such as patching / scaling / monitoring. If DAGs run infrequently and their dependencies are simple, Step Functions, which bills per state transition, can sometimes be even cheaper.

Summary - Guidelines for Making Use of MWAA

As a managed service for Apache Airflow, Amazon MWAA delivers data pipeline orchestration without operational burden. DAG definitions in Python have high affinity with version control and testing, and with its rich Operator integrations with AWS services, you can build ETL / analytics / ML pipelines efficiently. There are two deployment forms: the environment-based form that keeps an environment from mw1.micro to mw1.2xlarge running at all times and MWAA Serverless that charges only for execution time. Choosing the former is a good fit if you value ease of bringing in existing Airflow assets and high utilization, and choosing the latter is a good fit if runs are intermittent and you value permission separation per workflow. While Step Functions is suited to event-driven / serverless workflows, MWAA is ideal for batch processing that is schedule-driven / has complex dependencies. As a migration target from existing Airflow environments, and as a tool for improving the productivity of data engineering teams, MWAA is a strong option.