Apache Airflow for Data Engineering - Full Course (2 Hours): DAGs, TaskFlow, Scheduling, ETL Project – Source Code
Learn Apache Airflow 3 the way data engineers use it, from zero to a scheduled, tested, rerunnable pipeline, all locally with a Python virtual environment: DAGs and tasks, the web UI, operators and TaskFlow, dependencies, scheduling, catchup and backfills, templating, XComs, variables and connections, sensors, retries and callbacks, dynamic task mapping, testing DAGs with pytest, branching and run parameters, and asset-based scheduling. Then a final project: extract weather data from a public API, transform it with pandas, load it into DuckDB idempotently, and gate it with data quality checks.
- Duration
- 2:05:27 (20 chapters)
- Tools
- Apache Airflow 3.3.2, Python 3.13, pandas, DuckDB, pytest, VS Code, bash
Watch the course
Chapters
- 0:00:0001. Welcome: what orchestration is, and the plan
- 0:04:4802. Installing Airflow locally
- 0:11:3703. Starting Airflow and the web UI
- 0:17:2304. Your first DAG
- 0:25:4105. Tasks, dependencies and the graph
- 0:31:4506. The PythonOperator, and testing one task
- 0:36:1207. The TaskFlow API
- 0:41:2008. Scheduling, logical dates and catchup
- 0:49:1409. Templates and the run context
- 0:55:0910. XComs: passing data between tasks
- 0:59:3511. Variables and connections
- 1:06:1312. Sensors: waiting for something to happen
- 1:10:4213. Retries, timeouts, trigger rules and alerts
- 1:17:1014. Dynamic task mapping
- 1:22:1915. Testing DAGs with pytest
- 1:29:5616. Branching and run parameters
- 1:36:0817. Assets: data-aware scheduling
- 1:41:5718. Final project 1: extract weather data from a public API
- 1:48:2119. Final project 2: transform with pandas, load into DuckDB
- 1:54:5420. Final project 3: quality checks, backfill and the finished pipeline
What's in the zip
airflow-for-data-engineering/HOW_TO_RUN.txtairflow-for-data-engineering/LICENSEairflow-for-data-engineering/MANIFEST.sha256airflow-for-data-engineering/README.mdairflow-for-data-engineering/airflow.envairflow-for-data-engineering/dags/assets_demo.pyairflow-for-data-engineering/dags/branch_demo.pyairflow-for-data-engineering/dags/config_demo.pyairflow-for-data-engineering/dags/daily_report.pyairflow-for-data-engineering/dags/dependencies_demo.pyairflow-for-data-engineering/dags/hello_airflow.pyairflow-for-data-engineering/dags/mapping_demo.pyairflow-for-data-engineering/dags/python_demo.pyairflow-for-data-engineering/dags/reliability_demo.pyairflow-for-data-engineering/dags/sensor_demo.pyairflow-for-data-engineering/dags/taskflow_etl.pyairflow-for-data-engineering/dags/templates_demo.pyairflow-for-data-engineering/dags/weather_helpers.pyairflow-for-data-engineering/dags/weather_pipeline.pyairflow-for-data-engineering/dags/xcom_demo.pyairflow-for-data-engineering/dagtest.shairflow-for-data-engineering/pytest.iniairflow-for-data-engineering/query.pyairflow-for-data-engineering/start_airflow.shairflow-for-data-engineering/stop_airflow.shairflow-for-data-engineering/tests/test_dags.pyairflow-for-data-engineering/tests/test_helpers.py
How to run
Python 3.11+ (tested with 3.13) and apache-airflow 3.3.2 installed with the official constraints file. Linux or macOS; on Windows use WSL. Run every command from the course folder in a bash terminal. Airflow runs locally with SQLite; no cloud account.
python3 -m venv .venv && source .venv/bin/activate
pip install "apache-airflow==3.3.2" pandas duckdb pytest --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.3.2/constraints-3.13.txt"
source airflow.env
airflow db migrate
bash dagtest.sh hello_airflow
bash start_airflow.shSee HOW_TO_RUN.txt for every chapter's exact commands, including the variable and connection used by the final project.
Data
No datasets are bundled. The final project downloads daily weather history from the free public Open-Meteo archive API (no key needed); the other demo data is generated by the course code itself.
File: airflow-for-data-engineering-code.zip
Size: 16.8 KB (17,184 bytes)
SHA-256:56650c810fc7ad01be12418e72a4ba204a13c184261a1f1c03142b166fb853c3
Licence: code is released under the MIT licence, copyright PyInns. See LICENSE and README inside the zip. Free learning code, not production code.