Apache Airflow for Data Engineering - how to run the course code
================================================================
Tested with: Python 3.13, apache-airflow 3.3.2, pandas, duckdb, pytest. Linux/macOS, or WSL on Windows.
Run every command from this folder, in a bash terminal.

1) Install (ch 2)
   python3 -m venv .venv
   source .venv/bin/activate
   pip install "apache-airflow==3.3.2" pandas duckdb pytest --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.3.2/constraints-3.13.txt"
   (use the constraints file for YOUR Python version, e.g. constraints-3.12.txt)
   source airflow.env          (in every new terminal)
   airflow db migrate
   mkdir -p dags tests data

2) Start / stop Airflow, UI at http://localhost:8080 (ch 3)
   bash start_airflow.sh
   bash stop_airflow.sh

3) Run any DAG once, without the scheduler (ch 4+)
   bash dagtest.sh hello_airflow
   bash dagtest.sh daily_report 2026-10-20
   airflow tasks test python_demo greet 2026-10-01

4) Variables and connection used from ch 11 on
   airflow variables set cities '[{"name": "Karachi", "lat": 24.86, "lon": 67.01}, {"name": "Lahore", "lat": 31.55, "lon": 74.34}, {"name": "London", "lat": 51.51, "lon": -0.13}]'
   airflow connections add open_meteo --conn-type generic --conn-host archive-api.open-meteo.com --conn-extra '{"timeout": 30}'

5) Sensor demo (ch 12): start "bash dagtest.sh sensor_demo", then in a 2nd terminal:
   mkdir -p data/incoming && printf 'order_id,amount\n1,25.50\n' > data/incoming/orders.csv

6) Tests (ch 15)
   pytest -q

7) Params (ch 16)
   bash dagtest.sh branch_demo -c '{"mode": "incremental"}'

8) Assets (ch 17, needs the scheduler): start Airflow, then
   airflow dags reserialize
   airflow dags unpause orders_report ; airflow dags unpause orders_producer
   airflow dags trigger orders_producer

9) Final project (ch 18-20)
   bash dagtest.sh weather_pipeline 2026-10-01
   python3 query.py
   bash start_airflow.sh ; airflow dags reserialize
   airflow backfill create --dag-id weather_pipeline --from-date 2026-10-01 --to-date 2026-10-07   (DAG must be unpaused)
   python3 query.py "SELECT city, round(avg(temp_max_c), 1) AS avg_max FROM daily_weather GROUP BY city"
