dbt for Data Engineering - Full Course - how to run the course code
===================================================================
Tested with: Python 3.13, dbt-core 1.12.6, dbt-duckdb 1.11.1 (DuckDB 2.0.0), dbt_utils 1.4.1, apache-airflow 3.4.0.
Linux or macOS, or WSL on Windows. Run every command from this folder, in a bash terminal.
Everything is local: DuckDB is a single file (shopflow.duckdb). No cloud account, no API key, no Docker.

1. python3 -m venv .venv && source .venv/bin/activate
   pip install dbt-core==1.12.6 dbt-duckdb==1.11.1
2. Make and load the raw data (Extract + Load):
   python generate_data.py          (writes data/*.csv, seed 42, with a few messy rows on purpose)
   python load_raw.py               (raw.* tables in shopflow.duckdb)
3. dbt finds the profiles.yml in this folder by itself (run dbt from this folder):
   dbt debug
4. Packages, then everything in one go:
   dbt deps
   dbt source freshness
   dbt build                         (seeds, models, snapshot, data tests and unit tests)
5. The steps of the course one by one:
   dbt seed ; dbt run ; dbt test
   dbt test --select test_type:unit
   dbt docs generate && dbt docs serve --port 8080    (open http://localhost:8080)
   dbt snapshot ; python update_customers.py ; dbt snapshot      (SCD type 2)
   python new_events.py 500 ; dbt run --select fct_order_events  (incremental)
   python new_events.py 200 ; dbt run --select fct_order_events
   dbt run --select fct_order_events --full-refresh
   dbt ls --selector nightly --resource-type model
6. Prod target and state:
   python load_raw.py shopflow_prod.duckdb && python new_events.py 500 shopflow_prod.duckdb
   dbt build --target prod
   mkdir -p prod_state && cp target/manifest.json prod_state/
   dbt build --select state:modified+ --defer --state prod_state
7. Report for the business:
   dbt build --select +mart_monthly_revenue
   python export_report.py           (reports/monthly_revenue.csv)
8. Contracts, Python model, hooks, run-operation, analyses, exposures, retry:
   dbt build --select mart_monthly_revenue          (enforced contract)
   dbt build --select status_summary                (Python model, runs inside DuckDB)
   dbt show --inline "select * from main.dbt_run_log"   (written by the on-run-end hook)
   dbt run-operation row_counts --args '{schema_name: staging}'
   dbt show --select top_customers --limit 5        (analysis)
   dbt ls --select +exposure:monthly_revenue_report --resource-type model
   dbt retry                                        (after a failed run: reruns only failed + skipped nodes)
9. Airflow (in its own virtual environment, so it never conflicts with dbt):
   python3 -m venv ~/airflow-venv && ~/airflow-venv/bin/pip install apache-airflow==3.4.0
   Edit the PATH line in airflow_env.sh to point at YOUR airflow venv bin folder
   (in the video it is /tmp/dt/airflow-venv/bin).
   source airflow_env.sh && airflow db migrate
   airflow dags reserialize && airflow dags list
   airflow dags test shopflow_dbt 2026-11-09

Notes
- The dev database path can be changed with: export SHOPFLOW_DB=/path/to/file.duckdb
- To start over: dbt clean, then rm -f *.duckdb and run steps 2 and 4 again.
- The raw data is generated, not real customer data.
