dbt for Data Engineering - Full Course (dbt Core, DuckDB, Tests, Snapshots, Airflow Project) – Source Code
Learn dbt for data engineering from zero, locally: dbt Core with DuckDB, sources and freshness, staging, intermediate and mart models with ref(), seeds, materializations, Jinja macros, data tests, unit tests, docs, SCD type 2 snapshots, incremental models, dbt_utils, selectors and state, a report-ready revenue mart, an Airflow DAG, model contracts, a Python model, hooks and dbt retry.
- Duration
- 2:02:00 (29 chapters)
- Tools
- dbt Core 1.12.6, dbt-duckdb 1.11.1, DuckDB 2.0.0, dbt_utils 1.4.1, Python 3.13, Apache Airflow 3.4.0, VS Code, bash
Watch the course
Chapters
- 0:00:0001. Welcome: why dbt, ELT and the ShopFlow plan
- 0:04:1402. Setup: Python, dbt Core and the DuckDB adapter
- 0:08:1803. Generating the ShopFlow raw data
- 0:13:5304. Extract and load: raw data into DuckDB
- 0:17:1805. A dbt project from scratch: dbt_project.yml and folders
- 0:20:3906. profiles.yml, targets and dbt debug
- 0:23:3807. Sources and source freshness
- 0:26:3108. Your first models: staging with source()
- 0:30:5409. Seeds and custom schemas
- 0:34:5510. ref() and the DAG: intermediate models
- 0:38:5311. Materializations: view, table, ephemeral, incremental
- 0:44:3512. Jinja and macros: reusable SQL
- 0:48:1313. Data tests: unique, not_null, accepted_values, relationships
- 0:52:5214. Singular tests and your own generic test
- 0:56:2015. Unit tests: test model logic with fake rows
- 1:00:1816. Documentation and dbt docs
- 1:03:3817. Snapshots: keeping history with SCD type 2
- 1:08:0418. Incremental models for big, growing tables
- 1:13:2619. Packages: dbt_utils
- 1:16:5820. Selectors, tags, state and dev vs prod
- 1:21:1021. A report-ready mart and exporting the report
- 1:25:0622. Orchestrating dbt with Airflow on a schedule
- 1:30:2523. Model contracts and constraints
- 1:34:5924. Python models in dbt
- 1:38:2525. Hooks and run-operation macros
- 1:42:2026. Analyses, exposures and fixing failures with dbt retry
- 1:46:4227. A production checklist for dbt projects
- 1:50:1428. dbt interview questions
- 1:56:4929. Recap and next steps
What's in the zip
dbt-for-data-engineering/.gitignoredbt-for-data-engineering/HOW_TO_RUN.txtdbt-for-data-engineering/LICENSEdbt-for-data-engineering/MANIFEST.sha256dbt-for-data-engineering/README.mddbt-for-data-engineering/airflow_env.shdbt-for-data-engineering/analyses/.gitkeepdbt-for-data-engineering/analyses/top_customers.sqldbt-for-data-engineering/dags/shopflow_dbt.pydbt-for-data-engineering/dbt_project.ymldbt-for-data-engineering/export_report.pydbt-for-data-engineering/generate_data.pydbt-for-data-engineering/interview.mddbt-for-data-engineering/load_raw.pydbt-for-data-engineering/macros/cents_to_dollars.sqldbt-for-data-engineering/macros/generate_schema_name.sqldbt-for-data-engineering/macros/row_counts.sqldbt-for-data-engineering/models/docs.mddbt-for-data-engineering/models/intermediate/int_order_items_enriched.sqldbt-for-data-engineering/models/intermediate/int_payments_pivoted.sqldbt-for-data-engineering/models/marts/_exposures.ymldbt-for-data-engineering/models/marts/_marts.ymldbt-for-data-engineering/models/marts/dim_customers.sqldbt-for-data-engineering/models/marts/fct_order_events.sqldbt-for-data-engineering/models/marts/fct_order_items.sqldbt-for-data-engineering/models/marts/fct_orders.sqldbt-for-data-engineering/models/marts/mart_monthly_revenue.sqldbt-for-data-engineering/models/marts/status_summary.pydbt-for-data-engineering/models/staging/_sources.ymldbt-for-data-engineering/models/staging/_staging.ymldbt-for-data-engineering/models/staging/stg_customers.sqldbt-for-data-engineering/models/staging/stg_order_items.sqldbt-for-data-engineering/models/staging/stg_orders.sqldbt-for-data-engineering/models/staging/stg_payments.sqldbt-for-data-engineering/models/staging/stg_products.sqldbt-for-data-engineering/new_events.pydbt-for-data-engineering/notes/materializations.mddbt-for-data-engineering/notes/production_checklist.mddbt-for-data-engineering/package-lock.ymldbt-for-data-engineering/packages.ymldbt-for-data-engineering/plan.mddbt-for-data-engineering/profiles.ymldbt-for-data-engineering/seeds/country_regions.csvdbt-for-data-engineering/seeds/payment_methods.csvdbt-for-data-engineering/selectors.ymldbt-for-data-engineering/snapshots/customers_snapshot.ymldbt-for-data-engineering/tests/assert_payments_match_order_totals.sqldbt-for-data-engineering/tests/generic/positive_value.sqldbt-for-data-engineering/update_customers.py
How to run
Python 3.10+ (tested with 3.13). Linux or macOS; on Windows use WSL. Run every command from the course folder in a bash terminal.
python3 -m venv .venv && source .venv/bin/activate
pip install dbt-core==1.12.6 dbt-duckdb==1.11.1
python generate_data.py
python load_raw.py
dbt debug
dbt deps
dbt build
python export_report.pySee HOW_TO_RUN.txt for every step from the video: snapshots, incremental runs, the prod target and state, and the Airflow DAG.
Data
No datasets are bundled. generate_data.py generates fake ShopFlow data (seed 42, with a few messy rows on purpose) and load_raw.py loads it into a local DuckDB file. dbt packages are downloaded by dbt deps, not bundled. No cloud account and no API key.
File: dbt-for-data-engineering-code.zip
Size: 25.4 KB (26,004 bytes)
SHA-256:29d38dfec90dc6294cfd2490f2921019e0454525d156dc7a832ee5f252cfcee9
Licence: code is released under the MIT licence, copyright PyInns. See LICENSE and README inside the zip. Free learning code, not production code.