Spark Declarative Pipelines for Data Engineering - Full Course: Bronze, Silver, Gold ETL with PySpark 4 – Source Code
Learn Spark Declarative Pipelines (SDP), the new declarative way to build data pipelines in open source Apache Spark 4, by building a three-stage raw, cleaned and report-ready (bronze, silver, gold) ETL project locally with pyspark[pipelines] 4.3.0: spark-pipelines init and run, the pipeline spec, dp.table and dp.materialized_view, SQL materialized views, streaming tables that ingest CSV files incrementally, a persistent Hive metastore, data quality rules with a quarantine table, dry-run, checkpoints, refresh and full refresh.
- Duration
- 0:43:01 (8 chapters)
- Tools
- Apache Spark 4.3.0 (pyspark[pipelines]), Java 21, Python 3.13, Hive metastore (local Derby), Parquet, VS Code, bash
Watch the course
Chapters
- 0:00:0001. Welcome: what Spark Declarative Pipelines are, and setup
- 0:05:0002. Your first pipeline: spark-pipelines init and run
- 0:10:3903. A catalog that remembers: Hive metastore and a database
- 0:17:1204. Bronze: a streaming table that ingests CSV files
- 0:24:3905. Silver: clean data and a quarantine table in SQL
- 0:29:1006. Gold: business tables, and catching mistakes with dry-run
- 0:34:0607. Incremental runs, refresh and full refresh
- 0:38:3308. The finished pipeline, recap and next steps
What's in the zip
spark-declarative-pipelines/HOW_TO_RUN.txtspark-declarative-pipelines/LICENSEspark-declarative-pipelines/MANIFEST.sha256spark-declarative-pipelines/README.mdspark-declarative-pipelines/conf/spark-defaults.confspark-declarative-pipelines/env.shspark-declarative-pipelines/make_data.pyspark-declarative-pipelines/show.pyspark-declarative-pipelines/spark-pipeline.ymlspark-declarative-pipelines/transformations/bronze.pyspark-declarative-pipelines/transformations/gold.sqlspark-declarative-pipelines/transformations/products.sqlspark-declarative-pipelines/transformations/silver.sql
How to run
Python 3.11+ (tested with 3.13), Java 17 or 21 (tested with 21) and pyspark[pipelines] 4.3.0. Linux or macOS; on Windows use WSL. Run every command from the course folder in a bash terminal. First edit spark-pipeline.yml and set storage: to an absolute file:// folder on your machine.
python3 -m venv .venv && source .venv/bin/activate
pip install "pyspark[pipelines]==4.3.0" "pandas<3"
source env.sh
spark-sql -e "CREATE DATABASE IF NOT EXISTS shop"
python make_data.py 2026-11-01
spark-pipelines dry-run
spark-pipelines run
python show.py bronze_orders silver_orders quarantine_orders gold_daily_revenue gold_city_revenue gold_top_productsSee HOW_TO_RUN.txt for the incremental next-day run, --refresh and --full-refresh-all, and how to start over.
Data
No datasets are bundled. make_data.py generates fake customers and one day of orders per channel as CSV files; everything is stored locally as Parquet with a local Hive metastore. No cloud account and no API key.
File: spark-declarative-pipelines-code.zip
Size: 6.8 KB (6,946 bytes)
SHA-256:72e8f2e9cde1da52f7f513cfefe3939446e7e1e46afbdd676699ea3cf6a44717
Licence: code is released under the MIT licence, copyright PyInns. See LICENSE and README inside the zip. Free learning code, not production code.