Ollama Python Tutorial: Run Local LLMs with Chat, Streaming, Structured Outputs, Tools and Embeddings (2026) Python sqlite3 Tutorial: Tables, Transactions, JSON and Upserts with the Built-in Database (2026) Convert Python to EXE with PyInstaller: One-File Apps, Data Files and Hidden Imports (2026) Two-Factor Authentication in Python with pyotp: TOTP, QR Codes and FastAPI (2026) Hash Passwords in Python with Argon2: pwdlib, argon2-cffi and bcrypt Migration (2026) Encrypt and Decrypt in Python with Fernet: Strings, Files and Passwords (2026) psutil Tutorial: Monitor CPU, Memory, Disk and Processes in Python (2026) PyMuPDF Tutorial: PDF to Image, Extract Text and Images in Python (2026) MoviePy 2 Tutorial: Edit Videos with Python (Trim, Text, Merge, Audio) 2026 FastAPI Dependency Injection with Depends: Annotated, Yield and Overrides (2026) HTML to PDF in Python with WeasyPrint and Jinja2: Invoices and Reports (2026) Extract Tables from PDF in Python with pdfplumber: CSV, Excel and pandas (2026) OCR in Python: Extract Text from Images and Scanned PDFs with RapidOCR (2026) Text to Speech in Python with Piper: Offline Natural Voices, WAV Files and a TTS API (2026) Speech to Text in Python with faster-whisper: Transcribe Audio and Make Subtitles (2026) Send Email with Python: smtplib, HTML, Attachments and Bulk Mail (2026) Python QR Code Generator: Styled QR Codes with qrcode (2026) Pillow 13: Resize, Crop and Watermark Images in Python (2026) Parallel Programming with Dask in Python 2026: Intro Guide MarkItDown: Convert Files to Markdown for LLMs in Python (2026) Plotly Express: Interactive Charts in Python (2026) pandera: Validate pandas DataFrames in Python (2026) jsonschema: Validate JSON Data in Python (2026) Faker: Generate Fake Test Data in Python (2026) RapidFuzz: Fuzzy String Matching in Python (2026) sqlglot: Parse and Transpile SQL in Python (2026) structlog: Structured Logging in Python (2026) python-pptx in 2026: Automate PowerPoint Reports with Python python-docx: Create and Edit Word Documents with Python (2026) pypdf: Merge, Split and Edit PDFs with Python (2026) View all in this category →

Spark Declarative Pipelines for Data Engineering - Full Course: Bronze, Silver, Gold ETL with PySpark 4 – Source Code

Learn Spark Declarative Pipelines (SDP), the new declarative way to build data pipelines in open source Apache Spark 4, by building a three-stage raw, cleaned and report-ready (bronze, silver, gold) ETL project locally with pyspark[pipelines] 4.3.0: spark-pipelines init and run, the pipeline spec, dp.table and dp.materialized_view, SQL materialized views, streaming tables that ingest CSV files incrementally, a persistent Hive metastore, data quality rules with a quarantine table, dry-run, checkpoints, refresh and full refresh.

Duration
0:43:01 (8 chapters)
Tools
Apache Spark 4.3.0 (pyspark[pipelines]), Java 21, Python 3.13, Hive metastore (local Derby), Parquet, VS Code, bash

Watch the course

Watch on YouTube

Chapters

  1. 0:00:0001. Welcome: what Spark Declarative Pipelines are, and setup
  2. 0:05:0002. Your first pipeline: spark-pipelines init and run
  3. 0:10:3903. A catalog that remembers: Hive metastore and a database
  4. 0:17:1204. Bronze: a streaming table that ingests CSV files
  5. 0:24:3905. Silver: clean data and a quarantine table in SQL
  6. 0:29:1006. Gold: business tables, and catching mistakes with dry-run
  7. 0:34:0607. Incremental runs, refresh and full refresh
  8. 0:38:3308. The finished pipeline, recap and next steps

What's in the zip

  • spark-declarative-pipelines/HOW_TO_RUN.txt
  • spark-declarative-pipelines/LICENSE
  • spark-declarative-pipelines/MANIFEST.sha256
  • spark-declarative-pipelines/README.md
  • spark-declarative-pipelines/conf/spark-defaults.conf
  • spark-declarative-pipelines/env.sh
  • spark-declarative-pipelines/make_data.py
  • spark-declarative-pipelines/show.py
  • spark-declarative-pipelines/spark-pipeline.yml
  • spark-declarative-pipelines/transformations/bronze.py
  • spark-declarative-pipelines/transformations/gold.sql
  • spark-declarative-pipelines/transformations/products.sql
  • spark-declarative-pipelines/transformations/silver.sql

How to run

Python 3.11+ (tested with 3.13), Java 17 or 21 (tested with 21) and pyspark[pipelines] 4.3.0. Linux or macOS; on Windows use WSL. Run every command from the course folder in a bash terminal. First edit spark-pipeline.yml and set storage: to an absolute file:// folder on your machine.

python3 -m venv .venv && source .venv/bin/activate
pip install "pyspark[pipelines]==4.3.0" "pandas<3"
source env.sh
spark-sql -e "CREATE DATABASE IF NOT EXISTS shop"
python make_data.py 2026-11-01
spark-pipelines dry-run
spark-pipelines run
python show.py bronze_orders silver_orders quarantine_orders gold_daily_revenue gold_city_revenue gold_top_products

See HOW_TO_RUN.txt for the incremental next-day run, --refresh and --full-refresh-all, and how to start over.

Data

No datasets are bundled. make_data.py generates fake customers and one day of orders per channel as CSV files; everything is stored locally as Parquet with a local Hive metastore. No cloud account and no API key.

Download source code

File: spark-declarative-pipelines-code.zip

Size: 6.8 KB (6,946 bytes)

SHA-256:
72e8f2e9cde1da52f7f513cfefe3939446e7e1e46afbdd676699ea3cf6a44717

Licence: code is released under the MIT licence, copyright PyInns. See LICENSE and README inside the zip. Free learning code, not production code.