Ollama Python Tutorial: Run Local LLMs with Chat, Streaming, Structured Outputs, Tools and Embeddings (2026) Python sqlite3 Tutorial: Tables, Transactions, JSON and Upserts with the Built-in Database (2026) Convert Python to EXE with PyInstaller: One-File Apps, Data Files and Hidden Imports (2026) Two-Factor Authentication in Python with pyotp: TOTP, QR Codes and FastAPI (2026) Hash Passwords in Python with Argon2: pwdlib, argon2-cffi and bcrypt Migration (2026) Encrypt and Decrypt in Python with Fernet: Strings, Files and Passwords (2026) psutil Tutorial: Monitor CPU, Memory, Disk and Processes in Python (2026) PyMuPDF Tutorial: PDF to Image, Extract Text and Images in Python (2026) MoviePy 2 Tutorial: Edit Videos with Python (Trim, Text, Merge, Audio) 2026 FastAPI Dependency Injection with Depends: Annotated, Yield and Overrides (2026) HTML to PDF in Python with WeasyPrint and Jinja2: Invoices and Reports (2026) Extract Tables from PDF in Python with pdfplumber: CSV, Excel and pandas (2026) OCR in Python: Extract Text from Images and Scanned PDFs with RapidOCR (2026) Text to Speech in Python with Piper: Offline Natural Voices, WAV Files and a TTS API (2026) Speech to Text in Python with faster-whisper: Transcribe Audio and Make Subtitles (2026) Send Email with Python: smtplib, HTML, Attachments and Bulk Mail (2026) Python QR Code Generator: Styled QR Codes with qrcode (2026) Pillow 13: Resize, Crop and Watermark Images in Python (2026) Parallel Programming with Dask in Python 2026: Intro Guide MarkItDown: Convert Files to Markdown for LLMs in Python (2026) Plotly Express: Interactive Charts in Python (2026) pandera: Validate pandas DataFrames in Python (2026) jsonschema: Validate JSON Data in Python (2026) Faker: Generate Fake Test Data in Python (2026) RapidFuzz: Fuzzy String Matching in Python (2026) sqlglot: Parse and Transpile SQL in Python (2026) structlog: Structured Logging in Python (2026) python-pptx in 2026: Automate PowerPoint Reports with Python python-docx: Create and Edit Word Documents with Python (2026) pypdf: Merge, Split and Edit PDFs with Python (2026) View all in this category →

PySpark for Data Engineering - Full Course (2 Hours): DataFrames, Spark SQL, Parquet, ETL Pipeline – Source Code

Learn PySpark the way data engineers use it, from zero to a complete, rerunnable ETL pipeline, all locally with Java, a Python virtual environment and pip install pyspark: SparkSession and DataFrames, CSV, JSON Lines and Parquet, explicit schemas and bad records, transformations, groupBy, joins and broadcast joins, window functions, Spark SQL, built-in functions versus Python UDFs, nulls and duplicates, partitioned Parquet, lazy evaluation, explain plans, caching and shuffles. Then the data engineering part: incremental loads with a state file, upserts with dynamic partition overwrite, data quality gates with exit codes, and a final project from raw to clean to a star schema with aggregates, a Markdown report and a one-command pipeline that is safe to rerun.

Duration
2:20:56 (20 chapters)
Tools
PySpark 3.5.9, OpenJDK 21, Python 3.13.5, VS Code, bash

Watch the course

Watch on YouTube

Chapters

  1. 0:00:0001. Welcome: why Spark, and the plan
  2. 0:04:0602. Setup: Java, a virtual environment and PySpark
  3. 0:09:4303. SparkSession and your first DataFrames
  4. 0:16:3204. Generating the course data
  5. 0:25:1405. Reading CSV, JSON Lines and Parquet
  6. 0:31:0406. Schemas, types and bad records
  7. 0:39:1407. select, filter and withColumn
  8. 0:46:2508. Aggregations with groupBy
  9. 0:52:0609. Joins, and broadcast joins
  10. 0:59:1410. Window functions
  11. 1:05:5011. Spark SQL
  12. 1:12:3212. UDFs versus built-in functions
  13. 1:18:3113. Handling nulls and duplicates
  14. 1:26:3814. Partitioning and writing Parquet
  15. 1:32:4715. Lazy evaluation, explain plans and caching
  16. 1:39:3316. Performance basics: shuffles, repartition and coalesce
  17. 1:46:3417. Incremental loads and upserts
  18. 1:54:5718. Data quality checks
  19. 2:01:3319. Final project 1: raw to clean to a star schema
  20. 2:10:2920. Final project 2: aggregates, the report, and rerunning safely

What's in the zip

  • hello_spark.py – setup check (ch 2)
  • spark_session.py, dataframes.py, make_data.py – first DataFrames and the synthetic shop data (ch 3-4)
  • read_files.py, schemas.py, read_typed.py, bad_records.py, data/bad_orders.csv – reading files, schemas, bad records (ch 5-6)
  • sources.py, transform.py, aggregate.py, joins.py, windows.py, spark_sql.py, udfs.py – transformations to UDFs (ch 7-12)
  • nulls_dupes.py, cleaning.py, write_lake.py, read_lake.py, lazy.py, perf.py – cleaning, Parquet lake, plans, performance (ch 13-16)
  • incremental.py, quality.py – incremental loads, upserts, quality gate (ch 17-18)
  • etl.py, explore_star.py, aggregates.py, report.py, run_pipeline.sh – final project (ch 19-20)
  • README.md, HOW_TO_RUN.txt, LICENSE, MANIFEST.sha256

How to run

Java 17+ (tested with OpenJDK 21) and Python 3.11+ (tested with 3.13.5), pyspark 3.5.9. Linux or macOS; on Windows use WSL. Run every command from the course folder in a bash terminal. Spark runs in local mode; no cluster or cloud account.

java -version
python3 -m venv .venv && source .venv/bin/activate
pip install pyspark==3.5.9
python hello_spark.py
python make_data.py
bash run_pipeline.sh

If Spark complains about the hostname, run export SPARK_LOCAL_IP=127.0.0.1. See HOW_TO_RUN.txt for every chapter's exact commands.

Data

No third-party datasets are bundled or downloaded. All data is synthetic, generated by the course code itself (make_data.py, fixed seed).

Download source code

File: pyspark-for-data-engineering-code.zip

Size: 21.0 KB (21,501 bytes)

SHA-256:
64bf96dfcbb25d25c7a471d067426a84faebe00dac5a7b6179e4b03fafbeb8b2

Licence: code is released under the MIT licence, copyright PyInns. See LICENSE and README inside the zip. Free learning code, not production code.