PySpark for Data Engineering - Full Course (2 Hours): DataFrames, Spark SQL, Parquet, ETL Pipeline – Source Code
Learn PySpark the way data engineers use it, from zero to a complete, rerunnable ETL pipeline, all locally with Java, a Python virtual environment and pip install pyspark: SparkSession and DataFrames, CSV, JSON Lines and Parquet, explicit schemas and bad records, transformations, groupBy, joins and broadcast joins, window functions, Spark SQL, built-in functions versus Python UDFs, nulls and duplicates, partitioned Parquet, lazy evaluation, explain plans, caching and shuffles. Then the data engineering part: incremental loads with a state file, upserts with dynamic partition overwrite, data quality gates with exit codes, and a final project from raw to clean to a star schema with aggregates, a Markdown report and a one-command pipeline that is safe to rerun.
- Duration
- 2:20:56 (20 chapters)
- Tools
- PySpark 3.5.9, OpenJDK 21, Python 3.13.5, VS Code, bash
Watch the course
Chapters
- 0:00:0001. Welcome: why Spark, and the plan
- 0:04:0602. Setup: Java, a virtual environment and PySpark
- 0:09:4303. SparkSession and your first DataFrames
- 0:16:3204. Generating the course data
- 0:25:1405. Reading CSV, JSON Lines and Parquet
- 0:31:0406. Schemas, types and bad records
- 0:39:1407. select, filter and withColumn
- 0:46:2508. Aggregations with groupBy
- 0:52:0609. Joins, and broadcast joins
- 0:59:1410. Window functions
- 1:05:5011. Spark SQL
- 1:12:3212. UDFs versus built-in functions
- 1:18:3113. Handling nulls and duplicates
- 1:26:3814. Partitioning and writing Parquet
- 1:32:4715. Lazy evaluation, explain plans and caching
- 1:39:3316. Performance basics: shuffles, repartition and coalesce
- 1:46:3417. Incremental loads and upserts
- 1:54:5718. Data quality checks
- 2:01:3319. Final project 1: raw to clean to a star schema
- 2:10:2920. Final project 2: aggregates, the report, and rerunning safely
What's in the zip
hello_spark.py– setup check (ch 2)spark_session.py,dataframes.py,make_data.py– first DataFrames and the synthetic shop data (ch 3-4)read_files.py,schemas.py,read_typed.py,bad_records.py,data/bad_orders.csv– reading files, schemas, bad records (ch 5-6)sources.py,transform.py,aggregate.py,joins.py,windows.py,spark_sql.py,udfs.py– transformations to UDFs (ch 7-12)nulls_dupes.py,cleaning.py,write_lake.py,read_lake.py,lazy.py,perf.py– cleaning, Parquet lake, plans, performance (ch 13-16)incremental.py,quality.py– incremental loads, upserts, quality gate (ch 17-18)etl.py,explore_star.py,aggregates.py,report.py,run_pipeline.sh– final project (ch 19-20)README.md,HOW_TO_RUN.txt,LICENSE,MANIFEST.sha256
How to run
Java 17+ (tested with OpenJDK 21) and Python 3.11+ (tested with 3.13.5), pyspark 3.5.9. Linux or macOS; on Windows use WSL. Run every command from the course folder in a bash terminal. Spark runs in local mode; no cluster or cloud account.
java -version
python3 -m venv .venv && source .venv/bin/activate
pip install pyspark==3.5.9
python hello_spark.py
python make_data.py
bash run_pipeline.shIf Spark complains about the hostname, run export SPARK_LOCAL_IP=127.0.0.1. See HOW_TO_RUN.txt for every chapter's exact commands.
Data
No third-party datasets are bundled or downloaded. All data is synthetic, generated by the course code itself (make_data.py, fixed seed).
File: pyspark-for-data-engineering-code.zip
Size: 21.0 KB (21,501 bytes)
SHA-256:64bf96dfcbb25d25c7a471d067426a84faebe00dac5a7b6179e4b03fafbeb8b2
Licence: code is released under the MIT licence, copyright PyInns. See LICENSE and README inside the zip. Free learning code, not production code.