Python for Data Engineering - Full Course (2.5 Hours): pandas, polars, SQL, APIs, pytest, Airflow, PySpark – Source Code
Learn to use Python the way data engineers do: CSV, JSON, JSON Lines and Parquet, pandas and polars transforms, cleaning and validation, SQL from Python with SQLite and PostgreSQL, REST API ingestion with retries, a structured ETL pipeline, pytest, cron, Apache Airflow and PySpark. The final project is a weather pipeline on real Open-Meteo data, validated, loaded to Parquet and SQLite, tested and safe to rerun.
- Duration
- 2:38:11 (18 chapters)
- Tools
- Python 3.13.5, uv 0.13, VS Code, SQLite 3.46, PostgreSQL 17, pandas 3.0.6, polars 2.0.0, PyArrow 26.0.0, requests 2.34.2, psycopg 3.3.6, pytest 9.1.1, Apache Airflow 3.3.2, PySpark 4.2.0 (Java 21)
Watch the course
Chapters
- 0:00:0001. Welcome, the plan, and setup
- 0:07:1802. Python for data: generating and exploring data
- 0:17:0403. Files: CSV, JSON, JSON Lines and Parquet
- 0:24:2904. Transforms with pandas
- 0:31:0705. Transforms with polars, and a speed test
- 0:39:0906. Data cleaning and validation
- 0:46:2607. SQL from Python with SQLite
- 0:54:5808. PostgreSQL from Python
- 1:03:3609. Ingesting data from a REST API
- 1:13:2810. Logging, config and retries
- 1:22:3911. Structuring an ETL pipeline
- 1:33:0312. Testing pipelines with pytest
- 1:42:0113. Scheduling with cron
- 1:50:5614. An introduction to Apache Airflow
- 2:00:0915. An introduction to PySpark
- 2:07:5016. Final project 1: weather pipeline, extract and validate
- 2:22:3117. Final project 2: load, report, test and rerun
- 2:33:4218. What's next: an honest map
What's in the zip
requirements.txt,check_setup.py– setup (ch 1)make_data.py,explore.py,formats.py– data generation and file formats (ch 2-3)transform_pandas.py,transform_polars.py,race.py,cleaning.py– transforms and validation (ch 4-6)db_sqlite.py,db_postgres.py,pg.env– SQL from Python (ch 7-8)mock_api.py,ingest_api.py,ingest.py,config.toml– API ingestion, logging and retries (ch 9-10)pipeline/,tests/– ETL structure and pytest (ch 11-12)run_pipeline.sh,schedule_loop.py,airflow/dags/orders_dag.py– cron and Airflow (ch 13-14)spark_orders.py– PySpark (ch 15)weather/– final project (ch 16-17)README.md,HOW_TO_RUN.txt,LICENSE,MANIFEST.sha256
How to run
Python 3.11+ (tested with 3.13.5). Java 17/21 for PySpark, PostgreSQL 17 for chapter 8. Linux or macOS; on Windows use WSL (Airflow does not run natively on Windows).
uv venv --python 3.13 .venv && source .venv/bin/activate
uv pip install -r requirements.txt # or: python -m venv .venv ; pip install -r requirements.txt
python check_setup.py
python make_data.pyAirflow uses its own virtual environment. See HOW_TO_RUN.txt for every chapter's exact commands.
Data
No third-party datasets are bundled. Practice orders are synthetic, generated by make_data.py with a fixed seed. The final project fetches historical weather from the Open-Meteo API: weather data by Open-Meteo.com, licensed under CC BY 4.0. The mock API in chapters 9-14 runs locally and fails on purpose to practise retries.
File: python-for-data-engineering-code.zip
Size: 25.3 KB (25,865 bytes)
SHA-256:90f23fcc13272125315d91b716606290688585341c99f116f35ffa738e584d03
Licence: code is released under the MIT licence, copyright PyInns. See LICENSE and README inside the zip. Free learning code, not production code.