Ollama Python Tutorial: Run Local LLMs with Chat, Streaming, Structured Outputs, Tools and Embeddings (2026) Python sqlite3 Tutorial: Tables, Transactions, JSON and Upserts with the Built-in Database (2026) Convert Python to EXE with PyInstaller: One-File Apps, Data Files and Hidden Imports (2026) Two-Factor Authentication in Python with pyotp: TOTP, QR Codes and FastAPI (2026) Hash Passwords in Python with Argon2: pwdlib, argon2-cffi and bcrypt Migration (2026) Encrypt and Decrypt in Python with Fernet: Strings, Files and Passwords (2026) psutil Tutorial: Monitor CPU, Memory, Disk and Processes in Python (2026) PyMuPDF Tutorial: PDF to Image, Extract Text and Images in Python (2026) MoviePy 2 Tutorial: Edit Videos with Python (Trim, Text, Merge, Audio) 2026 FastAPI Dependency Injection with Depends: Annotated, Yield and Overrides (2026) HTML to PDF in Python with WeasyPrint and Jinja2: Invoices and Reports (2026) Extract Tables from PDF in Python with pdfplumber: CSV, Excel and pandas (2026) OCR in Python: Extract Text from Images and Scanned PDFs with RapidOCR (2026) Text to Speech in Python with Piper: Offline Natural Voices, WAV Files and a TTS API (2026) Speech to Text in Python with faster-whisper: Transcribe Audio and Make Subtitles (2026) Send Email with Python: smtplib, HTML, Attachments and Bulk Mail (2026) Python QR Code Generator: Styled QR Codes with qrcode (2026) Pillow 13: Resize, Crop and Watermark Images in Python (2026) Parallel Programming with Dask in Python 2026: Intro Guide MarkItDown: Convert Files to Markdown for LLMs in Python (2026) Plotly Express: Interactive Charts in Python (2026) pandera: Validate pandas DataFrames in Python (2026) jsonschema: Validate JSON Data in Python (2026) Faker: Generate Fake Test Data in Python (2026) RapidFuzz: Fuzzy String Matching in Python (2026) sqlglot: Parse and Transpile SQL in Python (2026) structlog: Structured Logging in Python (2026) python-pptx in 2026: Automate PowerPoint Reports with Python python-docx: Create and Edit Word Documents with Python (2026) pypdf: Merge, Split and Edit PDFs with Python (2026) View all in this category →

Python for Data Engineering - Full Course (2.5 Hours): pandas, polars, SQL, APIs, pytest, Airflow, PySpark – Source Code

Learn to use Python the way data engineers do: CSV, JSON, JSON Lines and Parquet, pandas and polars transforms, cleaning and validation, SQL from Python with SQLite and PostgreSQL, REST API ingestion with retries, a structured ETL pipeline, pytest, cron, Apache Airflow and PySpark. The final project is a weather pipeline on real Open-Meteo data, validated, loaded to Parquet and SQLite, tested and safe to rerun.

Duration
2:38:11 (18 chapters)
Tools
Python 3.13.5, uv 0.13, VS Code, SQLite 3.46, PostgreSQL 17, pandas 3.0.6, polars 2.0.0, PyArrow 26.0.0, requests 2.34.2, psycopg 3.3.6, pytest 9.1.1, Apache Airflow 3.3.2, PySpark 4.2.0 (Java 21)

Watch the course

Chapters

  1. 0:00:0001. Welcome, the plan, and setup
  2. 0:07:1802. Python for data: generating and exploring data
  3. 0:17:0403. Files: CSV, JSON, JSON Lines and Parquet
  4. 0:24:2904. Transforms with pandas
  5. 0:31:0705. Transforms with polars, and a speed test
  6. 0:39:0906. Data cleaning and validation
  7. 0:46:2607. SQL from Python with SQLite
  8. 0:54:5808. PostgreSQL from Python
  9. 1:03:3609. Ingesting data from a REST API
  10. 1:13:2810. Logging, config and retries
  11. 1:22:3911. Structuring an ETL pipeline
  12. 1:33:0312. Testing pipelines with pytest
  13. 1:42:0113. Scheduling with cron
  14. 1:50:5614. An introduction to Apache Airflow
  15. 2:00:0915. An introduction to PySpark
  16. 2:07:5016. Final project 1: weather pipeline, extract and validate
  17. 2:22:3117. Final project 2: load, report, test and rerun
  18. 2:33:4218. What's next: an honest map

What's in the zip

  • requirements.txt, check_setup.py – setup (ch 1)
  • make_data.py, explore.py, formats.py – data generation and file formats (ch 2-3)
  • transform_pandas.py, transform_polars.py, race.py, cleaning.py – transforms and validation (ch 4-6)
  • db_sqlite.py, db_postgres.py, pg.env – SQL from Python (ch 7-8)
  • mock_api.py, ingest_api.py, ingest.py, config.toml – API ingestion, logging and retries (ch 9-10)
  • pipeline/, tests/ – ETL structure and pytest (ch 11-12)
  • run_pipeline.sh, schedule_loop.py, airflow/dags/orders_dag.py – cron and Airflow (ch 13-14)
  • spark_orders.py – PySpark (ch 15)
  • weather/ – final project (ch 16-17)
  • README.md, HOW_TO_RUN.txt, LICENSE, MANIFEST.sha256

How to run

Python 3.11+ (tested with 3.13.5). Java 17/21 for PySpark, PostgreSQL 17 for chapter 8. Linux or macOS; on Windows use WSL (Airflow does not run natively on Windows).

uv venv --python 3.13 .venv && source .venv/bin/activate
uv pip install -r requirements.txt   # or: python -m venv .venv ; pip install -r requirements.txt
python check_setup.py
python make_data.py

Airflow uses its own virtual environment. See HOW_TO_RUN.txt for every chapter's exact commands.

Data

No third-party datasets are bundled. Practice orders are synthetic, generated by make_data.py with a fixed seed. The final project fetches historical weather from the Open-Meteo API: weather data by Open-Meteo.com, licensed under CC BY 4.0. The mock API in chapters 9-14 runs locally and fails on purpose to practise retries.

Download source code

File: python-for-data-engineering-code.zip

Size: 25.3 KB (25,865 bytes)

SHA-256:
90f23fcc13272125315d91b716606290688585341c99f116f35ffa738e584d03

Licence: code is released under the MIT licence, copyright PyInns. See LICENSE and README inside the zip. Free learning code, not production code.