All Articles (30)
structlog: Structured Logging in Python (2026) python-pptx in 2026: Automate PowerPoint Reports with Python python-docx: Create and Edit Word Documents with Python (2026) pypdf: Merge, Split and Edit PDFs with Python (2026) openpyxl: Automate Excel Files with Python (2026) Hypothesis: Property-Based Testing in Python (2026) tiktoken: Count LLM Tokens in Python (2026) pydantic-settings: Typed Env Config in Python (2026) orjson: Fast JSON in Python (2026) PyArrow Tables + Parquet in Python (2026) SQLAlchemy 2.0 ORM in Python (2026): Mapped Models + Session Build Semantic Search with ChromaDB in Python (2026) Build ML Demos with Gradio in Python (2026) Build Interactive Python Web UIs with NiceGUI (2026) Build Interactive Web Apps with FastHTML (2026) Type-Check Python Fast with Astral ty (2026) Stream Large Responses with FastAPI JSON Lines (2026) Build Interactive UIs with FastAPI + HTMX (2026) Deploy FastAPI on Cloudflare Python Workers (2026) Serve a SPA with FastAPI app.frontend() (2026) Safe SQL & HTML with Python 3.14 t-strings (PEP 750) Hybrid Search in LanceDB with Vector + BM25 (Python 2026) Compose Agents with Pydantic AI v2 Capabilities (2026) Build Terminal UIs with Textual in Python (2026) Portable DataFrames with Ibis and DuckDB in Python (2026) Code Agents with Hugging Face smolagents in Python (2026) Constrained LLM Generation with Outlines in Python (2026) One DataFrame API for Pandas and Polars with Narwhals (2026) Turn PDFs into LLM-Ready Markdown with Docling (2026) Trace LLM Apps with Langfuse in Python (2026)

Data Engineering Full Course for Beginners (4.5 Hours) — SQL, Python, PostgreSQL, dbt, Airflow, Spark, Kafka – Source Code

A practical tour of the modern data engineering stack, all run locally: Python and SQL basics, PostgreSQL, data modelling, ETL, file formats, DuckDB/polars, data quality checks and incremental loads. It then adds Airflow orchestration, dbt transformations, PySpark and Kafka streaming. The final project ingests, models, tests and schedules millions of real NYC taxi trips.

Duration
4:35:04
Tools
Python 3.13.5, uv 0.13.0, SQLite 3.46.1, PostgreSQL 17, pandas 3.0.6, polars 2.0.0, DuckDB 1.5.6, PyArrow 26.0.0, psycopg 3.3.6, pytest 9.1.1, Apache Airflow 3.3.2, dbt-core 1.12.5 + dbt-duckdb 1.11.0, PySpark 4.2.0 (Java 21), Apache Kafka 4.3.1 (KRaft), confluent-kafka 2.16.0, VS Code 1.141

Watch the course

Chapters

  1. 0:00:0001. What data engineering is (and the plan)
  2. 0:07:3302. Setup: Python, a virtual environment and the tools
  3. 0:14:5103. Python for data work: reading a messy CSV
  4. 0:25:0004. Python for data work: functions, JSON and dates
  5. 0:35:0505. SQL basics with SQLite
  6. 0:47:1706. SQL joins
  7. 0:56:4407. SQL aggregations: GROUP BY, HAVING and CASE
  8. 1:05:5808. SQL window functions
  9. 1:15:5309. CTEs and subqueries
  10. 1:24:4210. PostgreSQL on your own machine
  11. 1:42:0211. Data modelling: normalisation
  12. 1:50:1112. Data modelling: star schema, facts and dimensions
  13. 2:00:5913. ETL with pandas
  14. 2:10:0214. File formats: CSV, JSON and Parquet
  15. 2:21:1315. Faster transforms with polars and DuckDB
  16. 2:31:5616. Data quality checks and tests
  17. 2:43:2317. Incremental loads and idempotency
  18. 2:55:5018. Orchestration with Apache Airflow
  19. 3:10:0519. Transformations with dbt
  20. 3:25:3620. Big data intro with PySpark
  21. 3:35:3721. Streaming with Apache Kafka
  22. 3:48:4922. Cloud data warehouses, explained honestly
  23. 3:56:4123. Final project 1: ingesting NYC taxi trips
  24. 4:07:4624. Final project 2: modelling and testing with dbt
  25. 4:23:0425. Final project 3: scheduling and dashboard queries
  26. 4:32:2626. What's next: an honest road map
Download source code

File: data-engineering-full-course-code.zip

Size: 65.5 KB (67,049 bytes)

SHA-256:
3e0c5de93e08e4e52083707352ed814bcd6f273aa0b1948d58282ccf95b421dd

Licence: code is released under the MIT licence, copyright PyInns. See LICENSE and README inside the zip. Free learning code, not production code.

Built-in Functions
abs() in Python 2026: Absolute Value, Complex Numbers & Modern Use Cases aiter() in Python 2026: Asynchronous Iterator Protocol & Modern Async Usage all() in Python 2026: Check If All Elements Are True + Modern Patterns & Use Cases anext() in Python 2026: Asynchronous Next + Modern Async Iterator Control any() in Python 2026: Check If Any Element Is True + Modern Patterns & Use Cases ascii() in Python 2026: Safe String Representation & Modern Debugging Use Cases bin() in Python 2026: Binary Representation, Bit Manipulation & Modern Use Cases bool() in Python 2026: Truthy/Falsy Conversion + Modern Patterns & Use Cases breakpoint() in Python 2026: Modern Debugging with PDB, IDEs & Best Practices Built-in Functions in Python 2026 – Complete Guide & Best Practices bytearray() in Python 2026: Mutable Binary Sequences + Modern Use Cases & Best Practices bytes() in Python 2026: Immutable Binary Sequences + Modern Use Cases & Best Practices callable() in Python 2026: Check If Object Is Callable + Modern Patterns & Use Cases chr() in Python 2026: Unicode Character from Integer + Modern Encoding Use Cases classmethod() in Python 2026: Class Methods, Alternative Constructors & Modern Best Practices