Ollama Python Tutorial: Run Local LLMs with Chat, Streaming, Structured Outputs, Tools and Embeddings (2026) Python sqlite3 Tutorial: Tables, Transactions, JSON and Upserts with the Built-in Database (2026) Convert Python to EXE with PyInstaller: One-File Apps, Data Files and Hidden Imports (2026) Two-Factor Authentication in Python with pyotp: TOTP, QR Codes and FastAPI (2026) Hash Passwords in Python with Argon2: pwdlib, argon2-cffi and bcrypt Migration (2026) Encrypt and Decrypt in Python with Fernet: Strings, Files and Passwords (2026) psutil Tutorial: Monitor CPU, Memory, Disk and Processes in Python (2026) PyMuPDF Tutorial: PDF to Image, Extract Text and Images in Python (2026) MoviePy 2 Tutorial: Edit Videos with Python (Trim, Text, Merge, Audio) 2026 FastAPI Dependency Injection with Depends: Annotated, Yield and Overrides (2026) HTML to PDF in Python with WeasyPrint and Jinja2: Invoices and Reports (2026) Extract Tables from PDF in Python with pdfplumber: CSV, Excel and pandas (2026) OCR in Python: Extract Text from Images and Scanned PDFs with RapidOCR (2026) Text to Speech in Python with Piper: Offline Natural Voices, WAV Files and a TTS API (2026) Speech to Text in Python with faster-whisper: Transcribe Audio and Make Subtitles (2026) Send Email with Python: smtplib, HTML, Attachments and Bulk Mail (2026) Python QR Code Generator: Styled QR Codes with qrcode (2026) Pillow 13: Resize, Crop and Watermark Images in Python (2026) Parallel Programming with Dask in Python 2026: Intro Guide MarkItDown: Convert Files to Markdown for LLMs in Python (2026) Plotly Express: Interactive Charts in Python (2026) pandera: Validate pandas DataFrames in Python (2026) jsonschema: Validate JSON Data in Python (2026) Faker: Generate Fake Test Data in Python (2026) RapidFuzz: Fuzzy String Matching in Python (2026) sqlglot: Parse and Transpile SQL in Python (2026) structlog: Structured Logging in Python (2026) python-pptx in 2026: Automate PowerPoint Reports with Python python-docx: Create and Edit Word Documents with Python (2026) pypdf: Merge, Split and Edit PDFs with Python (2026) View all in this category →

Apache Kafka for Data Engineering - Full Course (Python, KRaft, Spark Streaming Project) – Source Code

Learn Apache Kafka for data engineering from zero, locally: Kafka 4.4 in KRaft mode, producers and consumers in Python with confluent-kafka, keys, delivery guarantees, consumer groups and rebalancing, offsets and replay, a dead-letter topic, exactly-once with transactions, and a project where a live order stream flows through Spark Structured Streaming into partitioned Parquet with checkpoints and a report-ready table.

Duration
2:27:24 (22 chapters)
Tools
Apache Kafka 4.4.0 (KRaft), Python 3.13, confluent-kafka 2.16.1, PySpark 4.3.0, Java 21, Parquet, VS Code, bash

Watch the course

Watch on YouTube

Chapters

  1. 0:00:0001. Welcome: ShopFlow goes real time, and the plan
  2. 0:05:2902. Setup: Kafka 4.4, Python, confluent-kafka and PySpark
  3. 0:13:4203. Kafka concepts: brokers, topics, partitions, offsets, groups, KRaft
  4. 0:19:4704. Running Kafka locally in KRaft mode
  5. 0:28:1505. Topics and the console tools
  6. 0:35:0006. Your first Python producer
  7. 0:39:5107. Your first Python consumer, offsets and lag
  8. 0:45:4808. Keys, partitions and ordering
  9. 0:51:3109. JSON events and a schema
  10. 0:59:2110. A safe producer: acks, idempotence and batching
  11. 1:06:4011. Manual commits and at-least-once processing
  12. 1:13:3212. Consumer groups and rebalancing
  13. 1:20:3113. Offsets, lag, replay and retention
  14. 1:27:0614. Error handling and a dead-letter topic
  15. 1:36:0115. Stream processing: consume, transform, produce, exactly once
  16. 1:43:5916. The live ShopFlow event stream
  17. 1:49:3917. PySpark Structured Streaming from Kafka
  18. 1:57:2018. Streaming into partitioned Parquet with checkpoints
  19. 2:04:5219. Event-time windows and watermarks
  20. 2:10:5420. Final project: live orders to Parquet and a report
  21. 2:17:3921. Interview questions about Kafka
  22. 2:23:2622. Recap and next steps

What's in the zip

  • kafka-for-data-engineering/HOW_TO_RUN.txt
  • kafka-for-data-engineering/LICENSE
  • kafka-for-data-engineering/MANIFEST.sha256
  • kafka-for-data-engineering/README.md
  • kafka-for-data-engineering/concepts.md
  • kafka-for-data-engineering/consumer.py
  • kafka-for-data-engineering/enrich.py
  • kafka-for-data-engineering/env.sh
  • kafka-for-data-engineering/get_jars.py
  • kafka-for-data-engineering/group_consumer.py
  • kafka-for-data-engineering/hello_consumer.py
  • kafka-for-data-engineering/hello_producer.py
  • kafka-for-data-engineering/kafka.properties
  • kafka-for-data-engineering/key_demo.py
  • kafka-for-data-engineering/orders.py
  • kafka-for-data-engineering/processor.py
  • kafka-for-data-engineering/producer.py
  • kafka-for-data-engineering/report.py
  • kafka-for-data-engineering/send_bad.py
  • kafka-for-data-engineering/shop_stream.py
  • kafka-for-data-engineering/spark_common.py
  • kafka-for-data-engineering/start_kafka.sh
  • kafka-for-data-engineering/stream_console.py
  • kafka-for-data-engineering/stream_job.py
  • kafka-for-data-engineering/window_job.py

How to run

Python 3.11+ (tested with 3.13), Java 17 or 21 (tested with 21). Linux or macOS; on Windows use WSL. Run every command from the course folder in a bash terminal.

curl -O https://archive.apache.org/dist/kafka/4.4.0/kafka_2.13-4.4.0.tgz
tar -xzf kafka_2.13-4.4.0.tgz && mv kafka_2.13-4.4.0 kafka
python3 -m venv .venv && source .venv/bin/activate
pip install confluent-kafka==2.16.1 pyspark==4.3.0
python get_jars.py
source env.sh
bash start_kafka.sh
kafka-topics.sh --bootstrap-server $BS --create --topic orders --partitions 3
python producer.py 300
python stream_job.py
python report.py

See HOW_TO_RUN.txt for every topic to create, the step-by-step commands from the video, the final live run and how to start over.

Data

No datasets are bundled. orders.py, producer.py and shop_stream.py generate fake ShopFlow orders; Kafka runs locally in KRaft mode and Spark writes a local Parquet lake. Kafka itself and the Spark connector jars are downloaded by the commands above, not bundled. No cloud account and no API key.

Download source code

File: kafka-for-data-engineering-code.zip

Size: 17.0 KB (17,386 bytes)

SHA-256:
49a8ccf853619a9788fddb47988ce5b7095b0a6ab22ffae2d960309ea752a3b9

Licence: code is released under the MIT licence, copyright PyInns. See LICENSE and README inside the zip. Free learning code, not production code.