Apache Kafka for Data Engineering - Full Course (Python, KRaft, Spark Streaming Project) – Source Code
Learn Apache Kafka for data engineering from zero, locally: Kafka 4.4 in KRaft mode, producers and consumers in Python with confluent-kafka, keys, delivery guarantees, consumer groups and rebalancing, offsets and replay, a dead-letter topic, exactly-once with transactions, and a project where a live order stream flows through Spark Structured Streaming into partitioned Parquet with checkpoints and a report-ready table.
- Duration
- 2:27:24 (22 chapters)
- Tools
- Apache Kafka 4.4.0 (KRaft), Python 3.13, confluent-kafka 2.16.1, PySpark 4.3.0, Java 21, Parquet, VS Code, bash
Watch the course
Chapters
- 0:00:0001. Welcome: ShopFlow goes real time, and the plan
- 0:05:2902. Setup: Kafka 4.4, Python, confluent-kafka and PySpark
- 0:13:4203. Kafka concepts: brokers, topics, partitions, offsets, groups, KRaft
- 0:19:4704. Running Kafka locally in KRaft mode
- 0:28:1505. Topics and the console tools
- 0:35:0006. Your first Python producer
- 0:39:5107. Your first Python consumer, offsets and lag
- 0:45:4808. Keys, partitions and ordering
- 0:51:3109. JSON events and a schema
- 0:59:2110. A safe producer: acks, idempotence and batching
- 1:06:4011. Manual commits and at-least-once processing
- 1:13:3212. Consumer groups and rebalancing
- 1:20:3113. Offsets, lag, replay and retention
- 1:27:0614. Error handling and a dead-letter topic
- 1:36:0115. Stream processing: consume, transform, produce, exactly once
- 1:43:5916. The live ShopFlow event stream
- 1:49:3917. PySpark Structured Streaming from Kafka
- 1:57:2018. Streaming into partitioned Parquet with checkpoints
- 2:04:5219. Event-time windows and watermarks
- 2:10:5420. Final project: live orders to Parquet and a report
- 2:17:3921. Interview questions about Kafka
- 2:23:2622. Recap and next steps
What's in the zip
kafka-for-data-engineering/HOW_TO_RUN.txtkafka-for-data-engineering/LICENSEkafka-for-data-engineering/MANIFEST.sha256kafka-for-data-engineering/README.mdkafka-for-data-engineering/concepts.mdkafka-for-data-engineering/consumer.pykafka-for-data-engineering/enrich.pykafka-for-data-engineering/env.shkafka-for-data-engineering/get_jars.pykafka-for-data-engineering/group_consumer.pykafka-for-data-engineering/hello_consumer.pykafka-for-data-engineering/hello_producer.pykafka-for-data-engineering/kafka.propertieskafka-for-data-engineering/key_demo.pykafka-for-data-engineering/orders.pykafka-for-data-engineering/processor.pykafka-for-data-engineering/producer.pykafka-for-data-engineering/report.pykafka-for-data-engineering/send_bad.pykafka-for-data-engineering/shop_stream.pykafka-for-data-engineering/spark_common.pykafka-for-data-engineering/start_kafka.shkafka-for-data-engineering/stream_console.pykafka-for-data-engineering/stream_job.pykafka-for-data-engineering/window_job.py
How to run
Python 3.11+ (tested with 3.13), Java 17 or 21 (tested with 21). Linux or macOS; on Windows use WSL. Run every command from the course folder in a bash terminal.
curl -O https://archive.apache.org/dist/kafka/4.4.0/kafka_2.13-4.4.0.tgz
tar -xzf kafka_2.13-4.4.0.tgz && mv kafka_2.13-4.4.0 kafka
python3 -m venv .venv && source .venv/bin/activate
pip install confluent-kafka==2.16.1 pyspark==4.3.0
python get_jars.py
source env.sh
bash start_kafka.sh
kafka-topics.sh --bootstrap-server $BS --create --topic orders --partitions 3
python producer.py 300
python stream_job.py
python report.pySee HOW_TO_RUN.txt for every topic to create, the step-by-step commands from the video, the final live run and how to start over.
Data
No datasets are bundled. orders.py, producer.py and shop_stream.py generate fake ShopFlow orders; Kafka runs locally in KRaft mode and Spark writes a local Parquet lake. Kafka itself and the Spark connector jars are downloaded by the commands above, not bundled. No cloud account and no API key.
File: kafka-for-data-engineering-code.zip
Size: 17.0 KB (17,386 bytes)
SHA-256:49a8ccf853619a9788fddb47988ce5b7095b0a6ab22ffae2d960309ea752a3b9
Licence: code is released under the MIT licence, copyright PyInns. See LICENSE and README inside the zip. Free learning code, not production code.