Apache Kafka for Data Engineering - Full Course - how to run the course code
============================================================================
Tested with: Python 3.13, Java 21, Apache Kafka 4.4.0 (Scala 2.13 build), confluent-kafka 2.16.1, pyspark 4.3.0.
Linux or macOS, or WSL on Windows. Run every command from this folder, in a bash terminal.

1. Download and unpack Kafka into a folder called kafka:
   curl -O https://archive.apache.org/dist/kafka/4.4.0/kafka_2.13-4.4.0.tgz
   tar -xzf kafka_2.13-4.4.0.tgz && mv kafka_2.13-4.4.0 kafka && rm kafka_2.13-4.4.0.tgz
2. python3 -m venv .venv && source .venv/bin/activate
   pip install confluent-kafka==2.16.1 pyspark==4.3.0
3. python get_jars.py            (Spark Kafka connector jars -> jars/, once)
4. source env.sh                 (in every new terminal)
5. bash start_kafka.sh           (formats kafka-data the first time, then starts the broker)
6. Create the topics:
   kafka-topics.sh --bootstrap-server $BS --create --topic hello --partitions 1
   kafka-topics.sh --bootstrap-server $BS --create --topic keys-demo --partitions 3
   kafka-topics.sh --bootstrap-server $BS --create --topic orders --partitions 3
   kafka-topics.sh --bootstrap-server $BS --create --topic orders-dlq --partitions 1
   kafka-topics.sh --bootstrap-server $BS --create --topic orders-enriched --partitions 3
7. Python, in the order of the course:
   python hello_producer.py ; python hello_consumer.py
   python key_demo.py ; python orders.py
   python producer.py 300 ; python consumer.py 120 ; python consumer.py
   python group_consumer.py A 60   (and in a second terminal: python group_consumer.py B 60)
   python producer.py 50 251       (duplicates on purpose) ; python consumer.py
   python send_bad.py ; python processor.py ; python enrich.py
   python shop_stream.py 1001 10 30
8. Spark Structured Streaming:
   python stream_console.py
   python stream_job.py            (available-now run; or: python stream_job.py 90 for 90 s, micro-batch every 5 s)
   python window_job.py
   python report.py                (lake/gold/city_daily_revenue and reports/city_daily_revenue/*.csv)
9. Final live run: terminal 1: python stream_job.py 90   terminal 2: python shop_stream.py 2001 10 40
   then: python report.py

Notes
- Kafka and Spark need Java 17 or newer (JAVA_HOME set, or java on PATH).
- Stop Kafka: kafka-server-stop.sh
- To start over: stop Kafka, then rm -rf kafka-data lake checkpoints reports
