Spark Declarative Pipelines for Data Engineering - how to run the course code
==============================================================================
Tested with: Python 3.13, Java 21, pyspark[pipelines] 4.3.0, pandas below 3. Linux/macOS, or WSL on Windows.
Run every command from this folder, in a bash terminal.

1. python3 -m venv .venv && source .venv/bin/activate
2. pip install "pyspark[pipelines]==4.3.0" "pandas<3"
3. Edit spark-pipeline.yml: replace the storage placeholder with an absolute file:// folder on your machine, for example
   storage: file:///path/to/your/shop/pipeline-storage   (an absolute file:// path)
4. source env.sh          (do this in every new terminal)
5. spark-sql -e "CREATE DATABASE IF NOT EXISTS shop"
6. python make_data.py 2026-11-01
7. spark-pipelines dry-run
8. spark-pipelines run
9. python show.py bronze_orders silver_orders quarantine_orders gold_daily_revenue gold_city_revenue gold_top_products
10. Next day: python make_data.py 2026-11-02, then spark-pipelines run again (only new files are ingested).
    Other options: spark-pipelines run --refresh gold_city_revenue   /   spark-pipelines run --full-refresh-all

Notes
- The local Hive metastore (metastore_db, derby.log) allows one Spark process at a time: run commands one after another.
- To start over: rm -rf spark-warehouse metastore_db pipeline-storage derby.log data
