SQL for Data Engineering - how to run the course code
======================================================
Tested with: PostgreSQL 17, DuckDB CLI 2.0.0, Python 3.13.5, psycopg 3.3.6. Linux/macOS, or WSL on Windows.
Run every command from this folder, in a bash terminal.

1) Local PostgreSQL server (ch 1) - a private server just for the course, no password (trust), local only
   initdb -D pgdata -U postgres --auth=trust
   echo "port = 5433" >> pgdata/postgresql.conf
   echo "unix_socket_directories = '/tmp'" >> pgdata/postgresql.conf
   pg_ctl -D pgdata -l pg.log start
   source pg.env
   createdb shop
   (stop it with: pg_ctl -D pgdata stop ; start it again at the start of every session)

2) Practice data and SQL basics (ch 2-10)
   psql -q -f shop/schema.sql ; psql -q -f shop/seed.sql
   then run any file with:  psql -q -f basics/select.sql   (and so on, folder by folder, in chapter order)

3) Staging, upserts, incremental loads (ch 11-12)
   python3 make_orders.py 2026-10-01
   psql -q -f loads/tables.sql
   psql -c "\copy stg_orders FROM data/orders_2026-10-01.csv CSV HEADER" ; psql -q -f loads/upsert.sql
   psql -q -f loads/watermark.sql
   python3 make_orders.py 2026-10-02
   psql -c "\copy stg_orders FROM data/orders_2026-10-02.csv CSV HEADER" ; psql -q -f loads/incremental.sql

4) SCD2, star schema, data quality (ch 13-15)
   psql -q -f scd/tables.sql ; bash scd/load.sh 2026-10-01 ; bash scd/load.sh 2026-10-02
   psql -q -f star/build.sql ; psql -q -f star/report.sql
   psql -q -f quality/checks.sql ; psql -q -f quality/gate.sql && echo "OK: publish the reports"

5) DuckDB (ch 16)
   for d in 2026-10-{01..21}; do python3 make_orders.py $d; done
   duckdb -f duck/explore.sql

6) Python ELT (ch 17)
   python3 -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt
   python elt.py 2026-10-03 2026-10-04

7) Final project (ch 18-19)
   createdb warehouse
   psql -q -d warehouse -f warehouse/schema.sql
   python run_warehouse.py 2026-10-01 2026-10-10      (one day or a range; safe to re-run any day)
   psql -q -d warehouse -f warehouse/report.sql
