Common developer workflows
Assumes Prerequisites & setup is done. This page covers two things: the deployment scenarios you'll actually run day to day, and what each script under scripts/ is for.
Deployment scenarios
| Scenario | Command | When |
|---|---|---|
| Standard run | dbt run --target <staging|prod> | New or changed non-incremental models; incremental models pick up only new/changed rows. |
| Full refresh, all models | dbt build --target <staging|prod> --full-refresh (or make dbt-full-refresh-stock[-prod]) | An incremental model's logic changed in a way that invalidates already-materialized rows. |
| Full refresh, one model | dbt run --select <model_name> --target <staging|prod> --full-refresh | Same, but scoped to a single model instead of the whole project. |
| Full refresh, model + downstream | dbt run --select <model_name>+ --target <staging|prod> --full-refresh | The model's own logic changed and downstream models depend on the corrected output. |
See Running dbt locally for the full --select syntax and project vars.
Full refresh is destructive — know what you're resetting
A full refresh drops and rebuilds an incremental model from its source query, rather than merging in just the new rows. For the stock project specifically:
dbt-full-refresh-stock/dbt-full-refresh-stock-prodresetdeactivated_atandis_soldon every row — this is called out directly in the Makefile's own comment on those targets.- Anything that depended on those columns holding their prior history loses it the moment the refresh runs.
Run a full refresh on prod deliberately, not as a first troubleshooting step. If you only need to fix one model's output, scope the refresh to that model with --select rather than refreshing everything.
Scripts (scripts/)
Each subdirectory is a standalone operational tool, not part of the dbt build. Deeper runbooks for the ones that need step-by-step instructions live in the Operations section; this table is the map.
| Directory | What it does | How to run |
|---|---|---|
full-stock-sync/ | Runs the Stock CDC release process end to end: ingest a stock CSV into STOCK_RAW, full-refresh the CDC models, seed stock_change_history from the historical seed table, validate row counts. | ./main-ingest.sh [staging|prod] [--no-clean] [--disable-chunking] [--skip-dbt] [--yes] |
full-redbook-sync/ | Fetches redbook_vehicles from Manifold MySQL (prod) to CSV, then loads it into Snowflake. | ./fetch-redbook-data.sh then ./load-redbook-csv.sh <staging|prod> [--skip-dbt] |
stock-compare/ | Compares stock counts between Snowflake (STOCK_INT) and Manifold (stock_v2.stock) to find where records diverge. Plain count comparison by default; --detail traces the funnel across raw → staging → intermediate to isolate the stage where rows are lost. Requires a .env with Manifold MySQL credentials and the snowflake-drive-prod-admin Snow CLI connection. | ./compare-stock.sh [prod] [--detail] |
snowflake-dead-tables/ | Finds tables/views with no dbt lineage and no recent query activity, by combining the dbt manifest, a live inventory of Snowflake objects, and access history. Three scripts run in sequence: pull the dbt manifest, pull the Snowflake object inventory, then diff them against access history. | ./get_manifest_json.sh [connection] [db] [schema] [project] → ./get_snowflake_objects.sh [connection] [raw_db] [processed_db] → ./snowflake_dead_object_audit.sh [model_db] [source_db] [excluded_schemas] |