Redbook Ingestion

Redbook has exactly one upstream source — Manifold — so ingestion here is one Lambda doing a copy-and-rename, not a fleet of per-provider parsers. This section is deliberately separate from Stock Ingestion rather than a subsection of it: the two paths solve genuinely different problems and share almost no architecture.

Why this isn't just a smaller version of Stock Ingestion

Stock Ingestion exists to absorb variety: 12 different dealer stock-management systems, each with its own column names, each needing validation, provider selection, and per-record enrichment. That's what the Chain of Responsibility, Factory, and Template Method machinery in Stock Ingestion is for.

Redbook has none of that problem to solve. There's one source, the files are already in a fixed, known shape, and nothing needs per-row transformation before it lands in Snowflake — the real transformation work (deduplication, standardisation, joins) happens entirely downstream in dbt, not in serverless code. So the ingestion Lambda here doesn't parse CSV rows, doesn't select between processors, and doesn't enrich anything — it copies whole files and renames them.

End-to-end: a Redbook file, from Manifold to STOCK_RAW's reference tables

1. The copy Lambda picks up a file and lands it in the bronze bucket:

2. The same shared topic fans out to five Snowpipes total. The two trickiest to tell apart — VEVehicle and VEVehicleMap share a prefix — need a disambiguating pattern:

The other three Snowpipes on the same topic — matching VEMake, VEFamily, and RBCipherKey — have no such ambiguity to resolve; see the full mapping just below.

One Lambda, one bucket, one shared SNS topic, five Snowpipes — each pipe's own PATTERN clause is the only thing deciding which of the five raw tables a given file's contents end up in. See Redbook feed ingestion for exactly how the rename works and why the patterns still match after it.

The filename-to-table mapping isn't 1:1 in the obvious way

VEVehicle.csv and VEVehicleMap.csv share a common prefix, so the plain-vehicles pipe's pattern uses a regex negative lookahead specifically to exclude the map file:

VEVehicle.csv     -> REDBOOK_VEHICLES_RAW           (pattern: .*VEVehicle(?!Map).*\.csv)
VEVehicleMap.csv  -> REDBOOK_LEGACY_MAP_RAW          (pattern: .*VEVehicleMap.*\.csv)
VEMake.csv        -> REDBOOK_VEHICLE_MAKES_RAW
VEFamily.csv      -> REDBOOK_VEHICLE_FAMILIES_RAW
RBCipherKey.txt   -> REDBOOK_CIPHER_KEY_RAW          (renamed to .csv on the way in — see feed ingestion)

Without that negative lookahead, VEVehicleMap.csv would match both the vehicles pipe's pattern and its own — the exclusion is deliberate, not incidental.

The enable_redbook_pipe flag: name and description are both wrong

Whether the Redbook Snowpipes auto-ingest at all is gated by a Terraform variable, enable_redbook_pipe — and both its name and its own description are wrong in ways worth knowing before you go looking for it:

• The variable's description field says "Whether to enable the STOCK_BRONZE_PIPE snowpipe for auto-ingestion" — STOCK_BRONZE_PIPE is the stock ingestion pipe, not any of the five Redbook pipes this variable actually controls. Copy-paste drift in the description, not a hint that it does something else.
• A code comment next to the default claims it's "Disabled in production for manual control" — but the default is true in both env-staging and env-prod, and neither environment overrides it. As checked in, Redbook auto-ingest is enabled everywhere, contradicting the comment.

At a glance

Start here

• How does the copy Lambda actually work? Redbook feed ingestion — the rename behaviour, the double filename enforcement, and why the rename doesn't break the Snowpipe match.