Stock Ingestion
platform/serverless is where a dealer's raw CSV, in whatever shape their stock-management system exports it, becomes rows in STOCK_RAW. Three Lambda functions do all of it — that's the entire serverless surface area for ingestion, there's nothing else. This section covers what each one does, how the application code inside them is structured, and how a file actually moves from a dealer's upload to a Snowflake table.
End-to-end: one dealer file, from upload to STOCK_RAW
Three stages, shown separately so each stays readable — they chain directly into each other in the order below.
1. A file lands, and the router decides what to do with it:
2. The processor picks it up and does the actual work:
stockInboundProcess is one box here on purpose — what it actually does internally (three cooperating design patterns) is its own set of diagrams in Provider column mapping.
3. From the bronze bucket, Snowpipe takes over:
The Redbook path is entirely separate — different bucket, different Lambda, no queue, no per-provider logic. See Redbook Ingestion.
The three functions, and what each one is actually for
| Function | Trigger | Role |
|---|---|---|
stockInboundQueue | S3 ObjectCreated on the dealer stock bucket, filtered per provider by Terraform's S3 notification config | Makes exactly one decision per file: archive it (already-processed folder) or hand it to the queue for real processing. Does no parsing itself. |
stockInboundProcess | SQS (stockProcessQueue), one message per invocation | Does the actual work: validates the key, picks a provider processor, downloads and parses the CSV, maps columns, runs enrichment, uploads JSON to the bronze bucket. |
copyRedBookToBronzeBucket | S3 ObjectCreated on the Manifold bucket, filtered to 5 exact Redbook filenames | Renames and copies a matched file straight into redbook-bronze. No parsing, no queue — Redbook has no per-provider logic to speak of, there's only one feed. |
Why split stockInboundQueue and stockInboundProcess into two functions instead of one: S3 can deliver a burst of ObjectCreated events all at once (a dealer uploading their whole catalogue), and the router's job for each one — a folder check plus an SQS write — comfortably fits inside a 120s timeout under that burst. The actual per-file work, which downloads, parses, and can call an external API, gets its own 900s budget on the other side of the queue, decoupled from how bursty the S3 event stream is. See AWS resources for the SQS/DLQ mechanics underneath this split.
At a glance
| Provider mapping | One TypeScript class per provider, 12 configured today |
| Selection pattern | Simple Factory (ProcessorFactory) over a Chain of Responsibility (ValidationChain) |
| Processing pattern | Template Method (BaseProviderProcessor.process()) |
| Canonical fields | RAW_STOCK_FIELDS, 46 fields, mirrors STOCK_RAW's columns |
| Enrichment | AGVI lookup (Autograb only) → NVIC fallback (downstream in dbt) → VIN lookup (every provider) — three attempts before a key is given up on |
| DLQ redrive tooling | None exists in this repo |
Start here
- Want the real architecture — the patterns, the fallback chain, the full field list? Provider column mapping — this is the deep-dive page for this section.
- Wondering how a Redbook Vehicle Key gets resolved during ingestion? Enrichment — what Autograb's AGVI lookup does and doesn't do.
- What happens when there's no key and no AGVI, only a VIN? VIN-based fallback (VIN lookup) — the last resolution attempt, and it applies to every provider.
- A file failed processing — what happens to it? Queue, retries & DLQ — the message shape, ordering guarantee, and which failures retry.
Looking for how Redbook data gets ingested instead? See Redbook Ingestion — a separate section, not a subsection here, because the two paths barely share any architecture.