Redbook feed ingestion
For the Snowpipe/bucket wiring, see Data flow & failure modes — this page covers what the copy Lambda actually does to each file.
Trigger: reactive, not scheduled
copyRedBookToBronzeBucket looks like a weekly cron job — its own description string says "Copy weekly RedBook CSV files to Bronze bucket" — but it isn't one. It's an S3 ObjectCreated event handler on the Manifold bucket, exactly like the stock path. The Lambda fires whenever Manifold happens to write a matching file; "weekly" describes how often Manifold produces one, not how this function is triggered.
The filter is enforced twice, independently, using the same 5 filenames in two different places:
- In Terraform (
serverless.yml): five separate S3 notification blocks, one per exact filename, each withprefix: redbook/and asuffixmatching one target file. - In the Lambda itself: a
TARGET_FILESarray with the same 5 names, checked again before processing.
Anything that doesn't match either filter is skipped and logged — not an error, just ignored. The double enforcement means a change to one list without the other only has an effect at whichever layer is stricter; they should be kept in sync manually, there's no shared source for this list between Terraform and the Lambda.
VEVehicle.csv
VEVehicleMap.csv
VEMake.csv
VEFamily.csv
RBCipherKey.txtIt renames on the way through, and it isn't a 1:1 copy
Each matched file is written to the redbook-bronze bucket under a new key:
{basename}.{yyyy-mm-dd-hh-mm}.csvThe timestamp comes from the S3 event's eventTime, not anything inside the file. Note that the extension is always rewritten to .csv — including for RBCipherKey.txt, which arrives as .txt and leaves as .csv. If you're checking a Snowpipe's file-format pattern against what's actually landing in the bucket, this is the detail that matters: the cipher-key pipe reads a .csv-named file whose content is unchanged text, not CSV that was reformatted.
Otherwise, this is a byte-for-byte CopyObjectCommand — no header inspection, no column parsing, no encoding conversion happens in this Lambda. The column layout of these files is defined by Redbook/Manifold on the source side, not anything in this repo.
Why the rename doesn't break the Snowpipe match
Each of the 5 Snowpipes matches files by a regex PATTERN against the filename in the bronze bucket, not an exact name — for example, the legacy map pipe's copy statement has:
PATTERN = '.*VEVehicleMap.*\.csv'The .* wildcards on both sides are exactly what let the renamed, timestamp-injected filename (VEVehicleMap.2026-01-15-08-30.csv) still match — the pattern only requires VEVehicleMap to appear as a substring, so the rename doesn't need to preserve any particular filename shape, just keep the original basename intact somewhere in the string. All 5 pipes share one SNS topic (redbook_bronze_bucket_notifications) for auto-ingest notifications; the PATTERN clause is what routes each file to the one pipe that should actually load it.
A bare return can skip a file in a multi-record batch
The Lambda's per-record loop has a bare return where a skipped (non-target) file is logged — that return exits the whole handler function, not just that loop iteration. If a single S3 event batch ever contains more than one record and a non-target file appears before a target file in that batch, the target file would never be processed. Whether this trigger ever actually delivers multi-record batches in practice isn't confirmed from the code alone — worth checking against real S3 event delivery for this bucket before treating it as inert.