AWS resources

The AWS-side building blocks Snowflake's ingestion depends on, and the design patterns behind them. For the end-to-end path these compose into, see Data flow & failure modes.

S3 buckets

Terraform only reads the two source buckets (data "aws_ssm_parameter" lookups on their id/ARN) — it doesn't create or manage them. They belong to whatever repo owns the dealer/Manifold ingestion side.

Bucket provisioning pattern

Every bucket this repo owns (bronze, export) is provisioned through a shared, versioned Terraform Registry module — drive-terraform-modules/drive-s3-module — rather than a hand-rolled aws_s3_bucket resource per use case. One module call sets:

• Private by default: all four public-access-block settings (block_public_acls, block_public_policy, ignore_public_acls, restrict_public_buckets) on, s3_has_public_access = false.
• Versioning enabled.
• Lifecycle rules tuned for a transient landing zone, not a system of record: transition to STANDARD_IA at 30 days, expire at 90 days, abort incomplete multipart uploads after 7 days.

The design intent: bronze data only needs to exist long enough for Snowpipe to pick it up and dbt to land it in STOCK_RAW. Once it's in Snowflake, the S3 copy is a 90-day safety net, not the source of truth — the lifecycle rules encode that directly instead of relying on someone remembering to clean it up.

Per-provider event routing, without per-provider Terraform

The bronze bucket's aws_s3_bucket_notification doesn't hardcode one filter per dealer-stock provider. It builds one lambda_function notification block per entry in var.stock_providers, each scoped by filter_prefix (the provider's S3 folder) and filter_suffix (its file extension), all invoking the same router Lambda.

Onboarding a new provider is a variable-map entry, not a new Terraform resource — the routing scales with the provider list, not with hand-written notification blocks.

The two-Lambda ingestion pattern

Two Lambdas split the work on the stock path, and the split is deliberate:

stockInboundQueue makes one routing decision per object, based on folder convention, not provider:

• If the object's parent folder is complete or unprocessable, it's a file a provider has already finished with — the function syncs it to Google Drive for record-keeping and stops. It does not re-enter the pipeline.
• Everything else is a fresh file: the function drops a small JSON pointer ({ bucket, key, eventName }) onto the SQS queue and returns immediately.

This is what the two-Lambda split buys: S3 can deliver a burst of ObjectCreated events all at once (a dealer uploading their full catalogue), and the router's job for each one is a cheap decision plus an SQS write — well within a 120s timeout even under load. The actual parsing and column-mapping work, which can run long, happens in stockInboundProcess off the back of the queue, decoupled from how bursty the S3 event stream is.

SQS: ordering scope and how it's enforced

stockProcessQueue is FIFO, but the ordering guarantee is scoped narrower than "the whole queue":

MessageGroupId = "<provider>/<dealerId>"

Messages for the same provider and dealer are strictly ordered; different dealers and different providers process concurrently. This is the trade-off FIFO ordering is actually paying for here — correctness for one dealer's sequence of file updates, not a global processing order, which would throttle throughput for no benefit.

ContentBasedDeduplication is off; the producer supplies an explicit MessageDeduplicationId per send (a fresh random UUID, not derived from the object key or its content).

Queue configuration ties directly to the consumer's needs: VisibilityTimeout: 900 matches stockInboundProcess's own 900s timeout, so SQS won't hand the same message to a second invocation while the first is still legitimately working on it. MessageRetentionPeriod is 7 days on the working queue, 14 days on the DLQ — enough headroom to investigate and manually redrive a stuck message before it's gone for good.

Partial-batch failure handling

stockInboundProcess consumes one message per invocation (batchSize: 1) with functionResponseType: ReportBatchItemFailures, and triages errors into two categories before deciding whether SQS should retry:

• Known, expected failures (an empty CSV, a file that's already gone) are logged, recorded as a CloudWatch custom metric (STOCK_CSV_EMPTY / STOCK_CSV_NOT_FOUND), and the message is treated as successfully processed — no retry, no DLQ. There's nothing a retry would fix.
• Unexpected exceptions are returned as a batchItemFailure, which puts the message back on the queue for another attempt. After maxReceiveCount: 3 failed attempts, it lands in the DLQ instead of retrying forever.

The distinction matters: without it, a permanently-empty file would retry 3 times and dead-letter for no reason, burying real failures in DLQ noise.

IAM

IAM roles in this repo exist almost entirely to let a Snowflake storage integration read from or write to a specific S3 bucket — one role per Snowpipe/export-S3 module instance, plus the trust-policy patch described in Snowflake resources & RBAC.

CloudWatch

• A dashboard (main.cloudwatch.tf) gives passive visibility into the pipeline.
• A QueueAgeAlarm and a DLQMessagesAlarm on the stock queue.

See also

• Data flow & failure modes
• Secrets & SSM conventions