Provider column mapping

Every dealer stock feed lands as a CSV with whatever column names its source system happens to use. Getting that into a common shape is the job of three cooperating design patterns inside stockInboundProcess, plus one class per provider. This page covers all of it: the patterns, the full canonical field list, and enough real provider examples to see the actual variety, not just one representative case.

The three patterns, in the order a file passes through them

A message off the queue moves through three cooperating patterns before anything gets uploaded: validate, select, process. Each one gets its own small diagram below, in order.

1. Chain of Responsibility — deciding whether a file is even valid

putStockHandler.ts builds a ValidationChain once, at module load:

const validationChain = ValidationChain.createDefault()
  .addValidator(new FileValidator())
  .addValidator(new ProviderValidator());

Each validator gets a turn; the chain stops at the first failure:

validate(objectKey: string): ValidationResult {
  for (const validator of this.validators) {
    const result = validator.validate(objectKey);
    if (!result.isValid) {
      return result;
    }
  }
  return { isValid: true };
}

• FileValidator checks the extension is .csv or .txt (ALLOWED_FILE_EXTENSIONS) and that the key sits under one of the allowed provider folders (ProviderConstants.isInAllowedFolder).
• ProviderValidator extracts the provider name (the first path segment) and checks it against ALLOWED_STOCK_PROVIDERS — the same 12-provider list FileValidator used for folder matching.

A file failing either check is logged and dropped — processRecord() returns an empty result array, no processor is ever selected.

2. Simple Factory — picking which provider handles the file

ProcessorFactory holds one pre-instantiated singleton per provider, built once at module load:

private static processors: IStockProcessor[] = [
  new AutograbProvider(),
  new CarsalesProvider(),
  new CarmaProvider(),
  // ... 9 more
];

static getProcessor(objectKey: string): IStockProcessor | null {
  return this.processors.find(processor => processor.canProcess(objectKey)) || null;
}

canProcess() (defined once, on the shared base class) just checks the key starts with <providerName>/ and has a valid extension — so selection is really a linear scan for the first folder-prefix match, not a lookup table. With 12 providers this never matters for performance; it matters for understanding that provider order in the array is otherwise irrelevant, since folder prefixes don't overlap.

3. Template Method — the actual processing algorithm

BaseProviderProcessor.process() defines a fixed algorithm every provider runs through unchanged. Providers only fill in the blanks:

public async process(context: StockProcessingContext): Promise<ProcessResult> {
  const salesforceId = this.extractSalesforceId(objectKey);
  const fileName = this.extractFileName(objectKey);

  const { content, versionId } = await this.downloadCsvWithCompleteFallback(s3Client, sourceBucket, objectKey);
  const csvRows = await parseCsvContent(content);

  const transformedHeaders = this.transformHeaders(csvRows[0]);   // provider-specific mapping
  const originalHeaders = this.sanitizeHeaders(csvRows[0]);

  const dealerId = extractDealerIdFromFilename(fileName);
  let jsonData = convertRowsToJson(csvRows, transformedHeaders, originalHeaders, dealerId);

  jsonData = await this.enrichData(jsonData, context);            // hook — no-op unless overridden

  const targetKey = this.generateTargetKey(salesforceId, versionId);
  await uploadJsonToS3(s3Client, targetBucket, targetKey, JSON.stringify(jsonData, null, 2));

  return { success: true, sourceKey: objectKey, targetKey, salesforceId, message: "OK" };
}

Three methods are abstract — every provider must implement them:

One method is overridable and only one provider overrides it:

• enrichData(jsonData, context) — default is a no-op (return jsonData). Only AutograbProvider overrides it, for the AGVI lookup — see Enrichment.

No provider overrides extractSalesforceId, extractFileName, or generateTargetKey — every one of the 12 providers uses the base class's default implementation for those three hooks unchanged.

The complete/unprocessable fallback: recovering from a router/processor race

downloadCsvWithCompleteFallback() doesn't just fetch the S3 object at the key the SQS message points to. It tries three locations in order:

1. The original key.
2. If that 404s (NoSuchKey), the same filename under a sibling complete/ folder.
3. If that also 404s, the same filename under a sibling unprocessable/ folder.

This exists because of a race with the router. stockInboundQueue treats any file already sitting in a complete/ or unprocessable/ folder as done — it archives that file to Google Drive and never queues it for processing (see Ingestion overview). But a file can be moved into one of those folders by an external process between the time the original ObjectCreated event fires and the time stockInboundProcess actually gets around to downloading it — SQS delivery isn't instantaneous, and a dealer's own feed tooling can relocate a file once it's picked it up. Without this fallback, that race would silently fail the whole record with a NoSuchKey error. With it, the processor keeps looking in the two places the file could legitimately have moved to.

The full canonical field list

RAW_STOCK_FIELDS (platform/serverless/types/RawStockFields.ts) is the complete set every provider's ColumnMapping maps into — 46 fields:

DATA_PROVIDER_STOCK_ID, PLATE, MAKE, MODEL, VARIANT, BODY, ODOMETER, TRANSMISSION,
FUEL, YEAR, EXTERIOR_COLOUR, COMMENTS, PRICE_IGC, PRICE_EGC, REDBOOK_VEHICLE_KEY,
VIN, STOCK_TYPE, SELLER_TYPE, STATUS, IS_DAP_CALCULATED, PROCESSING_STATUS, IDENTIFIER,
TYPE, SERIES, TITLE, DRIVE_TYPE, EXTERIOR_COLOUR_SPECIFIC, INTERIOR_COLOUR,
REGISTRATION_EXPIRY, BUILD_YEAR, BUILD_MONTH, COMPLIANCE_YEAR, COMPLIANCE_MONTH,
CERTIFICATIONS, PHOTOS, DEALER_NAME, ENGINE_NUMBER, OPTIONS, NVIC_CODE, IMAGE_COUNT,
LITRES, DOORS, SEATS, GEARS, CYLINDERS, LOCATION, AGVI

No provider maps all 46 — each only maps what its own feed actually contains. The file's own header comment says this list is sourced from /src/dbt/migrations/R__create_raw_stock_table.sql; the real migration is src/stock/dbt/migrations/V1__create_stock_raw_table.sql (a versioned file, not a repeatable one, and under src/stock/dbt/migrations) — the comment has drifted from the actual path, but the field list itself still matches STOCK_RAW's real columns.

Three providers, from simplest to most complex

Carma (CarmaProvider.ts, 31 lines total) — a provider with no enrichment is genuinely this small, nothing hidden:

export class CarmaProvider extends BaseProviderProcessor {
  protected getProviderName(): string { return "carma"; }

  protected getColumnMapping(): ColumnMapping {
    return {
      'stockno': RAW_STOCK_FIELDS.DATA_PROVIDER_STOCK_ID,
      'make': RAW_STOCK_FIELDS.MAKE,
      'model': RAW_STOCK_FIELDS.MODEL,
      'variant': RAW_STOCK_FIELDS.VARIANT,
      'yeargroup': RAW_STOCK_FIELDS.YEAR,
      'buildmonth': RAW_STOCK_FIELDS.BUILD_MONTH,
      'odometer': RAW_STOCK_FIELDS.ODOMETER,
      'bodycolour': RAW_STOCK_FIELDS.EXTERIOR_COLOUR,
      'egcprice': RAW_STOCK_FIELDS.PRICE_EGC,
      'listingtype': RAW_STOCK_FIELDS.STOCK_TYPE,
      'vinnumber': RAW_STOCK_FIELDS.VIN,
      'comments': RAW_STOCK_FIELDS.COMMENTS,
      'nvic': RAW_STOCK_FIELDS.NVIC_CODE,
      'photourllist': RAW_STOCK_FIELDS.PHOTOS,
    };
  }

  protected getAllowedFolders(): string[] { return ["carma"]; }
}

Carsales (CarsalesProvider.ts) — same shape, more fields mapped, and its mapping is explicitly ported from a legacy system:

protected getColumnMapping(): ColumnMapping {
  return {
    // Core drive-axle aligned fields (from drive-axle CarsalesStockItem.php)
    'stocknumber': RAW_STOCK_FIELDS.DATA_PROVIDER_STOCK_ID,
    'registrationnumber': RAW_STOCK_FIELDS.PLATE,
    'specificationcode': RAW_STOCK_FIELDS.REDBOOK_VEHICLE_KEY,
    'make': RAW_STOCK_FIELDS.MAKE,
    'model': RAW_STOCK_FIELDS.MODEL,
    'badge': RAW_STOCK_FIELDS.VARIANT,
    'bodystyle': RAW_STOCK_FIELDS.BODY,
    'odometer': RAW_STOCK_FIELDS.ODOMETER,
    'transmission': RAW_STOCK_FIELDS.TRANSMISSION,
    // ...
  };
}

Several other providers' mapping comments cite the same origin (drive-axle's PHP entity classes) — these column mappings predate this repo; they weren't designed fresh here.

Autograb (AutograbProvider.ts, 216 lines) — the one provider that's actually large, because it's the only one carrying enrichment logic on top of the standard mapping. See Enrichment for what that does.

Onboarding a new provider is three separate pieces of work

1. A Terraform entry in var.stock_providers (so S3 routes the file to stockInboundQueue at all — see AWS resources).
2. A new <Name>Provider.ts class implementing the three abstract methods, registered in ProcessorFactory.processors.
3. The new provider name added to ALLOWED_STOCK_PROVIDERS (platform/serverless/constants/providers.ts) — without this, ProviderValidator rejects the file before ProcessorFactory ever gets a chance to select a processor for it, regardless of whether step 2 was done correctly.

Missing any one of the three produces a different, non-obvious failure: missing (1) means the file never leaves the dealer bucket; missing (2) means ProcessorFactory.getProcessor() returns null and the record fails with "No processor found"; missing (3) means the file is rejected by validation before the factory is even consulted.

docs/stock_field_mapping.md in the repo is a rough working note (broken snippets, no structure) — it doesn't reflect the current mapping reliably. Read the *Provider.ts files directly rather than that file.

See also

• Enrichment
• Queue, retries & DLQ
• AWS resources — the S3/Lambda wiring that gets a file to this code