Ingestion reliability and limits

Three different identities solve three different problems

Do not generate a fresh idempotency key for each retry. Persist the source identity and the exact serialized request until you receive a run reference or a non-retryable response.

Atomic append, upsert, and replace

Every accepted run validates before mutation. The commit transaction re-resolves the target, rechecks the principal's current grant, compares the target schema version, applies the full mutation set, and records the run's commit identity. Readers see the complete before-state or complete after-state, never a partially applied run.

• Append inserts every valid row in one atomic commit.
• Upsert requires non-empty declared keyColumns. They form the pipeline's logical uniqueness contract; a physical unique index is not required. Matching rows are updated and new keys are inserted in one commit.
• Duplicate key tuples within one submitted batch fail validation.
• Concurrent upserts serialize and schema/version-CAS at table scope; they do not create two current rows for one logical key.
• Replace atomically swaps the complete row set while preserving the Table node ID and current grants. It requires an admin grant because the commit deletes every prior row.

Revoking a grant while a run is queued prevents the final commit. A successful data commit remains successful if later staging cleanup has to retry.

Schema policy

Safe coercions must be configured explicitly, such as an exactly representable integer-to-float conversion or a declared RFC 3339 string-to-timestamp mapping. Ambiguous and lossy conversions fail. The validated schema version is checked again at commit; a concurrent incompatible schema change returns a stable conflict instead of committing stale assumptions.

{
  "mode": "strict",
  "coercions": [
    {
      "sourceColumn": "occurred_at",
      "targetColumn": "occurred_at",
      "kind": "iso8601ToTimestamp"
    }
  ]
}

Phase 1 ships with create-if-missing schema inference disabled. Declare the target schema before loading data. Additive mode remains available only where an admin has configured and authorized it.

Error policy

failRun is the Phase 1 default. One invalid row fails the run without changing the table. Validation-only creates an inspectable run but never mutates data. skipRows is feature-flagged off at launch; when enabled later, a run may finish as partial with explicit inserted and rejected counters.

Raw rejected rows are not retained. Quarantine is hard-disabled in Phase 1: setting its reserved runtime switch to true is rejected at startup, and a request with quarantine: true is denied. A future opt-in design would use encrypted storage with a maximum seven-day retention independent of normal staging cleanup.

Phase 1 release defaults

These conservative defaults define the initial customer contract. They are admission and retention limits, not claims of warehouse-scale throughput.

Additional request-rate, tiny-run, parser time, decoded expansion, row, column, nesting, and field-size controls protect shared capacity. When admission is full, Loam returns 429 with Retry-After before creating a run. Clients should wait for that duration and retry the same body with the same idempotency key.

if (response.status === 429) {
  const retryAfterSeconds = Number(
    response.headers.get("retry-after") ?? "1",
  );
  await new Promise((resolve) =>
    setTimeout(resolve, retryAfterSeconds * 1_000),
  );
  // Repeat the exact request with the same Idempotency-Key.
}

Staging and cleanup

Signed uploads are temporary capabilities for one exact server-generated object. The production bucket stores ingestion staging under a dedicated, versioned namespace that is disjoint from durable table parts and anonymous file-conversion temporary objects. Clients never receive permission to list the bucket or choose a neighboring object key.

ingestion-staging/v1/YYYY-MM-DD/
  workspaces/{verifiedWorkspaceId}/
  subjects/{verifiedPrincipalId}/
  pipelines/{pipelineId}/
  targets/{tableNodeId}/
  uploads/{uploadId}/...

Every identity segment is derived from the verified JWT, workspace catalog, and server-generated IDs. The path is useful for isolation, incident response, and exact cleanup, but it is not an authorization mechanism. The Engine still checks the same current grant before commit.

Terminal runs enqueue exact-prefix deletion immediately. Cleanup is durable, idempotent, paged, and retried with backoff. A dated sweeper reconciles abandoned objects after the 24-hour grace period. Cleanup does not traverse durable table data, and table readers do not traverse staging.

Initial readiness targets

General availability requires production-shaped load tests to meet all of these targets under the published Phase 1 limits:

• p95 submission acceptance below 500 ms;
• p95 queue-to-start below 30 seconds under target load;
• 99% of supported loads at or below 100 MiB reach a terminal state within 5 minutes;
• zero duplicate commits across retry and worker-crash tests; and
• cleanup backlog remains below 24 hours.

These are release gates for Loam's current architecture, not parity claims with large managed data warehouses.