Ingestion system comparison
How Loam pipelines map to Snowflake, BigQuery, Databricks, and Redshift.
Functional mapping, not scale parity
Loam's Phase 1 goal is the practical center of a modern ingestion product: a saved configuration, direct rows and staged files, safe retries, load history, explicit schema and error policy, atomic append/upsert, and one authorization model. It does not claim equivalent throughput, connector breadth, autoscaling maturity, or ecosystem coverage.
| System | Established pattern | Loam's closest Phase 1 concept | Important difference |
|---|---|---|---|
| Snowflake | Stages plus COPY INTO for files, Snowpipe for continuous micro-batches, Snowpipe Streaming for direct rows, and MERGE for matched/unmatched changes. | Upload = isolated stage object; pipeline = saved load definition; run = load history; rows API = direct write; keyed upsert = constrained merge behavior. | Loam does not expose general customer-managed stages, warehouse sizing, continuous pipe notifications, or Snowflake-scale throughput. |
| BigQuery | Load jobs for files and Storage Write API streams for rows; offsets can provide exactly-once stream appends, and pending streams can be committed as a batch. | Durable runs and required retry keys provide safe batch acceptance; atomic commit keeps a run invisible until complete. | Loam Phase 1 has no ordered stream/channel protocol or client-managed offsets. |
| Databricks | Auto Loader discovers cloud files incrementally and uses Structured Streaming checkpoints, schema-evolution modes, and rescued data for drift. | Strict/additive schema policies, fail-run validation, and bounded safe error metadata cover policy at an explicitly submitted run boundary. | Loam does not yet watch customer cloud prefixes or provide the Databricks connector and transformation ecosystem. |
| Amazon Redshift | COPY loads S3 files in parallel; COPY JOB saves a command, watches for new files, and tracks filenames to avoid loading one twice. | A pipeline saves destination/policy, while sourceBatchId provides source-load identity, a file checksum provides content identity, and a run records status. | Loam uploads one bounded object through its API and does not yet monitor a customer S3 location. |
What is semantically comparable at launch
- a saved, versioned ingestion definition;
- bulk file and direct row submission;
- retry-safe requests and source-load history;
- durable status, counters, and bounded error inspection;
- strict/additive schema policy and validation-only runs;
- atomic append, constrained keyed upsert, and whole-table replace; and
- least-privilege authorization on the destination.
What remains deferred
- ordered high-throughput streaming channels and producer offsets;
- automatic object-store discovery and event subscriptions;
- managed connectors and CDC source capture;
- arbitrary transformation SQL inside a pipeline;
- partition-scoped replacement modes; and
- warehouse-scale throughput or connector parity.
Official sources
- Snowflake: data-loading overview, COPY INTO, Snowpipe Streaming, and MERGE.
- BigQuery: load jobs, Storage Write API, and pending-stream batch loading.
- Databricks: Auto Loader and schema inference and evolution.
- Amazon Redshift: loading data and COPY JOB.
Last updated 2026-07-25