Skip to content

perf(motherduck): chunked zstd loads with COPY, and an opt-in external stage - #361

Draft
kvlonge wants to merge 3 commits into
mainfrom
motherduck-load-path
Draft

kvlonge wants to merge 3 commits into
mainfrom
motherduck-load-path

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Three changes, one commit each. Not for merge yet: it waits for a final OK after the live MotherDuck measurement below.

Small pages reach every target in row-group chunks

A source that yields a page at a time, such as sl.http with sl.offset(limit=50), sent each page to the target as its own write: on MotherDuck one temporary Parquet file and one statement per page. Rows are now held per table and written in chunks of up to 122,880 rows (DuckDB's row group) or 64 MiB, whichever comes first (or the batch's own size when smaller). The chunk is written before anything reads, commits, alters or settles what the rows go into, and it is dropped with a batch that rolls back. Under send = "at_commit" the held batches go through the same chunks.

This lives in the loader, so every target gets it: Postgres sent one COPY per page, BigQuery one append request per page, and Snowflake's streaming and insert routes one request per page. Snowflake and Databricks Parquet parts and ClickHouse inserts already pooled pages, and now get larger row groups.

The MotherDuck bridge's temporary Parquet is now zstd; it was uncompressed.

MotherDuck loads each chunk with COPY

Each chunk now goes in with COPY <table> (<columns>) FROM ? (FORMAT parquet) instead of INSERT … SELECT … FROM read_parquet(?). plan names it: route read as Arrow -> zstd Parquet, about 122,880 rows a file -> MotherDuck COPY.

Two choices, both decided:

  1. COPY replaces INSERT. Both run the same plan in DuckDB, and live they send the same bytes. COPY is the only statement on the local route, for staging, direct and build tables alike. INSERT … SELECT FROM read_parquet remains only where MotherDuck reads files on its servers: the external stage, and sources whose files MotherDuck pulls itself.
  2. VARIANT columns are written as Parquet JSON. COPY into a VARIANT column stores a JSON string as VARCHAR. With the column typed JSON in the file, COPY parses it, as the old CAST(… AS JSON) did (checked on duckdb 1.5.6 and 2.0.0; 1.4 has no VARIANT). Staged tables already land json columns as VARCHAR and convert them in the merge, so this matters for direct writes (an append or replace straight into the table). Every table takes the one path.

load_method = { type = "external_stage" } on MotherDuck connections

Opt-in: each chunk is written as zstd Parquet under an s3:// or gs:// prefix, and MotherDuck's servers read the chunks in the commit (read_parquet([...], MD_RUN = REMOTE)), so this machine only uploads to the bucket. An optional connection names the s3 or gcs connection whose settings and keys SQLLoad writes with and MotherDuck reads with; gs:// needs its HMAC key. A custom endpoint is refused, since MotherDuck's servers can't reach it.

  • plan writes a one-row file under the prefix, has MotherDuck's servers count it, and removes it, once per target. Either failure fails the table with the fix, and there is no fallback.
  • Each load fails unless MotherDuck reports as many rows as the chunks' footers hold.
  • Each chunk is removed once its rows commit (on a staged table, once they reach the staging table), unless keep_staged_files = true. Chunks of a batch that fails before MotherDuck reads them are always removed.
  • The default, { type = "local" }, sends the chunks from this machine.
  • Known limit: the transaction of a direct or build table opens only when MotherDuck reads its chunks, right after the bucket's secret is set. With keys from the AWS credential chain, a table whose columns change in the middle of a batch opens it at that change instead, so the batch's last read uses the secret set then.

Tests

  • A source yielding 400 pages of 50 rows loads into the MotherDuck stand-in as one zstd chunk through COPY, under both send modes. Before: 400 uncompressed files.
  • The existing json_type = "variant" test fails with VARCHAR instead of OBJECT(user) if the JSON typing is removed.
  • The plan route line, and load_method parsing and refusals (gs without HMAC, custom endpoint, local folder, wrong connection, bucket mismatch, unknown key). 0.81.0 refused load_method on a MotherDuck connection.
  • The external stage has no offline test: MotherDuck's servers can't reach a local S3, and the session secret carries no endpoint. It was checked by hand against the stand-in and a local SeaweedFS (pages chunked, one remote read, VARIANT types kept, files removed or kept, an unreadable bucket failing plan), and in the live run below.

Live, against MotherDuck

Five variants, each loading the same 1M API rows (sl.http, limit=50, so 20,000 pages, with a VARIANT column) by merge into MotherDuck, each in its own container. Sent is the container's network bytes out during sqlload load. Statements and the longest transaction come from a log of every statement the bridge ran.

Variant Sent Statements Wall Time in MotherDuck statements Longest transaction Rows
v1: 0.81.0 (stopped early, projected) ~340 MB projected ~20,000 projected (one INSERT per page) ~3 h projected 0.45 s per page INSERT not compared
v2: chunks and zstd, INSERT 44.5 MB 23 (9 INSERT) 139.9 s 21.8 s 7.40 s 1,000,000
v3: v2 plus COPY 44.5 MB 23 (9 COPY) 139.2 s 21.0 s 7.38 s 1,000,000
v4: one zstd Parquet, COPY from the duckdb CLI 58.9 MB 1 COPY 13.6 s 12.1 s 1,000,000 (reference)
v5: v3 plus external_stage on S3 24.0 MB, the S3 upload included 18 (1 INSERT reading 9 chunks) 137.4 s 13.3 s 7.43 s 1,000,000
  • v1 was stopped after 16 minutes, so its row is a projection: it had sent 32.3 MB over 1,917 statements (1,909 page INSERTs), about 17 KB and 0.52 s a page, which comes to about 340 MB, 20,000 statements and 3 hours for the 1M rows. It never reached the merge, so it has no row match or transaction to report.
  • In v2, v3 and v5 about 118 s of the wall time is spent fetching the 20,000 pages from the local test API. MotherDuck's share is the statement time.
  • COPY against INSERT (choice 1): the same bytes (44,489,065 against 44,502,562) and the same wall time. The single longest chunk write was 2.5 s with COPY and 3.5 s with INSERT.
  • Every field of every row matches v4: id, updated_at, name, amount, status, and each field inside attrs, compared as typed values. In 16,960 rows of each run, attrs keeps the source's key order (tier, score, …) instead of the sorted order (active, address, …) the other rows get. That is 1,000,000 − 8 × 122,880, but a different, scattered set of ids each run (v2 and v3 disagree on 30,274 rows). The key order is DuckDB's own: with 20,000 rows, 0.81.0, this branch and the hand-made reference all keep the source order in every row. A plain local DuckDB 1.5.6, with no SQLLoad involved, keeps source order when it converts 20,000 rows to VARIANT and sorts the keys for 122,880 or 200,000 rows. DuckDB compares the two orders as equal. SQLLoad sends the JSON text unchanged, and this change doesn't touch that path.
  • The stage's chunks were removed after the commit and the bucket was left empty.

Verify passed in CI on this head (b4677ca): make check-ci, make test-unit, make test-python, the container suites, Windows and the performance gate.

BEGIN_COMMIT_OVERRIDE
perf(loader): merge small pages into row-group chunks and write zstd bridge Parquet

perf(motherduck): load each chunk with COPY, VARIANT columns typed JSON in the file

feat(motherduck): stage chunks in S3 or GCS with load_method = external_stage
END_COMMIT_OVERRIDE

…bridge Parquet

Sources that yield a page at a time, such as sl.http, now reach a
target in chunks of about 122,880 rows (or 64 MiB) instead of one write
per page. The MotherDuck bridge writes its temporary Parquet with zstd.
…ON in the file

The bridge now loads each zstd Parquet chunk with COPY instead of
INSERT ... SELECT FROM read_parquet. VARIANT columns are written as
Parquet JSON, so COPY parses them as the cast did. plan's route for a
MotherDuck table names the chunks and the COPY.
…al_stage

A MotherDuck connection may set load_method = { type = "external_stage",
url = "s3://..." | "gs://..." }: each chunk is written there as zstd
Parquet, and MotherDuck's servers read the chunks in the commit
(MD_RUN = REMOTE). An optional s3 or gcs connection gives the bucket's
settings and keys; gs:// needs its HMAC key. plan writes a one-row file,
has MotherDuck read it and removes it, failing with the fix otherwise.
Each load checks MotherDuck's row count against the chunks' footers, and
the files go after the commit unless keep_staged_files is set. The
default, { type = "local" }, sends the chunks from this machine.
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying sqlload with  Cloudflare Pages  Cloudflare Pages

Latest commit: b4677ca
Status: ✅  Deploy successful!
Preview URL: https://38020c64.sqlload.pages.dev
Branch Preview URL: https://motherduck-load-path.sqlload.pages.dev

View logs

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant