Repository navigation
Conversation
…bridge Parquet Sources that yield a page at a time, such as sl.http, now reach a target in chunks of about 122,880 rows (or 64 MiB) instead of one write per page. The MotherDuck bridge writes its temporary Parquet with zstd.
…ON in the file The bridge now loads each zstd Parquet chunk with COPY instead of INSERT ... SELECT FROM read_parquet. VARIANT columns are written as Parquet JSON, so COPY parses them as the cast did. plan's route for a MotherDuck table names the chunks and the COPY.
…al_stage
A MotherDuck connection may set load_method = { type = "external_stage",
url = "s3://..." | "gs://..." }: each chunk is written there as zstd
Parquet, and MotherDuck's servers read the chunks in the commit
(MD_RUN = REMOTE). An optional s3 or gcs connection gives the bucket's
settings and keys; gs:// needs its HMAC key. plan writes a one-row file,
has MotherDuck read it and removes it, failing with the fix otherwise.
Each load checks MotherDuck's row count against the chunks' footers, and
the files go after the commit unless keep_staged_files is set. The
default, { type = "local" }, sends the chunks from this machine.
Deploying sqlload with
|
| Latest commit: |
b4677ca
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://38020c64.sqlload.pages.dev |
| Branch Preview URL: | https://motherduck-load-path.sqlload.pages.dev |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three changes, one commit each. Not for merge yet: it waits for a final OK after the live MotherDuck measurement below.
Small pages reach every target in row-group chunks
A source that yields a page at a time, such as
sl.httpwithsl.offset(limit=50), sent each page to the target as its own write: on MotherDuck one temporary Parquet file and one statement per page. Rows are now held per table and written in chunks of up to 122,880 rows (DuckDB's row group) or 64 MiB, whichever comes first (or the batch's own size when smaller). The chunk is written before anything reads, commits, alters or settles what the rows go into, and it is dropped with a batch that rolls back. Undersend = "at_commit"the held batches go through the same chunks.This lives in the loader, so every target gets it: Postgres sent one
COPYper page, BigQuery one append request per page, and Snowflake's streaming and insert routes one request per page. Snowflake and Databricks Parquet parts and ClickHouse inserts already pooled pages, and now get larger row groups.The MotherDuck bridge's temporary Parquet is now zstd; it was uncompressed.
MotherDuck loads each chunk with
COPYEach chunk now goes in with
COPY <table> (<columns>) FROM ? (FORMAT parquet)instead ofINSERT … SELECT … FROM read_parquet(?).plannames it:route read as Arrow -> zstd Parquet, about 122,880 rows a file -> MotherDuck COPY.Two choices, both decided:
COPYreplacesINSERT. Both run the same plan in DuckDB, and live they send the same bytes.COPYis the only statement on the local route, for staging, direct and build tables alike.INSERT … SELECT FROM read_parquetremains only where MotherDuck reads files on its servers: the external stage, and sources whose files MotherDuck pulls itself.VARIANTcolumns are written as ParquetJSON.COPYinto aVARIANTcolumn stores a JSON string asVARCHAR. With the column typedJSONin the file,COPYparses it, as the oldCAST(… AS JSON)did (checked on duckdb 1.5.6 and 2.0.0; 1.4 has noVARIANT). Staged tables already landjsoncolumns asVARCHARand convert them in the merge, so this matters for direct writes (an append or replace straight into the table). Every table takes the one path.load_method = { type = "external_stage" }on MotherDuck connectionsOpt-in: each chunk is written as zstd Parquet under an
s3://orgs://prefix, and MotherDuck's servers read the chunks in the commit (read_parquet([...], MD_RUN = REMOTE)), so this machine only uploads to the bucket. An optionalconnectionnames thes3orgcsconnection whose settings and keys SQLLoad writes with and MotherDuck reads with;gs://needs its HMAC key. A custom endpoint is refused, since MotherDuck's servers can't reach it.planwrites a one-row file under the prefix, has MotherDuck's servers count it, and removes it, once per target. Either failure fails the table with the fix, and there is no fallback.keep_staged_files = true. Chunks of a batch that fails before MotherDuck reads them are always removed.{ type = "local" }, sends the chunks from this machine.Tests
COPY, under bothsendmodes. Before: 400 uncompressed files.json_type = "variant"test fails withVARCHARinstead ofOBJECT(user)if the JSON typing is removed.planroute line, andload_methodparsing and refusals (gs without HMAC, custom endpoint, local folder, wrong connection, bucket mismatch, unknown key). 0.81.0 refusedload_methodon a MotherDuck connection.plan), and in the live run below.Live, against MotherDuck
Five variants, each loading the same 1M API rows (
sl.http,limit=50, so 20,000 pages, with aVARIANTcolumn) bymergeinto MotherDuck, each in its own container. Sent is the container's network bytes out duringsqlload load. Statements and the longest transaction come from a log of every statement the bridge ran.INSERTper page)INSERTINSERTINSERT)COPYCOPY)COPYfrom the duckdb CLICOPYexternal_stageon S3INSERTreading 9 chunks)INSERTs), about 17 KB and 0.52 s a page, which comes to about 340 MB, 20,000 statements and 3 hours for the 1M rows. It never reached the merge, so it has no row match or transaction to report.COPYagainstINSERT(choice 1): the same bytes (44,489,065 against 44,502,562) and the same wall time. The single longest chunk write was 2.5 s withCOPYand 3.5 s withINSERT.id,updated_at,name,amount,status, and each field insideattrs, compared as typed values. In 16,960 rows of each run,attrskeeps the source's key order (tier, score, …) instead of the sorted order (active, address, …) the other rows get. That is 1,000,000 − 8 × 122,880, but a different, scattered set of ids each run (v2 and v3 disagree on 30,274 rows). The key order is DuckDB's own: with 20,000 rows, 0.81.0, this branch and the hand-made reference all keep the source order in every row. A plain local DuckDB 1.5.6, with no SQLLoad involved, keeps source order when it converts 20,000 rows toVARIANTand sorts the keys for 122,880 or 200,000 rows. DuckDB compares the two orders as equal. SQLLoad sends the JSON text unchanged, and this change doesn't touch that path.Verify passed in CI on this head (b4677ca):
make check-ci,make test-unit,make test-python, the container suites, Windows and the performance gate.BEGIN_COMMIT_OVERRIDE
perf(loader): merge small pages into row-group chunks and write zstd bridge Parquet
perf(motherduck): load each chunk with COPY, VARIANT columns typed JSON in the file
feat(motherduck): stage chunks in S3 or GCS with load_method = external_stage
END_COMMIT_OVERRIDE