Skip to content

feat: page-number short pages, optional descriptions, batched files merges - #357

Merged
kvlonge merged 3 commits into
mainfrom
page-number-short-page
Nov 20, 2026
Merged

kvlonge merged 3 commits into
mainfrom
page-number-short-page

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Three changes, one commit each.

sl.page_number(page_size=N) ends at a short page

sl.page_number took no page size, so it stopped only at an empty page, and a fan-out whose keys mostly fit in one page sent two requests per key. page_size=N now sends the size on every page (page_size_param, default "page_size"; None for an API with a fixed size) and ends at a page shorter than it. A fan-out's keys share what they learn: once a key shows the API caps its pages below page_size, the others keep paging past such pages to an empty one, every row loads, and a warning names the page_size to set.

Measured against a local stub API at 6 requests/s, 3,121 keys and 59,175 rows:

Requests Time
0.80.0 6,127 1,021.7 s
this branch 3,153 526.0 s

Table descriptions are optional

plan no longer fails a table without a description, and contract generate --write writes columns for a table whose new entry would have none. A description, when given, is still the table's comment. The home page's refusal example is now a type refusal, checked against the 0.80.0 CLI.

Files merge and delete_insert by replace_key commit many files at once

These loaded one file per commit. Consecutive files now commit together, up to 1,000 files or about 1 GB, in the planned order, with the same rows as committing them one by one:

  • A merge ranks a batch's rows by file, then row, so the newest file wins a key several files hold, and a file's last row wins a key it repeats.
  • delete_insert checks each file of a batch in order, as its own commit would, under on_overlapping_files; with last_wins only the last file's rows of a value stay.
  • Each commit's run record lists its files, so a failed batch rolls back whole, names the file, and the next load starts from that batch.
  • A merge with on_duplicate_keys = "fail" still commits each file. So does delete_insert under last_wins outside DuckDB and PostgreSQL, whose staging delete isn't written for the other warehouses, or without _sqlload_source_row.
  • Under a DuckDB memory budget, a later file's piece now replaces an earlier file's rows of a value that an earlier piece of the same load committed; until now those were kept.

New loader tests compare each table with the same files loaded one per run (EXCEPT ALL both ways) on DuckDB and Postgres, for CSV and Parquet, with keys repeated within and across files, and load 1,030 files in 2 commits, including a bad file in the second batch and the rerun after it is fixed. Under a memory budget, two files of 1 MiB rows whose commits are cut into pieces keep only the later file's row of a restated value, batched or one file per commit.

Measured with release builds, 1,155 Parquet files of 173 rows (199,815 rows, 150,000 keys) merged into DuckDB:

Commits Time
0.80.0 1,155 24.6 s
this branch 2 1.0 s

Both end with the same 150,000 rows.

Locally on this head: make verify passed (loader container suite 145, exporter 17, lib 409, clippy variants, fensu), and make test-python passed (625, 3 skipped).

BEGIN_COMMIT_OVERRIDE
feat(http): sl.page_number(page_size=N) ends at a short page

feat(plan): table descriptions are optional

BREAKING CHANGE: plan no longer fails a table that has no description, and contract generate --write no longer refuses such a table.

feat(files): merge and delete_insert by replace_key commit many files at once

BREAKING CHANGE: a failing file rolls back every file of its commit, and a run record of a files merge or delete_insert by replace_key lists all the files of its commit. Excluding _sqlload_source_row no longer makes a files merge refuse repeated keys or newest_by.
END_COMMIT_OVERRIDE

With page_size, a page-number run ends at a page holding fewer rows
than a full page, rather than asking for an empty page after it, so a
fan-out whose keys mostly fit in one page sends one request per key
instead of two. page_size goes with every page as page_size_param
(page_size by default); page_size_param=None only describes an API
whose page size is fixed.

An API can send fewer rows than asked, so a short page ends the run
once the table's requests have shown the API sends that many: after
a full page, or after a short page the next request showed was the
last. Until then the next page checks it, and an API that caps its
pages lower still loads every row, paged to an empty page, with a
warning naming the page_size to set. sl.offset's check of a short
first page now uses the same knowledge, so a fan-out checks it only
until the API has shown its pages are that long.
A description, when given, is still the destination table's comment. A
table without one now loads with no comment, and contract generate
--write writes columns for a table whose new entry has no description.

BREAKING CHANGE: plan no longer fails a table that has no description,
and contract generate --write no longer refuses such a table.
… at once

A files source's merge, or delete_insert by replace_key, now loads
consecutive files in one commit, up to 1,000 files or about 1 GB, in
the planned order. Rows come out as when each file committed on its
own: a merge ranks a batch's rows by file, then row, so the newest file
wins a repeated key, and delete_insert checks each file of the batch in
order, under on_overlapping_files, with last_wins keeping only the last
file's rows of a value. Each commit's run record lists its files, so a
load resumes after the last committed batch. A merge under
on_duplicate_keys = "fail" still commits each file, and so does
delete_insert under last_wins outside DuckDB and PostgreSQL, or
without _sqlload_source_row.

BREAKING CHANGE: a failing file rolls back every file of its commit,
and a run record of a files merge or delete_insert by replace_key lists
all the files of its commit. Excluding _sqlload_source_row no longer
makes a files merge refuse repeated keys or newest_by.
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying sqlload with  Cloudflare Pages  Cloudflare Pages

Latest commit: bbb14a6
Status: ✅  Deploy successful!
Preview URL: https://bb649132.sqlload.pages.dev
Branch Preview URL: https://page-number-short-page.sqlload.pages.dev

View logs

@kvlonge
kvlonge merged commit 938fabd into main Nov 20, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant