Repository navigation
feat: page-number short pages, optional descriptions, batched files merges - #357
Merged
Merged
Conversation
With page_size, a page-number run ends at a page holding fewer rows than a full page, rather than asking for an empty page after it, so a fan-out whose keys mostly fit in one page sends one request per key instead of two. page_size goes with every page as page_size_param (page_size by default); page_size_param=None only describes an API whose page size is fixed. An API can send fewer rows than asked, so a short page ends the run once the table's requests have shown the API sends that many: after a full page, or after a short page the next request showed was the last. Until then the next page checks it, and an API that caps its pages lower still loads every row, paged to an empty page, with a warning naming the page_size to set. sl.offset's check of a short first page now uses the same knowledge, so a fan-out checks it only until the API has shown its pages are that long.
A description, when given, is still the destination table's comment. A table without one now loads with no comment, and contract generate --write writes columns for a table whose new entry has no description. BREAKING CHANGE: plan no longer fails a table that has no description, and contract generate --write no longer refuses such a table.
… at once A files source's merge, or delete_insert by replace_key, now loads consecutive files in one commit, up to 1,000 files or about 1 GB, in the planned order. Rows come out as when each file committed on its own: a merge ranks a batch's rows by file, then row, so the newest file wins a repeated key, and delete_insert checks each file of the batch in order, under on_overlapping_files, with last_wins keeping only the last file's rows of a value. Each commit's run record lists its files, so a load resumes after the last committed batch. A merge under on_duplicate_keys = "fail" still commits each file, and so does delete_insert under last_wins outside DuckDB and PostgreSQL, or without _sqlload_source_row. BREAKING CHANGE: a failing file rolls back every file of its commit, and a run record of a files merge or delete_insert by replace_key lists all the files of its commit. Excluding _sqlload_source_row no longer makes a files merge refuse repeated keys or newest_by.
Deploying sqlload with
|
| Latest commit: |
bbb14a6
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://bb649132.sqlload.pages.dev |
| Branch Preview URL: | https://page-number-short-page.sqlload.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three changes, one commit each.
sl.page_number(page_size=N)ends at a short pagesl.page_numbertook no page size, so it stopped only at an empty page, and a fan-out whose keys mostly fit in one page sent two requests per key.page_size=Nnow sends the size on every page (page_size_param, default"page_size";Nonefor an API with a fixed size) and ends at a page shorter than it. A fan-out's keys share what they learn: once a key shows the API caps its pages belowpage_size, the others keep paging past such pages to an empty one, every row loads, and a warning names thepage_sizeto set.Measured against a local stub API at 6 requests/s, 3,121 keys and 59,175 rows:
Table descriptions are optional
planno longer fails a table without a description, andcontract generate --writewrites columns for a table whose new entry would have none. A description, when given, is still the table's comment. The home page's refusal example is now a type refusal, checked against the 0.80.0 CLI.Files
mergeanddelete_insertbyreplace_keycommit many files at onceThese loaded one file per commit. Consecutive files now commit together, up to 1,000 files or about 1 GB, in the planned order, with the same rows as committing them one by one:
delete_insertchecks each file of a batch in order, as its own commit would, underon_overlapping_files; withlast_winsonly the last file's rows of a value stay.on_duplicate_keys = "fail"still commits each file. So doesdelete_insertunderlast_winsoutside DuckDB and PostgreSQL, whose staging delete isn't written for the other warehouses, or without_sqlload_source_row.New loader tests compare each table with the same files loaded one per run (
EXCEPT ALLboth ways) on DuckDB and Postgres, for CSV and Parquet, with keys repeated within and across files, and load 1,030 files in 2 commits, including a bad file in the second batch and the rerun after it is fixed. Under a memory budget, two files of 1 MiB rows whose commits are cut into pieces keep only the later file's row of a restated value, batched or one file per commit.Measured with release builds, 1,155 Parquet files of 173 rows (199,815 rows, 150,000 keys) merged into DuckDB:
Both end with the same 150,000 rows.
Locally on this head:
make verifypassed (loader container suite 145, exporter 17, lib 409, clippy variants, fensu), andmake test-pythonpassed (625, 3 skipped).BEGIN_COMMIT_OVERRIDE
feat(http): sl.page_number(page_size=N) ends at a short page
feat(plan): table descriptions are optional
BREAKING CHANGE: plan no longer fails a table that has no description, and contract generate --write no longer refuses such a table.
feat(files): merge and delete_insert by replace_key commit many files at once
BREAKING CHANGE: a failing file rolls back every file of its commit, and a run record of a files merge or delete_insert by replace_key lists all the files of its commit. Excluding _sqlload_source_row no longer makes a files merge refuse repeated keys or newest_by.
END_COMMIT_OVERRIDE