Repository navigation
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1241 +/- ##
==========================================
+ Coverage 93.79% 93.99% +0.20%
==========================================
Files 294 298 +4
Lines 30476 31477 +1001
Branches 3100 3205 +105
==========================================
+ Hits 28584 29586 +1002
+ Misses 1870 1869 -1
Partials 22 22 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
🚀 Deploying Preview to Cloudflare 🚀Preview Deployments by commit
View all previews: View all previews ↗ |
65d6aeb to
96dd2b7
Compare
96dd2b7 to
0c220f7
Compare
0c220f7 to
8410a38
Compare
8410a38 to
8c2601a
Compare
8c2601a to
1b6904e
Compare
1b6904e to
f6b39c9
Compare
f6b39c9 to
67967e0
Compare
03967db to
3ae80f0
Compare
3ae80f0 to
54ee66f
Compare
54ee66f to
21eac55
Compare
21eac55 to
b3f5826
Compare
b3f5826 to
759ea01
Compare
With more than one thread, jsx-ast builds pages as their entries arrive
and html compiles, bundles and renders them as it receives them, so
page building and rendering overlap with the generators before them.
The generated files are unchanged.
jsx-ast yields its pages in the order their entries arrived, and once
every page has, a { manifest } item with all their heads in that order.
html compiles each page as it comes, bundles the component library and
the client assets as soon as the manifest does, and has the workers
render the pages as they are compiled. To list the pages as the site
does, headingSortFn is now exported. Bundler adapters' compile can now
run before and alongside buildServer and buildClient.
createChannel is the queue html hands its compiled pages to the workers
on, and merge yields from several async iterables as items arrive: jsx-ast
announces the manifest next to its pages with it.
Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
Assisted-by: Claude Sonnet 5.5 <noreply@anthropic.com>
worker.stream now reads the next batch once there's room for its chunks, so jsx-ast's manifest goes out once every page has been read that way, still without waiting for the pages being built. The changeset also named @doc-kit/core/utils, which isn't a module: createChannel and merge come from utils/channel.mjs and utils/generators.mjs. Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
… the channel jsx-ast's batches hold whole modules, as metadata's and section-pages' do, so it reads them with toBatches instead of regrouping them. The replay no longer waits on a signal, so createSignal comes with the channel, its only caller. Co-authored-by: Aviv Keller <me@aviv.sh> Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
worker.stream takes its next batch only once there's room for its chunks, so jsx-ast's describe() was read no faster than the pool built pages, and the manifest went out only once the last of section-pages' batches got room: on Node.js's docs that's with 12 to 18 page chunks left. html bundles once the manifest arrives and renders nothing until the bundles are built, so bundling stopped overlapping with building the pages, and so did rendering, all.html's included. jsx-ast now describes every page as it arrives into a channel, which worker.stream reads as it has room. The manifest goes out once the input ends, as before the backpressure. A failure reading the input reaches the output through the manifest. The channel holds what the replay of section-pages' output would hold until read anyway. Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
createSignal had the channel as its only caller, so the channel keeps its own promise to wake its reader, and misc.mjs is as on main. The channel queues items, not batches: merge pushes single items through it. jsx-ast's describe pushes into the channel itself, named described as it holds descriptors, and readPages goes. html's alongside becomes writeWhileCompiling, which says what it decides. The test of the manifest not waiting for the pages goes, as the one of worker.stream holding back the next batch checks the same and more. html's test of the bundles' timing no longer reads all.html back, which its sidebar-order test checks, and the channel's tests use Array.fromAsync. Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
Since the layer below, a generator that streams its input always gets an async iterable, so jsx-ast and html drop toBatches and take only that type. html's tests give it its pages with once, and jsx-ast's test of its whole input compares two batches with one. Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
759ea01 to
3cf9f57
Compare
|
avivkeller
left a comment
There was a problem hiding this comment.
I wonder if we can do this without the merging of AsyncIterables (since that, IMO can be confusing 😓)
The only reason jsx-ast needs merge is to slip { manifest } in mid-stream. What if jsx-ast sent the manifest first instead?
const descriptors = [];
for await (const entries of input) {
const grouped = groupNodesByModule(entries);
for (const head of entries.filter(n => n.heading.depth === 1)) {
descriptors.push({ head, entries: grouped.get(head.api) });
}
}
descriptors.push(...synthetic);
yield [{ manifest: descriptors.map(({ head }) => head) }];
yield* worker.stream(descriptors);…t as the site does From the review, which asked for no extra export. jsx-ast now lists the manifest's heads in the order the site lists the pages, the synthetic ones last, so nothing reading its output needs headingSortFn, and the react package no longer exports getSortedHeadNodes. merge and createChannel move from core into jsx-ast's utils: jsx-ast and html are their only callers, and no generator outside the package needs them. Sending the manifest first, before the pages, would need no merge, but jsx-ast could then build no page before its whole input arrived: on the Node.js docs that made the site build 5% slower at 4 threads and 7% at 6. once, which only core and tests used, is the cache's own again, and the all.html head helper is inlined. Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
|
@avivkeller tried it, it's simpler but 5-7% slower, jsx-ast can't start a page until its input ends. Kept the merge, just private to jsx-ast now, wdyt? |
Description
This PR lets
jsx-astandhtmlstart on their dependency's first chunk, so with more than one thread, page building and rendering overlap with the generators before them. It's part of the stack that splits #1205.jsx-astbuilds each page as its module's entries arrive. Once its input ends, it emits{ manifest }, every page's head in the order the site lists them, without waiting for the pages still being built.htmlcompiles each page as it arrives, bundles the component library and the client assets as soon as the manifest does, and renders the pages as they're compiled.compilecan now run before and alongsidebuildServerandbuildClient.jsx-astnow yields pages in the order their entries arrive, plus the{ manifest }item, which is a breaking change for anything reading its output.mainFull Node.js docs, medians of 5 runs. The extra memory is stages that ran one after the other now running at once.
Validation
Added tests for
jsx-astbuilding pages as they arrive and sending the manifest whileworker.streamholds back its next batch, and forhtmlbundling once the manifest arrives and building the same site from its whole input. On 12 of Node.js's API docs, the output is byte-identical tomain's at 1 and 4 threads.Related Issues
Refs: #1205
Check List
node --run testand all tests passed.node --run format:check&node --run lint.