Skip to content

perf: drop node positions after metadata - #1236

Open
ovflowd wants to merge 3 commits into
mainfrom
perf/drop-positions
Open

ovflowd wants to merge 3 commits into
mainfrom
perf/drop-positions

Conversation

@ovflowd

@ovflowd ovflowd commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Description

This PR drops position from every node of metadata's entries once they're parsed. Nothing after metadata reads it (the type warnings that print positions run while it parses), and on Node.js's docs it's ~480k objects, about 80MB, that the main thread holds and copies to and from the workers. json-simple writes the metadata as is, so it loses them too, at about a third of its size.

Build Threads Before This PR
site 4 9.25s, 1797MB 8.75s (-5%), 1762MB (-2%)
site 6 8.50s, 2216MB 8.04s (-5%), 2110MB (-5%)
legacy 4 4.24s, 1539MB 4.04s (-5%), 1502MB (-2%)
legacy 6 4.20s, 1972MB 3.99s (-5%), 1920MB (-3%)

Validation

Added drops position from every node of an entry, which checks that no node of a parsed entry's content has a position. json-simple is the only target whose output changes.

Related Issues

Split out of #1205.

Check List

  • I have read the Contributing Guidelines and made commit messages that follow the guideline.
  • I have run node --run test and all tests passed.
  • I have check code formatting with node --run format:check & node --run lint.
  • I've covered new added functionality with unit tests if necessary.

@vercel

vercel Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
api-docs-tooling Ready Ready Preview Nov 11, 2026 12:07am UTC

Request Review

@codecov

codecov Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.68293% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.18%. Comparing base (61fa0b9) to head (2ab0ad0).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...generators/metadata/utils/__tests__/parse.test.mjs 90.62% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1236      +/-   ##
==========================================
+ Coverage   94.13%   94.18%   +0.04%     
==========================================
  Files         291      291              
  Lines       28900    29255     +355     
  Branches     2932     2976      +44     
==========================================
+ Hits        27206    27553     +347     
- Misses       1672     1680       +8     
  Partials       22       22              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@ovflowd ovflowd mentioned this pull request Nov 3, 2026
4 tasks done
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

api-links Generator

Performance estimate (single CI run)

  • Generation time: 13.4% faster (820.00 ms → 710.00 ms)
  • Peak memory: 0.7% lower (283.63 MB → 281.78 MB)

json Generator

Performance estimate (single CI run)

  • Generation time: 14.0% faster (5.98 s → 5.14 s)
  • Peak memory: 10.3% lower (1.00 GB → 919.23 MB)

legacy-html Generator

Performance estimate (single CI run)

  • Generation time: 35.1% slower (11.52 s → 15.56 s)
  • Peak memory: 5.3% lower (1.43 GB → 1.36 GB)

legacy-json Generator

Performance estimate (single CI run)

  • Generation time: 19.2% slower (8.23 s → 9.81 s)
  • Peak memory: 12.8% lower (1015.46 MB → 885.03 MB)

llms-txt Generator

Performance estimate (single CI run)

  • Generation time: 7.4% faster (6.10 s → 5.65 s)
  • Peak memory: 0.8% lower (866.38 MB → 859.76 MB)

orama-db Generator

Performance estimate (single CI run)

  • Generation time: 27.0% faster (8.52 s → 6.22 s)
  • Peak memory: 9.3% higher (945.88 MB → 1.01 GB)

web Generator

Performance estimate (single CI run)

  • Generation time: 6.4% faster (14.70 s → 13.76 s)
  • Peak memory: 5.4% lower (1.52 GB → 1.44 GB)

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

🚀 Deploying Preview to Cloudflare 🚀

Preview Deployments by commit

Status Deployment URL Commit Updated (UTC) See this deployment's details
  • Build: Success ✅

View logs ↗
No Preview URL. Enable ↗ 2ab0ad0 2026-11-11T00:08:52.135Z View logs ↗
  • Build: Success ✅

View logs ↗
No Preview URL. Enable ↗ db7ba56 2026-11-06T19:44:19.758Z View logs ↗
  • Build: Success ✅

View logs ↗
No Preview URL. Enable ↗ 357bf41 2026-11-06T19:31:32.553Z View logs ↗
  • Build: Success ✅

View logs ↗
No Preview URL. Enable ↗ b88b1d5 2026-11-04T18:24:21.010Z View logs ↗
  • Build: Success ✅

View logs ↗
No Preview URL. Enable ↗ b65c0b7 2026-11-03T19:34:31.832Z View logs ↗

@ovflowd
ovflowd marked this pull request as ready for review November 3, 2026 20:39
@ovflowd
ovflowd requested a review from a team as a code owner November 3, 2026 20:39
kgeminicdev

This comment was marked as low quality.

@ovflowd

ovflowd commented Nov 4, 2026 •

Copy link
Copy Markdown
Member Author

Went with the nit, one walk per entry now :)

Nothing after `metadata` reads where a node came from in the source:
the type warnings that print positions run while it parses. Dropping
them shrinks every entry the main thread holds and copies to and from
the workers. On Node.js's docs that is ~480k position objects, about
80MB, and builds are ~5% faster.

`json-simple`, which writes the metadata as it is, no longer has them
either, and is about a third of its size.

Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
Assisted-by: Claude Sonnet 5.5 <noreply@anthropic.com>
An entry's heading and stability are nodes of its content, so walking
the content alone covers them, without a loop over the three of them
and a check for a missing one. The test now checks the heading and the
stability too.

Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
The test and the comment say position rather than where a node came
from, and the comment says which reader of it already ran. The test no
longer checks how many children the entry's content has, which isn't
about positions.

Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview – api-docs-tooling — 2ab0ad03 Deployed Nov 11, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants