Skip to content

Improve retrieval and fetch in the docs MCP sample - #214

Merged
carldebilly merged 3 commits into
yllibed:mainfrom
autocarl:agent/docs-mcp-improvements-213
Nov 14, 2026
Merged

carldebilly merged 3 commits into
yllibed:mainfrom
autocarl:agent/docs-mcp-improvements-213

Conversation

@autocarl

@autocarl autocarl commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Addresses the eight improvements in #213, keeping the implementation local to the sample. No new packages, service layers, retained snapshots, background jobs, or retrieval framework.

  • Ignore YAML front matter and select the first real Markdown heading as the page title. Preserve original source text/line ranges; prefer newline boundaries for oversized passages, with a surrogate/CRLF-safe fallback for long lines.
  • Describe the public MCP parameters and their limits, and supply tool details explaining search references and continuation.
  • Add docs fetch --offset / MCP offset, nextOffset, and a same-heading nextId. A small sample-specific helper handles bounded UTF-16 pages. Each call still reads the current generation; callers restart if it changes. This is not a snapshot protocol.
  • Budget and embed the same bounded title/current-heading context. Each context field is clipped to 32 UTF-16 code units without splitting surrogate pairs; raw metadata/text are unchanged. Bump the existing index schema version so an unchanged-source warm index rebuilds its old body-only vectors.
  • Tell an agent to retry shortly while starting or building; retain distinct cancelled/failed states and explain that cancellation can be retried by a later scan.
  • Add two short, relevant documents: the bundled corpus now has 24 passages across six files, above the default 20-candidate window. Correct the lexical-candidate explanation: FTS MATCH still filters rows, regardless of corpus size.
  • Allow the net10.0 sample to roll forward to a newer major runtime, removing the separate .NET 10 runtime prerequisite and the Docker build-time environment workaround. The final container continues using its existing .NET 10 runtime.

Closes #213.

Verification

With the pinned assets fetched and DOCS_TEST_ASSETS pointing to that directory:

  • dotnet build src/Repl.DocsSampleTests/Repl.DocsSampleTests.csproj -c Release -f net10.0 -m:1 --disable-build-servers -p:UseSharedCompilation=false -nr:false -warnaserror — zero warnings/errors.
  • Same Release build for net11.0 — zero warnings/errors.
  • dotnet src/Repl.DocsSampleTests/bin/Release/net10.0/Repl.DocsSampleTests.dll --progress off --report-trx — 105 passed, 0 failed, 0 skipped.
  • Same test assembly for net11.0 — 105 passed, 0 failed, 0 skipped. These include the real ONNX reference vectors/tokenizer, native SQLite, HTTP MCP parameter descriptions and offset binding, pipes/WebSockets, lossless >20k document reconstruction, sibling boundaries, cancellation, and unchanged-source warm-index invalidation.
  • Negative controls: original parsing with this PR's MarkdownCorpus.EmbeddingText helper fails 10 of the 15 parser cases; the old index implementation fails the contextual-input and fetch-continuation tests; old readiness/schema behavior fails the two added worker regressions.
  • Actual sample smoke in an isolated installation exposing only .NET/ASP.NET 11.0.0, with no roll-forward environment override: verified asset reuse, server startup, readiness, real-model indexing/search, and graceful SIGINT exit (130). The project runtime configuration supplies Major roll-forward.
  • Real E5 before/after comparison on the identical six-file corpus and five fixed queries; the small observed ranking changes and unrelated-query control are recorded in the README. This is not a general claim of improved relevance.
  • git diff --cached --check and added-line security scan passed.

Local native verification was on Linux x64. Windows/macOS and the Docker image remain for the existing CI workflow; this PR does not change CI policy. Split code blocks retain raw source fragments rather than inventing repaired Markdown fences. Ancestor-heading context and durable multi-request snapshots are intentionally not added.

Review follow-up (29780a8)

  • Console views now print usable offset/sibling fetch commands, covered with ANSI-enabled rendering tests.
  • MCP continuation tests assert exact public JSON names and null semantics with JsonDocument, not a producer DTO.
  • Chunk token probes build context once per range and use string.Concat(prefix, text.AsSpan(...)); whitespace-only fragments are not indexed. LF/CRLF regression cases failed before the correction.
  • Tightened paging result/helper contracts, centralized surrogate boundaries, bounded sibling lookup by adjacent rowid plus document/anchor, clarified console-only help, and added cold-start/schema-diagnostic/corpus-size coverage and test-style corrections.
  • Final Release builds with warnings as errors and 105 tests per runtime passed locally on .NET 10 and .NET 11 (no skips). The .NET 11-only smoke and all five documented real-model ranking orders were rechecked. The bounded independent seven-lens corrective review found no medium-or-higher blocker.
  • Normal follow-up commit; no amend, squash, or force-push. CI for 29780a8 is green: the main CI and Documentation sample workflows succeeded, including Windows/Linux/macOS native tests and container checks. The release job is intentionally skipped on this PR.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-14T17:03:06.180258Z e10620a Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 940a56df03

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@autocarl

Copy link
Copy Markdown
Contributor Author

Review fixes are published in 29780a88fb3d64ec03628d7caddfe7e5447b03dc (incremental commit, original history retained). Addressed the console continuation dead end, exact MCP JSON contract assertions, parser allocation copies, whitespace-only passages, and MCP help ambiguity, plus the accompanying contract/operability/style points. The inline whitespace finding has an individual reply and is resolved. Local Release builds are clean; 105 tests passed on each of .NET 10 and .NET 11, without skips. The actual .NET 11-only runtime smoke and the documented real-model ranking check pass. Independent corrective review across seven lenses found no medium-or-higher blocker. CI for this new SHA is now pending; the previous head's green runs are not being reused as final evidence.

@autocarl

Copy link
Copy Markdown
Contributor Author

Exact-head CI is now green for 29780a88fb3d64ec03628d7caddfe7e5447b03dc: CI and Documentation sample both succeeded. Native jobs passed on Windows, Linux and macOS; documentation container and framework smoke checks passed. The release job is skipped as expected on a PR. Local corrective review and 105 sample tests per runtime are complete; no merge performed.

@carldebilly

Copy link
Copy Markdown
Member

@codex review please

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 29780a88fb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread samples/10-docs-mcp/ConsoleViews.cs Resolved

@autocarl autocarl left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed the new scoped-console continuation finding in e10620a. Narrow two-file correction: explicit root-scope instructions for both continuation commands, with both ANSI rendering regressions observed RED then GREEN (2/2). Release warning-as-error build passed. No routing/API change, no merge, and no bot re-review triggered. New-head CI is pending; prior green results remain scoped to 29780a8.

@autocarl

Copy link
Copy Markdown
Contributor Author

Exact-head CI is green for the scope-hint fix e10620a7600a144832bdc51e71272721e418c1f9: both CI (run 43544551343) and Documentation sample (run 43544551339) succeeded, including documentation runtimes on Windows/Linux/macOS and the documentation container. The two targeted ANSI rendering regressions also passed locally after failing before the fix. No merge performed.

@carldebilly

Copy link
Copy Markdown
Member

@codex another review please

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: e10620a760

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@carldebilly
carldebilly merged commit 4599666 into yllibed:main Nov 14, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

samples: make docs-mcp hold up on a real corpus and read better to agents

2 participants