Skip to content

Report unhandled components in LLM bundles - #4316

Draft
khadni wants to merge 1 commit into
mainfrom
feat/component-coverage-report
Draft

khadni wants to merge 1 commit into
mainfrom
feat/component-coverage-report

Conversation

@khadni

@khadni khadni commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Note

On hold, evaluating a merge with the existing Markdown fidelity checker.
Option: keep only the small onDropComponent hook from this PR and use it to feed the fidelity checker's unsupported-component list from the actual transform. Today, that list drifts from the transform (for example, it still reports DocCards and FeedPage as unsupported) and cannot see some dropped elements (img, em, CopyText).

Description

Unhandled JSX components are silently removed when MDX pages are converted to the LLM TXT bundles (llms-full*.txt). This PR adds a diagnostic that reports which components are dropped, on which pages, and roughly how much text is lost. It does not change the generated Markdown.

[TEMP - to be addressed ASAP] The first report already shows a significant gap: MethodTabs removes about 91K characters (66 occurrences on 21 ACE/CREC pages), so the UI and API procedures of these guides are missing from the bundles. A separate PR will fix that.

Changes

  • Detection: transformMarkdown records each unhandled component it removes (name and approximate text length). Only successful transforms are counted. Go/TypeScript variants of a page count once (per-page maximum).
  • CI: npm run generate:llms writes reports/component-coverage.json (ignored by Git). In validate-llms, components missing from src/scripts/component-coverage-baseline.json produce non-blocking warnings and a job summary listing the affected pages. The report is uploaded as an artifact.
  • Baseline: it lists the 49 components dropped today, marked "pending review", so CI only flags new ones.
  • Public report: npm run report:component-coverage regenerates public/reports/component-coverage.json without rewriting the TXT files. Run it weekly from an up-to-date main and commit the file alone in a dedicated PR. Content PRs never touch it, and the build pipeline is unchanged.
  • Other:
    • generate-llms now rejects unknown options, missing option values, and unknown sections, and sets process.exitCode instead of calling process.exit.
    • .gitignore now anchors reports/ to the repository root, so the public report can be versioned.
    • The validate-llms job also runs on LLM-related config and dependency changes, and its final checks run even if regeneration fails.

Limitation: approxChars only measures text inside the removed MDX node. Self-closing components (for example <Common callout="…" />) report 0 even when their rendered content is lost.

Validation

  • Markdown output and transform modes are identical before and after on all 1,093 pages (1,164 renders). TXT bundles are unchanged.
  • Jest (src/lib/markdown, src/scripts), typecheck, ESLint, and Prettier pass.
  • report:component-coverage takes about 15 s locally. A failed run exits with a non-zero code and keeps the previous public report.

@vercel

vercel Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
documentation Ready Ready Preview Oct 14, 2026 10:08pm UTC

Request Review

This branch was successfully deployed

1 active deployment
Preview — 33e4290b Deployed Oct 14, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant