Add client-side "Did you mean" suggestion to the 404 page

[claude]

Requested by Matt Linville · Slack thread

Description

Prototype for review. Adds one new root file, 404-suggest.js.

Before. Today, visiting docs.wandb.ai/models1 shows three suggested pages, all under /inference, and never /models.

After. The same URL shows a "Did you mean /models?" link above those suggestions, plus up to two smaller alternates.

Mintlify's own suggestions are kept, not replaced. Its list is generated from page content semantics, which is exactly right for a page that was renamed and exactly wrong for a slug that was mistyped. This change adds the second half: a /models1 that is one character away from /models is a spelling problem, and lexical matching on the path solves it deterministically. The two mechanisms are complementary, so ours is inserted above theirs rather than in place of it.

How it works

On page load the script checks for Mintlify's documented 404 DOM hooks. Only after a 404 is detected does it fetch /sitemap.xml — which Mintlify auto-generates for every site — to get the live page list. That means normal pageviews pay nothing for this, nothing is hardcoded, the list can never drift out of sync with the docs, and the ~300 servable pages that aren't in docs.json's navigation are covered for free. If the fetch fails it degrades to a 29-entry built-in list of top-level landing pages, which still resolves /models1 correctly.

Ranking blends a Sørensen–Dice bigram score with normalized Levenshtein distance on the last path segment, plus full-path similarity, segment-vocabulary overlap, and a nearest-existing-ancestor bonus. Typo junk (trailing digits, _/+/%20 separators, case, plurals, .html suffixes, wrong path depth) is normalized away first. A cheap bigram prefilter runs ahead of any edit-distance work; it also means bot noise like /wp-admin and /.env produces no suggestions at all.

The panel is inserted immediately above not-found-recommended-pages-list, so our answer is read first. It carries two escape hatches: the primary button opens the docs search modal prefilled with the de-junked slug (search is free), and a de-emphasized "Ask AI instead" opens the already-installed Kapa widget with the query prefilled and submit: false. That flag matters — 404s attract heavy bot traffic and assistant messages are billed per message, so the assistant is never auto-fired; it only ever runs from a real human click, and the button isn't rendered at all if the widget is absent.

No build step, no dependencies, no config change, ~9 KB gzip. It follows the conventions of the existing root scripts (local-links-same-tab.js, dropdown-tabs-navigate.js, code-group-language-persist.js): dependency-free IIFE, var only, JSDoc throughout, and the same DOMContentLoaded + delayed-retry + debounced MutationObserver + pageshow boot pattern. It also pushes a docs_404_suggestion_shown event to the existing GA4/GTM dataLayer, since Mintlify fires no analytics event for 404s — without it there's no way to tell whether this helps.

Accuracy — measured, not estimated

• 99.4% top-1 / 100% top-3 on a 2,182-case synthetic typo benchmark (real page paths mutated the way people actually mistype them, recovered against the full 1,068-page list). This is the capability the feature exists for.
• 63.1% top-1 / 77.2% top-3 on the 298 usable historical redirects in docs.json. That number is honest but mixes two different problems together. Decomposed:
  • 85.6% top-1 / 97.8% top-3 on the lexically recoverable subset (n=180).
  • The remainder are semantic renames no string matcher can ever reach — e.g. /models/support/dark_mode → /support/models/articles/how-do-i-enable-dark-mode. That is precisely what Mintlify's AI list is good at, and it is why that list stays.

models1 ranks /models first at 0.87, ahead of /support/models (0.74) and /inference/models (0.73).

Open risk — please read before approving

The not-found-* DOM hooks are not verified against the live DOM. They are officially documented by Mintlify, but documented as element selectors and explicitly marked "subject to change" — and I had no egress to docs.wandb.ai to confirm the rendered markup. The preview deployment is the test: load /models1 on the preview URL. If the panel doesn't appear, the selector form is wrong and that's the fix.

Partial mitigations already in place: a document.title / body-text fallback if the hooks change, gated on #page-title being absent so it cannot fire on the real article support/inference/articles/api-error-code-404-model-not-found (whose title contains "404"); and suggestion hrefs use the real slug rather than the normalized one, because 80 of 1,068 pages have underscores or uppercase in their path and linking to a hyphenated form would itself 404. Both of those were live bugs caught in testing.

Two smaller risks: errors.404.redirect isn't pinned in docs.json, so if anyone turns on "Redirect to home" in the Mintlify dashboard this script silently never runs; and the Kapa open({query, submit}) signature is unverified, though it's fully typeof-guarded and falls back to search.

Known third-party error — not from this change. Clicking "Ask AI" on the preview surfaces Error in verifying browser for feedback submission. Captcha token could not be obtained. That comes from the kapa.ai widget the site already loads. Verified against kapa's published bundle: their question-submission path requests an ask_ai reCAPTCHA action but reports the failure using the string from their feedback path, so the "feedback submission" wording is misleading. Kapa.open({query, submit: false}) — what this script calls — only sets the query text in the widget's own React context; the user-initiated send that follows runs the identical submitQuery path a direct submit: true call would, so this script can neither introduce nor bypass the captcha step. The underlying failure is a reCAPTCHA Enterprise token error on kapa's side, most often caused by the reCAPTCHA script being blocked client-side. Worth re-verifying on production after merge.

This is a stopgap. The durable fix is real redirects for dead paths that actually get traffic. This helps the long tail nobody will ever enumerate, and the dataLayer event is there to show whether it's worth keeping.

Testing

• node --check passes; a harness boots the real file in a stubbed DOM and asserts 30/30 ranking cases, the 404-detection truth table including the false-positive case, and that every suggested href is a real page
• Preview deployment: load /models1 on the preview URL and confirm the panel renders above Mintlify's suggestions — not yet done, this is the one thing that needs a browser
• Local build succeeds without errors (mint dev)
• PR tests succeed

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
wandb	🟢 Ready	View Preview	Aug 5, 2026, 6:53 PM