docs: fix search ranking, add relevance smoke test and heading discoverability

.github/workflows/search-smoke.yml

@@ -0,0 +1,24 @@
+name: Search relevance smoke test
+
+# Queries the live Algolia index, which only reflects `main` after a crawl — so this
+# runs on a schedule rather than per pull request, where it would report on content
+# that isn't published yet. Use the manual trigger after an index or crawler change.
+on:
+  schedule:
+    - cron: '0 6 * * *'
+  workflow_dispatch:
+
+jobs:
+  smoke:
+    name: Golden search queries
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-node@v7
+        with:
+          node-version: 20
+
+      # No install step: the script has no dependencies and uses the public
+      # search key already committed in docusaurus.config.ts.
+      - name: Run search smoke test
+        run: npm run test:search

docs/platform-engineer-guide/air-gapped-installation.mdx

@@ -530,7 +530,7 @@ upgrading a module across chart versions, pass your full values instead:
 chart's defaults.
 :::
 
-## Authenticated mirrors
+## Authenticated mirrors (imagePullSecrets) {#authenticated-mirrors}
 
 If your mirror requires authentication, create a registry credential secret
 in each namespace and reference it per chart.

docs/platform-engineer-guide/observability-alerting.mdx

@@ -167,12 +167,12 @@ Observer handles the authentication and authorization based on OpenChoreo user i
 
 ---
 
-### Traces
+### Traces (OpenTelemetry) {#traces}
 
 OpenChoreo observability tracing module collects traces from applications that are instrumented to publish traces via [OpenTelemetry Protocol (OTLP)](https://opentelemetry.io/docs/specs/otlp/).
 The traces are enriched with Kubernetes metadata to support querying by OpenChoreo concepts (projects, components, environments, etc.).
 
-#### Instrumenting Applications
+#### Instrumenting Applications with the OpenTelemetry Collector (OTLP) {#instrumenting-applications}
 
 Applications must be instrumented to send traces to the OpenTelemetry Collector. Configure your application to send OTLP traces to one of the following endpoints when using single-cluster mode:
 

docs/platform-engineer-guide/secret-management.mdx

@@ -55,7 +55,7 @@ spec:
   # ... other configuration
 ```
 
-## Configuring a ClusterSecretStore
+## Configuring a ClusterSecretStore (Vault, AWS, GCP, Azure) {#configuring-a-clustersecretstore}
 
 To connect ESO to your secret backend, create a `ClusterSecretStore` resource. For provider-specific configuration (authentication methods, endpoints, etc.), refer to the official ESO provider documentation:
 

docs/platform-engineer-guide/upgrades/v1.1-to-v1.2.mdx

@@ -333,7 +333,7 @@ kubectl get namespaces -l openchoreo.dev/project \
 kubectl get pods -A -l openchoreo.dev/project
 ```
 
-### GitOps installations
+### GitOps installations (Flux, Argo CD) {#gitops-installations}
 
 If `Project` manifests live in Git and are synced by Flux or ArgoCD, the cluster-side backfill alone is not enough: a later sync from a manifest without `spec.type` would attempt to remove the field and be rejected by the strict CRD (or silently strip it under the relaxed one).
 

package.json

@@ -15,6 +15,7 @@
     "write-translations": "docusaurus write-translations",
     "write-heading-ids": "docusaurus write-heading-ids",
     "typecheck": "tsc",
+    "test:search": "node scripts/search-smoke.mjs",
     "format": "prettier --write \"docs/**/*.{md,mdx}\"",
     "format:check": "prettier --check \"docs/**/*.{md,mdx}\""
   },

scripts/ALGOLIA.md

@@ -0,0 +1,73 @@
+# Algolia search configuration
+
+Search on openchoreo.dev is Algolia DocSearch (app `B8ST9KVWVJ`, index `openchoreo`),
+wired up in `docusaurus.config.ts` under `themeConfig.docsearch`. The index is built by
+the Algolia Crawler, and its **settings live in the Algolia dashboard, not in this repo**.
+
+This file explains *why* the current settings look the way they do. It deliberately does
+not duplicate their values — nothing here is applied automatically, so a copy would drift
+out of date silently. The settings themselves live in two places that must agree:
+
+- **Index → Configuration** in the Algolia dashboard (takes effect immediately)
+- the crawler's `initialIndexSettings["openchoreo"]` at crawler.algolia.com
+
+The crawler re-applies `initialIndexSettings` on every **full reindex**, so if the two
+disagree, a reindex silently reverts the index to whatever the crawler holds. Change both.
+
+## Why the settings look like this
+
+Two deviations from the Algolia Crawler's stock DocSearch v3 defaults:
+
+**`searchableAttributes` puts `hierarchy.lvl0` second-to-last.** `lvl0` is the *sidebar
+category label*, `lvl1` is the page's H1. The stock v3 order searches `lvl0` first, so a
+page ranked above others purely because its sidebar folder matched the query — an
+API-reference category named `Authorization` outranked the guide titled "Authorization in
+OpenChoreo", which sat at rank #57. Demoting `lvl0` makes the page title the strongest
+signal and moved that page to #2.
+
+(The upstream fix for this class of bug is DocSearch record format `v2`, whose
+`hierarchy_radio.*` attributes rank "the record's own heading" above ancestry without
+inverting the ancestry order. That needs a crawler `recordVersion` change plus a full
+reindex; the reorder here is the settings-only equivalent.)
+
+**`attributeForDistinct` is `url_without_anchor` with `distinct: 2`.** The stock default
+dedupes on `url`, which *includes the anchor*, so one page contributes a record per heading
+and can occupy nearly every slot in the 20-hit modal. Capping at 2 keeps the page title plus
+its best-matching heading.
+
+## What actually affects search ranking
+
+Worth knowing before trying to make a page more findable:
+
+| Record attribute | Comes from | Ranking weight |
+|---|---|---|
+| `hierarchy.lvl1` | the page's **H1** (not the front-matter `title`) | highest |
+| `hierarchy.lvl2`-`lvl6` | H2-H6 | high, descending |
+| `hierarchy.lvl0` | the sidebar category label | low |
+| `content` | body prose | lowest |
+
+Two consequences that have already caught us out:
+
+- **Front-matter `keywords:` does nothing for search.** It renders a `<meta name="keywords">`
+  tag, but the crawler does not put it in the record and it is not in `searchableAttributes`.
+  `docs/platform-engineer-guide/authorization/overview.md` and `conditions.md` still carry
+  such blocks, added in `0660b29` to "improve search discoverability" — they had no effect.
+  Leave them for SEO if you like, but do not expect search to read them.
+- **A term that appears only in body prose is close to unfindable**, because `content` is the
+  lowest-priority attribute. If users search for a word, it needs to be in a heading. This is
+  how "OpenTelemetry" / "OTLP" went missing: the content documents the collector endpoints,
+  but no heading contains those words.
+
+For abbreviations users type that the docs spell out (`k8s`/`kubernetes`, `idp`/`identity
+provider`, `crd`/`custom resource definition`), the fix is **Synonyms** in the dashboard
+rather than editing pages.
+
+## Guarding against regressions
+
+`scripts/search-smoke.mjs` replays the site's real queries against the live index and
+asserts both properties. Run it after any settings or crawler change:
+
+```sh
+npm run test:search
+npm run test:search -- --index some_scratch_index
+```

scripts/search-smoke.mjs

@@ -0,0 +1,128 @@
+#!/usr/bin/env node
+/**
+ * Search relevance smoke test for the openchoreo.dev DocSearch index.
+ *
+ * Replays the exact query the site's search modal sends (same contextual facet
+ * filters, same hitsPerPage) and asserts two things per golden query:
+ *
+ *   1. relevance  - the expected page appears within RANK_LIMIT
+ *   2. anti-flood - the 20 returned hits span at least MIN_DISTINCT_PAGES pages,
+ *                   so a single doc can't monopolise the modal
+ *
+ * Read-only: uses the public search key already committed in docusaurus.config.ts.
+ *
+ *   node scripts/search-smoke.mjs
+ *   node scripts/search-smoke.mjs --index openchoreo_ranktest
+ */
+
+import {readFileSync} from 'node:fs';
+import {dirname, resolve} from 'node:path';
+import {fileURLToPath} from 'node:url';
+
+const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..');
+
+const HITS_PER_PAGE = 20; // what the DocSearch modal requests
+const DEFAULT_RANK_LIMIT = 3; // expected page must land in the top N unless a case overrides it
+const MIN_DISTINCT_PAGES = 8; // of the 20 hits, at least N distinct pages
+const RANK_SCAN_PAGES = 5; // how far to look when reporting an out-of-range rank
+
+/**
+ * Golden queries: terms real users type, mapped to the page they should reach.
+ * Paths are matched against `url_without_anchor`, so a deep-link to the right
+ * page still counts as a hit.
+ */
+const GOLDEN = [
+  {query: 'authorization', path: '/docs/platform-engineer-guide/authorization/overview/'},
+  {query: 'rbac', path: '/docs/platform-engineer-guide/authorization/overview/'},
+  {query: 'custom roles', path: '/docs/platform-engineer-guide/authorization/custom-roles/'},
+  {
+    query: 'observability',
+    path: '/docs/platform-engineer-guide/observability-alerting/',
+    // Genuinely ambiguous term. ObservabilityAlertRule and
+    // ObservabilityAlertsNotificationChannel match on their own page titles, so they
+    // rank above the guide and no index setting demotes them. Held to top 5.
+    rankLimit: 5,
+  },
+  {query: 'cel conditions', path: '/docs/platform-engineer-guide/authorization/conditions/'},
+];
+
+function readDocsearchConfig() {
+  const src = readFileSync(resolve(ROOT, 'docusaurus.config.ts'), 'utf8');
+  const pick = (key) => {
+    const m = src.match(new RegExp(`${key}:\\s*'([^']+)'`));
+    if (!m) throw new Error(`could not find algolia "${key}" in docusaurus.config.ts`);
+    return m[1];
+  };
+  return {appId: pick('appId'), apiKey: pick('apiKey'), indexName: pick('indexName')};
+}
+
+/** The docs version served unprefixed at /docs/* — same source Docusaurus uses. */
+function readPinnedVersion() {
+  return JSON.parse(readFileSync(resolve(ROOT, 'versions.json'), 'utf8'))[0];
+}
+
+async function search({appId, apiKey, indexName}, query, {page = 0, facetFilters}) {
+  const res = await fetch(`https://${appId}-dsn.algolia.net/1/indexes/${encodeURIComponent(indexName)}/query`, {
+    method: 'POST',
+    headers: {'X-Algolia-API-Key': apiKey, 'X-Algolia-Application-Id': appId},
+    body: JSON.stringify({query, page, hitsPerPage: HITS_PER_PAGE, facetFilters, attributesToHighlight: []}),
+  });
+  if (!res.ok) throw new Error(`algolia ${res.status}: ${(await res.text()).slice(0, 200)}`);
+  return res.json();
+}
+
+/** Rank of the first hit on `path`, scanning past page 1 so failures report a real number. */
+async function rankOf(cfg, query, path, facetFilters) {
+  let scanned = 0;
+  for (let page = 0; page < RANK_SCAN_PAGES; page++) {
+    const {hits, nbPages} = await search(cfg, query, {page, facetFilters});
+    const idx = hits.findIndex((h) => new URL(h.url_without_anchor).pathname === path);
+    if (idx !== -1) return {rank: scanned + idx + 1, firstPage: page === 0 ? hits : null};
+    scanned += hits.length;
+    if (page >= nbPages - 1) break;
+  }
+  return {rank: null, firstPage: null};
+}
+
+async function main() {
+  const argIdx = process.argv.indexOf('--index');
+  const cfg = readDocsearchConfig();
+  if (argIdx !== -1) cfg.indexName = process.argv[argIdx + 1];
+
+  const version = readPinnedVersion();
+  // Mirrors @docsearch/docusaurus-adapter contextual search: locale + docs version tags.
+  const facetFilters = ['language:en', ['docusaurus_tag:default', `docusaurus_tag:docs-default-${version}`]];
+
+  console.log(`index: ${cfg.indexName}   docs version: ${version}\n`);
+  console.log('query'.padEnd(16) + 'rank'.padStart(6) + 'pages/20'.padStart(10) + '  result');
+  console.log('-'.repeat(64));
+
+  let failures = 0;
+  for (const {query, path: expected, rankLimit = DEFAULT_RANK_LIMIT} of GOLDEN) {
+    const {rank} = await rankOf(cfg, query, expected, facetFilters);
+    const {hits} = await search(cfg, query, {facetFilters});
+    const distinct = new Set(hits.map((h) => h.url_without_anchor)).size;
+
+    const problems = [];
+    if (rank === null || rank > rankLimit) problems.push(`expected ${expected} at rank <= ${rankLimit}, got ${rank ?? 'not found'}`);
+    if (distinct < MIN_DISTINCT_PAGES) problems.push(`only ${distinct} distinct pages in top ${HITS_PER_PAGE} (want >= ${MIN_DISTINCT_PAGES})`);
+    if (problems.length) failures++;
+
+    console.log(
+      query.padEnd(16) +
+        String(rank ?? '-').padStart(6) +
+        String(distinct).padStart(10) +
+        `  ${problems.length ? 'FAIL' : 'ok'}`,
+    );
+    for (const p of problems) console.log(' '.repeat(32) + `- ${p}`);
+  }
+
+  console.log('-'.repeat(64));
+  console.log(`${GOLDEN.length - failures}/${GOLDEN.length} passed`);
+  if (failures) process.exitCode = 1;
+}
+
+main().catch((err) => {
+  console.error(err.message);
+  process.exitCode = 1;
+});

versioned_docs/version-v1.2.x/platform-engineer-guide/air-gapped-installation.mdx

@@ -508,7 +508,7 @@ upgrading a module across chart versions, pass your full values instead:
 chart's defaults.
 :::
 
-## Authenticated mirrors
+## Authenticated mirrors (imagePullSecrets) {#authenticated-mirrors}
 
 If your mirror requires authentication, create a registry credential secret
 in each namespace and reference it per chart.

versioned_docs/version-v1.2.x/platform-engineer-guide/observability-alerting.mdx

@@ -167,12 +167,12 @@ Observer handles the authentication and authorization based on OpenChoreo user i
 
 ---
 
-### Traces
+### Traces (OpenTelemetry) {#traces}
 
 OpenChoreo observability tracing module collects traces from applications that are instrumented to publish traces via [OpenTelemetry Protocol (OTLP)](https://opentelemetry.io/docs/specs/otlp/).
 The traces are enriched with Kubernetes metadata to support querying by OpenChoreo concepts (projects, components, environments, etc.).
 
-#### Instrumenting Applications
+#### Instrumenting Applications with the OpenTelemetry Collector (OTLP) {#instrumenting-applications}
 
 Applications must be instrumented to send traces to the OpenTelemetry Collector. Configure your application to send OTLP traces to one of the following endpoints when using single-cluster mode:
 

versioned_docs/version-v1.2.x/platform-engineer-guide/secret-management.mdx

@@ -55,7 +55,7 @@ spec:
   # ... other configuration
 ```
 
-## Configuring a ClusterSecretStore
+## Configuring a ClusterSecretStore (Vault, AWS, GCP, Azure) {#configuring-a-clustersecretstore}
 
 To connect ESO to your secret backend, create a `ClusterSecretStore` resource. For provider-specific configuration (authentication methods, endpoints, etc.), refer to the official ESO provider documentation:
 

versioned_docs/version-v1.2.x/platform-engineer-guide/upgrades/v1.1-to-v1.2.mdx

@@ -333,7 +333,7 @@ kubectl get namespaces -l openchoreo.dev/project \
 kubectl get pods -A -l openchoreo.dev/project
 ```
 
-### GitOps installations
+### GitOps installations (Flux, Argo CD) {#gitops-installations}
 
 If `Project` manifests live in Git and are synced by Flux or ArgoCD, the cluster-side backfill alone is not enough: a later sync from a manifest without `spec.type` would attempt to remove the field and be rejected by the strict CRD (or silently strip it under the relaxed one).