diff --git a/.github/workflows/search-smoke.yml b/.github/workflows/search-smoke.yml new file mode 100644 index 00000000..9f30e310 --- /dev/null +++ b/.github/workflows/search-smoke.yml @@ -0,0 +1,24 @@ +name: Search relevance smoke test + +# Queries the live Algolia index, which only reflects `main` after a crawl — so this +# runs on a schedule rather than per pull request, where it would report on content +# that isn't published yet. Use the manual trigger after an index or crawler change. +on: + schedule: + - cron: '0 6 * * *' + workflow_dispatch: + +jobs: + smoke: + name: Golden search queries + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + - uses: actions/setup-node@v7 + with: + node-version: 20 + + # No install step: the script has no dependencies and uses the public + # search key already committed in docusaurus.config.ts. + - name: Run search smoke test + run: npm run test:search diff --git a/docs/platform-engineer-guide/air-gapped-installation.mdx b/docs/platform-engineer-guide/air-gapped-installation.mdx index 047b828c..145dcebb 100644 --- a/docs/platform-engineer-guide/air-gapped-installation.mdx +++ b/docs/platform-engineer-guide/air-gapped-installation.mdx @@ -530,7 +530,7 @@ upgrading a module across chart versions, pass your full values instead: chart's defaults. ::: -## Authenticated mirrors +## Authenticated mirrors (imagePullSecrets) {#authenticated-mirrors} If your mirror requires authentication, create a registry credential secret in each namespace and reference it per chart. diff --git a/docs/platform-engineer-guide/observability-alerting.mdx b/docs/platform-engineer-guide/observability-alerting.mdx index 88b9f7ac..ada0cce9 100644 --- a/docs/platform-engineer-guide/observability-alerting.mdx +++ b/docs/platform-engineer-guide/observability-alerting.mdx @@ -167,12 +167,12 @@ Observer handles the authentication and authorization based on OpenChoreo user i --- -### Traces +### Traces (OpenTelemetry) {#traces} OpenChoreo observability tracing module collects traces from applications that are instrumented to publish traces via [OpenTelemetry Protocol (OTLP)](https://opentelemetry.io/docs/specs/otlp/). The traces are enriched with Kubernetes metadata to support querying by OpenChoreo concepts (projects, components, environments, etc.). -#### Instrumenting Applications +#### Instrumenting Applications with the OpenTelemetry Collector (OTLP) {#instrumenting-applications} Applications must be instrumented to send traces to the OpenTelemetry Collector. Configure your application to send OTLP traces to one of the following endpoints when using single-cluster mode: diff --git a/docs/platform-engineer-guide/secret-management.mdx b/docs/platform-engineer-guide/secret-management.mdx index da3b8231..ce76bf1f 100644 --- a/docs/platform-engineer-guide/secret-management.mdx +++ b/docs/platform-engineer-guide/secret-management.mdx @@ -55,7 +55,7 @@ spec: # ... other configuration ``` -## Configuring a ClusterSecretStore +## Configuring a ClusterSecretStore (Vault, AWS, GCP, Azure) {#configuring-a-clustersecretstore} To connect ESO to your secret backend, create a `ClusterSecretStore` resource. For provider-specific configuration (authentication methods, endpoints, etc.), refer to the official ESO provider documentation: diff --git a/docs/platform-engineer-guide/upgrades/v1.1-to-v1.2.mdx b/docs/platform-engineer-guide/upgrades/v1.1-to-v1.2.mdx index c486e3a6..7e609fe4 100644 --- a/docs/platform-engineer-guide/upgrades/v1.1-to-v1.2.mdx +++ b/docs/platform-engineer-guide/upgrades/v1.1-to-v1.2.mdx @@ -333,7 +333,7 @@ kubectl get namespaces -l openchoreo.dev/project \ kubectl get pods -A -l openchoreo.dev/project ``` -### GitOps installations +### GitOps installations (Flux, Argo CD) {#gitops-installations} If `Project` manifests live in Git and are synced by Flux or ArgoCD, the cluster-side backfill alone is not enough: a later sync from a manifest without `spec.type` would attempt to remove the field and be rejected by the strict CRD (or silently strip it under the relaxed one). diff --git a/package.json b/package.json index 69924113..e3609d3f 100644 --- a/package.json +++ b/package.json @@ -15,6 +15,7 @@ "write-translations": "docusaurus write-translations", "write-heading-ids": "docusaurus write-heading-ids", "typecheck": "tsc", + "test:search": "node scripts/search-smoke.mjs", "format": "prettier --write \"docs/**/*.{md,mdx}\"", "format:check": "prettier --check \"docs/**/*.{md,mdx}\"" }, diff --git a/scripts/ALGOLIA.md b/scripts/ALGOLIA.md new file mode 100644 index 00000000..4f090010 --- /dev/null +++ b/scripts/ALGOLIA.md @@ -0,0 +1,73 @@ +# Algolia search configuration + +Search on openchoreo.dev is Algolia DocSearch (app `B8ST9KVWVJ`, index `openchoreo`), +wired up in `docusaurus.config.ts` under `themeConfig.docsearch`. The index is built by +the Algolia Crawler, and its **settings live in the Algolia dashboard, not in this repo**. + +This file explains *why* the current settings look the way they do. It deliberately does +not duplicate their values — nothing here is applied automatically, so a copy would drift +out of date silently. The settings themselves live in two places that must agree: + +- **Index → Configuration** in the Algolia dashboard (takes effect immediately) +- the crawler's `initialIndexSettings["openchoreo"]` at crawler.algolia.com + +The crawler re-applies `initialIndexSettings` on every **full reindex**, so if the two +disagree, a reindex silently reverts the index to whatever the crawler holds. Change both. + +## Why the settings look like this + +Two deviations from the Algolia Crawler's stock DocSearch v3 defaults: + +**`searchableAttributes` puts `hierarchy.lvl0` second-to-last.** `lvl0` is the *sidebar +category label*, `lvl1` is the page's H1. The stock v3 order searches `lvl0` first, so a +page ranked above others purely because its sidebar folder matched the query — an +API-reference category named `Authorization` outranked the guide titled "Authorization in +OpenChoreo", which sat at rank #57. Demoting `lvl0` makes the page title the strongest +signal and moved that page to #2. + +(The upstream fix for this class of bug is DocSearch record format `v2`, whose +`hierarchy_radio.*` attributes rank "the record's own heading" above ancestry without +inverting the ancestry order. That needs a crawler `recordVersion` change plus a full +reindex; the reorder here is the settings-only equivalent.) + +**`attributeForDistinct` is `url_without_anchor` with `distinct: 2`.** The stock default +dedupes on `url`, which *includes the anchor*, so one page contributes a record per heading +and can occupy nearly every slot in the 20-hit modal. Capping at 2 keeps the page title plus +its best-matching heading. + +## What actually affects search ranking + +Worth knowing before trying to make a page more findable: + +| Record attribute | Comes from | Ranking weight | +|---|---|---| +| `hierarchy.lvl1` | the page's **H1** (not the front-matter `title`) | highest | +| `hierarchy.lvl2`-`lvl6` | H2-H6 | high, descending | +| `hierarchy.lvl0` | the sidebar category label | low | +| `content` | body prose | lowest | + +Two consequences that have already caught us out: + +- **Front-matter `keywords:` does nothing for search.** It renders a `` + tag, but the crawler does not put it in the record and it is not in `searchableAttributes`. + `docs/platform-engineer-guide/authorization/overview.md` and `conditions.md` still carry + such blocks, added in `0660b29` to "improve search discoverability" — they had no effect. + Leave them for SEO if you like, but do not expect search to read them. +- **A term that appears only in body prose is close to unfindable**, because `content` is the + lowest-priority attribute. If users search for a word, it needs to be in a heading. This is + how "OpenTelemetry" / "OTLP" went missing: the content documents the collector endpoints, + but no heading contains those words. + +For abbreviations users type that the docs spell out (`k8s`/`kubernetes`, `idp`/`identity +provider`, `crd`/`custom resource definition`), the fix is **Synonyms** in the dashboard +rather than editing pages. + +## Guarding against regressions + +`scripts/search-smoke.mjs` replays the site's real queries against the live index and +asserts both properties. Run it after any settings or crawler change: + +```sh +npm run test:search +npm run test:search -- --index some_scratch_index +``` diff --git a/scripts/search-smoke.mjs b/scripts/search-smoke.mjs new file mode 100644 index 00000000..f84eeca9 --- /dev/null +++ b/scripts/search-smoke.mjs @@ -0,0 +1,128 @@ +#!/usr/bin/env node +/** + * Search relevance smoke test for the openchoreo.dev DocSearch index. + * + * Replays the exact query the site's search modal sends (same contextual facet + * filters, same hitsPerPage) and asserts two things per golden query: + * + * 1. relevance - the expected page appears within RANK_LIMIT + * 2. anti-flood - the 20 returned hits span at least MIN_DISTINCT_PAGES pages, + * so a single doc can't monopolise the modal + * + * Read-only: uses the public search key already committed in docusaurus.config.ts. + * + * node scripts/search-smoke.mjs + * node scripts/search-smoke.mjs --index openchoreo_ranktest + */ + +import {readFileSync} from 'node:fs'; +import {dirname, resolve} from 'node:path'; +import {fileURLToPath} from 'node:url'; + +const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..'); + +const HITS_PER_PAGE = 20; // what the DocSearch modal requests +const DEFAULT_RANK_LIMIT = 3; // expected page must land in the top N unless a case overrides it +const MIN_DISTINCT_PAGES = 8; // of the 20 hits, at least N distinct pages +const RANK_SCAN_PAGES = 5; // how far to look when reporting an out-of-range rank + +/** + * Golden queries: terms real users type, mapped to the page they should reach. + * Paths are matched against `url_without_anchor`, so a deep-link to the right + * page still counts as a hit. + */ +const GOLDEN = [ + {query: 'authorization', path: '/docs/platform-engineer-guide/authorization/overview/'}, + {query: 'rbac', path: '/docs/platform-engineer-guide/authorization/overview/'}, + {query: 'custom roles', path: '/docs/platform-engineer-guide/authorization/custom-roles/'}, + { + query: 'observability', + path: '/docs/platform-engineer-guide/observability-alerting/', + // Genuinely ambiguous term. ObservabilityAlertRule and + // ObservabilityAlertsNotificationChannel match on their own page titles, so they + // rank above the guide and no index setting demotes them. Held to top 5. + rankLimit: 5, + }, + {query: 'cel conditions', path: '/docs/platform-engineer-guide/authorization/conditions/'}, +]; + +function readDocsearchConfig() { + const src = readFileSync(resolve(ROOT, 'docusaurus.config.ts'), 'utf8'); + const pick = (key) => { + const m = src.match(new RegExp(`${key}:\\s*'([^']+)'`)); + if (!m) throw new Error(`could not find algolia "${key}" in docusaurus.config.ts`); + return m[1]; + }; + return {appId: pick('appId'), apiKey: pick('apiKey'), indexName: pick('indexName')}; +} + +/** The docs version served unprefixed at /docs/* — same source Docusaurus uses. */ +function readPinnedVersion() { + return JSON.parse(readFileSync(resolve(ROOT, 'versions.json'), 'utf8'))[0]; +} + +async function search({appId, apiKey, indexName}, query, {page = 0, facetFilters}) { + const res = await fetch(`https://${appId}-dsn.algolia.net/1/indexes/${encodeURIComponent(indexName)}/query`, { + method: 'POST', + headers: {'X-Algolia-API-Key': apiKey, 'X-Algolia-Application-Id': appId}, + body: JSON.stringify({query, page, hitsPerPage: HITS_PER_PAGE, facetFilters, attributesToHighlight: []}), + }); + if (!res.ok) throw new Error(`algolia ${res.status}: ${(await res.text()).slice(0, 200)}`); + return res.json(); +} + +/** Rank of the first hit on `path`, scanning past page 1 so failures report a real number. */ +async function rankOf(cfg, query, path, facetFilters) { + let scanned = 0; + for (let page = 0; page < RANK_SCAN_PAGES; page++) { + const {hits, nbPages} = await search(cfg, query, {page, facetFilters}); + const idx = hits.findIndex((h) => new URL(h.url_without_anchor).pathname === path); + if (idx !== -1) return {rank: scanned + idx + 1, firstPage: page === 0 ? hits : null}; + scanned += hits.length; + if (page >= nbPages - 1) break; + } + return {rank: null, firstPage: null}; +} + +async function main() { + const argIdx = process.argv.indexOf('--index'); + const cfg = readDocsearchConfig(); + if (argIdx !== -1) cfg.indexName = process.argv[argIdx + 1]; + + const version = readPinnedVersion(); + // Mirrors @docsearch/docusaurus-adapter contextual search: locale + docs version tags. + const facetFilters = ['language:en', ['docusaurus_tag:default', `docusaurus_tag:docs-default-${version}`]]; + + console.log(`index: ${cfg.indexName} docs version: ${version}\n`); + console.log('query'.padEnd(16) + 'rank'.padStart(6) + 'pages/20'.padStart(10) + ' result'); + console.log('-'.repeat(64)); + + let failures = 0; + for (const {query, path: expected, rankLimit = DEFAULT_RANK_LIMIT} of GOLDEN) { + const {rank} = await rankOf(cfg, query, expected, facetFilters); + const {hits} = await search(cfg, query, {facetFilters}); + const distinct = new Set(hits.map((h) => h.url_without_anchor)).size; + + const problems = []; + if (rank === null || rank > rankLimit) problems.push(`expected ${expected} at rank <= ${rankLimit}, got ${rank ?? 'not found'}`); + if (distinct < MIN_DISTINCT_PAGES) problems.push(`only ${distinct} distinct pages in top ${HITS_PER_PAGE} (want >= ${MIN_DISTINCT_PAGES})`); + if (problems.length) failures++; + + console.log( + query.padEnd(16) + + String(rank ?? '-').padStart(6) + + String(distinct).padStart(10) + + ` ${problems.length ? 'FAIL' : 'ok'}`, + ); + for (const p of problems) console.log(' '.repeat(32) + `- ${p}`); + } + + console.log('-'.repeat(64)); + console.log(`${GOLDEN.length - failures}/${GOLDEN.length} passed`); + if (failures) process.exitCode = 1; +} + +main().catch((err) => { + console.error(err.message); + process.exitCode = 1; +}); diff --git a/versioned_docs/version-v1.2.x/platform-engineer-guide/air-gapped-installation.mdx b/versioned_docs/version-v1.2.x/platform-engineer-guide/air-gapped-installation.mdx index 80fa7d29..6d7095b1 100644 --- a/versioned_docs/version-v1.2.x/platform-engineer-guide/air-gapped-installation.mdx +++ b/versioned_docs/version-v1.2.x/platform-engineer-guide/air-gapped-installation.mdx @@ -508,7 +508,7 @@ upgrading a module across chart versions, pass your full values instead: chart's defaults. ::: -## Authenticated mirrors +## Authenticated mirrors (imagePullSecrets) {#authenticated-mirrors} If your mirror requires authentication, create a registry credential secret in each namespace and reference it per chart. diff --git a/versioned_docs/version-v1.2.x/platform-engineer-guide/observability-alerting.mdx b/versioned_docs/version-v1.2.x/platform-engineer-guide/observability-alerting.mdx index 88b9f7ac..ada0cce9 100644 --- a/versioned_docs/version-v1.2.x/platform-engineer-guide/observability-alerting.mdx +++ b/versioned_docs/version-v1.2.x/platform-engineer-guide/observability-alerting.mdx @@ -167,12 +167,12 @@ Observer handles the authentication and authorization based on OpenChoreo user i --- -### Traces +### Traces (OpenTelemetry) {#traces} OpenChoreo observability tracing module collects traces from applications that are instrumented to publish traces via [OpenTelemetry Protocol (OTLP)](https://opentelemetry.io/docs/specs/otlp/). The traces are enriched with Kubernetes metadata to support querying by OpenChoreo concepts (projects, components, environments, etc.). -#### Instrumenting Applications +#### Instrumenting Applications with the OpenTelemetry Collector (OTLP) {#instrumenting-applications} Applications must be instrumented to send traces to the OpenTelemetry Collector. Configure your application to send OTLP traces to one of the following endpoints when using single-cluster mode: diff --git a/versioned_docs/version-v1.2.x/platform-engineer-guide/secret-management.mdx b/versioned_docs/version-v1.2.x/platform-engineer-guide/secret-management.mdx index da3b8231..ce76bf1f 100644 --- a/versioned_docs/version-v1.2.x/platform-engineer-guide/secret-management.mdx +++ b/versioned_docs/version-v1.2.x/platform-engineer-guide/secret-management.mdx @@ -55,7 +55,7 @@ spec: # ... other configuration ``` -## Configuring a ClusterSecretStore +## Configuring a ClusterSecretStore (Vault, AWS, GCP, Azure) {#configuring-a-clustersecretstore} To connect ESO to your secret backend, create a `ClusterSecretStore` resource. For provider-specific configuration (authentication methods, endpoints, etc.), refer to the official ESO provider documentation: diff --git a/versioned_docs/version-v1.2.x/platform-engineer-guide/upgrades/v1.1-to-v1.2.mdx b/versioned_docs/version-v1.2.x/platform-engineer-guide/upgrades/v1.1-to-v1.2.mdx index c486e3a6..7e609fe4 100644 --- a/versioned_docs/version-v1.2.x/platform-engineer-guide/upgrades/v1.1-to-v1.2.mdx +++ b/versioned_docs/version-v1.2.x/platform-engineer-guide/upgrades/v1.1-to-v1.2.mdx @@ -333,7 +333,7 @@ kubectl get namespaces -l openchoreo.dev/project \ kubectl get pods -A -l openchoreo.dev/project ``` -### GitOps installations +### GitOps installations (Flux, Argo CD) {#gitops-installations} If `Project` manifests live in Git and are synced by Flux or ArgoCD, the cluster-side backfill alone is not enough: a later sync from a manifest without `spec.type` would attempt to remove the field and be rejected by the strict CRD (or silently strip it under the relaxed one).