Harden toolkit docs generation and indexing

.github/workflows/generate-toolkit-docs.md

@@ -4,8 +4,8 @@ This workflow regenerates toolkit JSON and opens a PR with the changes. It can b
 
 ## What it does
 
-1. Builds the toolkit docs generator.
-2. Generates toolkit JSON in `toolkit-docs-generator/data/toolkits` using the Engine tool metadata and summary endpoints.
+1. Type-checks and tests the toolkit docs generator.
+2. Generates toolkit JSON in `toolkit-docs-generator/data/toolkits` using the Engine tool metadata endpoint.
 3. Syncs integrations sidebar navigation from the generated JSON.
 4. Creates or updates a PR on the stable `automation/toolkit-docs` branch if any files changed. Later runs overwrite that open PR with the latest generated docs.
 

.github/workflows/generate-toolkit-docs.yml

@@ -1,7 +1,7 @@
 name: Generate toolkit docs
 # Description: Generate toolkit JSON from Engine API, then sync sidebar navigation.
 # Flow:
-# 1) Build the toolkit docs generator
+# 1) Type-check and test the toolkit docs generator
 # 2) Generate toolkit JSON into toolkit-docs-generator/data/toolkits
 # 3) Sync integrations sidebar _meta.tsx from toolkit-docs-generator/data/toolkits
 # 4) Create or update a PR if changes were produced
@@ -48,9 +48,8 @@ jobs:
       - name: Install dependencies
         run: pnpm install --frozen-lockfile
 
-      - name: Build toolkit docs generator
-        run: pnpm build
-        working-directory: toolkit-docs-generator
+      - name: Validate toolkit docs generator
+        run: pnpm run toolkit-docs:check
 
       - name: Generate toolkit docs
         run: |

.github/workflows/llmstxt.yml

@@ -9,12 +9,14 @@ on:
       - "app/en/**/*.mdx"
       - "app/en/**/_meta.tsx"
       - "scripts/generate-llmstxt.ts"
+      - "toolkit-docs-generator/data/toolkits/**"
   pull_request:
     types: [opened, synchronize, reopened]
     paths:
       - "app/en/**/*.mdx"
       - "app/en/**/_meta.tsx"
       - "scripts/generate-llmstxt.ts"
+      - "toolkit-docs-generator/data/toolkits/**"
 
 permissions:
   contents: write
@@ -24,6 +26,9 @@ jobs:
   llmstxt:
     name: Generate LLMSTXT
     runs-on: ubuntu-latest
+    if: >
+      (github.event_name != 'pull_request' ||
+      github.event.pull_request.head.repo.full_name == github.repository)
 
     permissions:
       contents: write
@@ -40,6 +45,7 @@ jobs:
         with:
           ref: ${{ github.event.pull_request.head.ref || github.ref }}
           token: ${{ secrets.DOCS_PUBLISHABLE_GH_TOKEN }}
+          fetch-depth: 0
 
       - name: Install dependencies
         run: npm install -g pnpm

CLAUDE.md

@@ -6,7 +6,7 @@ Arcade documentation site built with Next.js + Nextra (App Router), using pnpm a
 
 ```bash
 pnpm dev              # Local dev server (port 3000)
-pnpm build            # Full production build (toolkit-markdown → next build → pagefind)
+pnpm build            # Full Next.js production build
 pnpm lint             # Lint with Ultracite (Biome-based)
 pnpm format           # Auto-format with Ultracite
 pnpm test             # Run all Vitest tests
@@ -27,7 +27,7 @@ pnpm vitest run tests/broken-link-check.test.ts
 - **`app/_lib/`** — Data-fetching utilities (toolkit catalog, slug generation, static params).
 - **`app/api/`** — API routes (markdown export, toolkit-data, glossary).
 - **`toolkit-docs-generator/`** — Generates MCP toolkit documentation from server metadata JSON files in `toolkit-docs-generator/data/toolkits/`.
-- **`scripts/`** — Build/CI scripts (Vale style fixes, redirect checking, pagefind indexing, i18n sync).
+- **`scripts/`** — Build/CI scripts (Vale style fixes, redirect checking, llms.txt generation, i18n sync).
 - **`tests/`** — Vitest tests (broken links, internal link validation, sitemap, smoke tests).
 - **`lib/`** — Next.js utilities (glossary remark plugin, llmstxt plugin).
 - **`next.config.ts`** — Contains ~138 redirect rules.

README.md

@@ -54,11 +54,11 @@ The `vale:fix` command requires an API key in `.env.local`. Without one, you can
 - [STYLEGUIDE.md](./STYLEGUIDE.md) - Writing standards for voice, tone, and structure
 - [AGENTS.md](./AGENTS.md) - Instructions for AI assistants working on docs
 
-## llms.txt Generation
+## llms.txt generation
 
-The project includes a Next.js plugin that automatically generates an `llms.txt` file following the [llms.txt specification](https://llmstxt.org/). This file helps LLMs understand and navigate the documentation.
+The project includes a script that generates an `llms.txt` file following the [llms.txt specification](https://llmstxt.org/). This file helps LLMs understand and navigate the documentation.
 
-**Automatic generation**: Runs during production builds (`pnpm build`)
+**Automatic generation**: Runs in the `Generate LLMs.txt` GitHub workflow
 
 **Manual generation**: Run `pnpm llmstxt` to regenerate the file
 

app/_components/algolia-search.tsx

@@ -138,28 +138,40 @@ function SearchHit({ hit }: { hit: DocSearchRecord }) {
   );
 }
 
-function EmptyQuery() {
-  const { indexUiState } = useInstantSearch();
-  if (indexUiState.query) {
-    return null;
+function SearchResults() {
+  const { indexUiState, results, status } = useInstantSearch();
+  if (!indexUiState.query?.trim()) {
+    return (
+      <p className="px-4 py-8 text-center text-sm text-muted-foreground">
+        Start typing to search the docs…
+      </p>
+    );
   }
-  return (
-    <p className="px-4 py-8 text-center text-sm text-muted-foreground">
-      Start typing to search the docs…
-    </p>
-  );
-}
 
-function NoResults() {
-  const { results } = useInstantSearch();
-  if (!results?.query || results.nbHits > 0) {
-    return null;
+  if (!results || status === "loading" || status === "stalled") {
+    return (
+      <p className="px-4 py-8 text-center text-sm text-muted-foreground">
+        Searching…
+      </p>
+    );
   }
+
+  if (results.nbHits === 0) {
+    return (
+      <p className="px-4 py-8 text-center text-sm text-muted-foreground">
+        No results for{" "}
+        <strong className="text-foreground">"{results.query}"</strong>
+      </p>
+    );
+  }
+
   return (
-    <p className="px-4 py-8 text-center text-sm text-muted-foreground">
-      No results for{" "}
-      <strong className="text-foreground">"{results.query}"</strong>
-    </p>
+    <Hits
+      classNames={{ item: "", list: "space-y-0.5", root: "" }}
+      hitComponent={({ hit }) => (
+        <SearchHit hit={hit as unknown as DocSearchRecord} />
+      )}
+    />
   );
 }
 
@@ -245,14 +257,7 @@ export function AlgoliaSearch() {
                   />
                 </div>
                 <div className="max-h-[60vh] overflow-y-auto p-2">
-                  <EmptyQuery />
-                  <NoResults />
-                  <Hits
-                    classNames={{ item: "", list: "space-y-0.5", root: "" }}
-                    hitComponent={({ hit }) => (
-                      <SearchHit hit={hit as unknown as DocSearchRecord} />
-                    )}
-                  />
+                  <SearchResults />
                 </div>
               </InstantSearch>
             ) : (

app/_lib/toolkit-data.ts

@@ -65,14 +65,40 @@ const DEFAULT_DATA_DIR = join(
 const resolveDataDir = (options?: ToolkitDataOptions): string =>
   options?.dataDir ?? process.env.TOOLKIT_DATA_DIR ?? DEFAULT_DATA_DIR;
 
+const isRecord = (value: unknown): value is Record<string, unknown> =>
+  typeof value === "object" && value !== null;
+
 const isValidToolkitData = (parsed: unknown): parsed is ToolkitData =>
-  typeof parsed === "object" &&
-  parsed !== null &&
+  isRecord(parsed) &&
   "id" in parsed &&
   ("label" in parsed || "name" in parsed) &&
   "metadata" in parsed &&
-  typeof (parsed as Record<string, unknown>).metadata === "object" &&
-  (parsed as Record<string, unknown>).metadata !== null;
+  isRecord(parsed.metadata);
+
+const isToolkitIndexEntry = (value: unknown): value is ToolkitIndexEntry => {
+  if (!isRecord(value)) {
+    return false;
+  }
+
+  return (
+    typeof value.id === "string" &&
+    typeof value.label === "string" &&
+    typeof value.version === "string" &&
+    typeof value.category === "string" &&
+    (value.type === undefined || typeof value.type === "string") &&
+    typeof value.toolCount === "number" &&
+    Number.isInteger(value.toolCount) &&
+    value.toolCount >= 0 &&
+    typeof value.authType === "string"
+  );
+};
+
+const isToolkitIndex = (value: unknown): value is ToolkitIndex =>
+  isRecord(value) &&
+  typeof value.generatedAt === "string" &&
+  typeof value.version === "string" &&
+  Array.isArray(value.toolkits) &&
+  value.toolkits.every(isToolkitIndexEntry);
 
 const readToolkitFile = async (
   filePath: string
@@ -148,17 +174,11 @@ export const readToolkitIndex = async (
     const content = await readFile(filePath, "utf-8");
     const parsed: unknown = JSON.parse(content);
 
-    // Basic runtime validation - ensure it's an object with required fields
-    if (
-      typeof parsed !== "object" ||
-      parsed === null ||
-      !("toolkits" in parsed) ||
-      !Array.isArray((parsed as { toolkits: unknown }).toolkits)
-    ) {
+    if (!isToolkitIndex(parsed)) {
       return null;
     }
 
-    return parsed as ToolkitIndex;
+    return parsed;
   } catch {
     return null;
   }

app/_lib/toolkit-markdown.ts

@@ -1,4 +1,5 @@
 import type {
+  DocumentationChunk,
   ToolDefinition,
   ToolkitData,
   ToolParameter,
@@ -12,6 +13,32 @@ import type {
  * independent of the rendered HTML.
  */
 const JSON_INDENT = 2;
+const DEFAULT_CHUNK_PRIORITY = 100;
+
+function documentationBlocks(
+  chunks: readonly DocumentationChunk[] | null | undefined
+): string[] {
+  const blocks: string[] = [];
+  const sorted = [...(chunks ?? [])].sort(
+    (left, right) =>
+      (left.priority ?? DEFAULT_CHUNK_PRIORITY) -
+        (right.priority ?? DEFAULT_CHUNK_PRIORITY) ||
+      (left.header ?? "").localeCompare(right.header ?? "") ||
+      left.content.localeCompare(right.content)
+  );
+
+  for (const chunk of sorted) {
+    const content = chunk.content.trim();
+    if (!content) {
+      continue;
+    }
+    blocks.push(
+      chunk.type === "code" ? `\`\`\`text\n${content}\n\`\`\`` : content
+    );
+  }
+
+  return blocks;
+}
 
 /** Collapse newlines and escape pipes so a value is safe inside a table cell. */
 function cell(value: string | null | undefined): string {
@@ -51,6 +78,7 @@ function toolBlock(tool: ToolDefinition): string {
   if (tool.description) {
     blocks.push(tool.description.trim());
   }
+  blocks.push(...documentationBlocks(tool.documentationChunks));
 
   const scopes = tool.auth?.scopes ?? [];
   if (scopes.length > 0) {
@@ -97,6 +125,7 @@ export function toToolkitMarkdown(data: ToolkitData): string {
   if (data.summary) {
     blocks.push(data.summary.trim());
   }
+  blocks.push(...documentationBlocks(data.documentationChunks));
 
   const tools = data.tools ?? [];
   blocks.push(`## Tools (${tools.length})`);

app/_lib/toolkit-slug.ts

@@ -2,6 +2,7 @@ import type { Toolkit } from "@arcadeai/design-system";
 
 const TOOLKIT_ID_NORMALIZER = /[^a-z0-9]+/g;
 const CAMEL_BOUNDARY = /([a-z0-9])([A-Z])/g;
+const SAFE_SLUG = /^[a-z0-9](?:[a-z0-9_-]*[a-z0-9])?$/;
 
 export type ToolkitSlugSource = {
   id: string;
@@ -43,7 +44,8 @@ export function toKebabCase(value: string): string {
 
 const extractSlugFromPath = (path: string): string | null => {
   const segments = path.split("/").filter(Boolean);
-  return segments.at(-1) ?? null;
+  const slug = segments.at(-1);
+  return slug && SAFE_SLUG.test(slug) ? slug : null;
 };
 
 export function getToolkitSlug({ id, docsLink }: ToolkitSlugSource): string {
@@ -62,5 +64,6 @@ export function getToolkitSlug({ id, docsLink }: ToolkitSlugSource): string {
     }
   }
 
-  return toKebabCase(id);
+  const kebabId = toKebabCase(id);
+  return SAFE_SLUG.test(kebabId) ? kebabId : normalizeToolkitId(id);
 }

app/_lib/toolkit-static-params.ts

@@ -31,6 +31,13 @@ export type ToolkitRouteEntry = {
   category: IntegrationCategory;
 };
 
+const sortToolkitRoutes = (routes: ToolkitRouteEntry[]): ToolkitRouteEntry[] =>
+  routes.sort(
+    (left, right) =>
+      left.category.localeCompare(right.category) ||
+      left.toolkitId.localeCompare(right.toolkitId)
+  );
+
 const DESIGN_SYSTEM_TOOLKITS_FOR_ROUTES: ToolkitCatalogEntry[] =
   DESIGN_SYSTEM_TOOLKITS.map((toolkit) => ({
     id: toolkit.id,
@@ -125,7 +132,7 @@ const listToolkitRoutesFromDataDir = async (options?: {
     }
   }
 
-  return [...unique.values()];
+  return sortToolkitRoutes([...unique.values()]);
 };
 
 const resolveToolkitRoute = async (
@@ -192,7 +199,7 @@ export async function listToolkitRoutes(options?: {
     unique.set(route.toolkitId, route);
   }
 
-  return [...unique.values()];
+  return sortToolkitRoutes([...unique.values()]);
 }
 
 export async function getToolkitStaticParamsForCategory(

app/en/get-started/setup/connect-arcade-docs/page.mdx

@@ -5,14 +5,11 @@ description: "Learn how to speed up your development with agents in your IDEs"
 
 # Agentic development
 
-Every page on the Arcade docs site serves as markdown. When an AI agent or coding assistant visits any docs URL, the site automatically returns `Content-Type: text/markdown` instead of HTML if:
+Arcade publishes an [`llms.txt`](/llms.txt) index that agents can use to discover documentation pages.
 
-- The request `User-Agent` header matches a known AI agent (Claude, ChatGPT, Cursor, etc.)
-- The request includes the `Accept: text/markdown` header
+Generated toolkit reference pages also provide complete Markdown through the **Copy page** action. This Markdown includes tool parameters, authentication requirements, outputs, examples, and custom documentation sections that load on demand in the rendered page.
 
-This means you can point your agent directly at any docs page. No need to copy and paste or use the `llms.txt` file. The agent will receive well-formatted markdown out of the box.
-
-For example, you can tell your agent to visit `docs.arcade.dev/get-started/quickstarts/call-tool-agent` and it will automatically get the markdown version of that page.
+Other documentation URLs serve HTML. Use `llms.txt` to give an agent the documentation index, or use **Copy page** when you need Markdown for a specific page.
 
 ## LLMs.txt