fix: server-side content list search + locale-aware list indexes#1226
Conversation
The admin content list filtered only the rows already loaded on the current page, so entries far back in a large collection were unfindable until you navigated near them. Add a `q` query param to the content list endpoint that performs a case-insensitive substring search across the collection's title/name/slug columns (LIKE wildcards escaped), and wire the admin search box to drive it (debounced) instead of filtering in memory. Add locale-aware composite indexes so i18n-filtered lists stay index-served on large tables. Closes #1219.
🦋 Changeset detectedLatest commit: 3a9b2e7 The changes in this PR will be included in the next version bump. This PR includes changesets to release 14 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
@emdash-cms/admin
@emdash-cms/auth
@emdash-cms/auth-atproto
@emdash-cms/blocks
@emdash-cms/cloudflare
@emdash-cms/contentful-to-portable-text
emdash
create-emdash
@emdash-cms/gutenberg-to-portable-text
@emdash-cms/plugin-cli
@emdash-cms/plugin-types
@emdash-cms/registry-client
@emdash-cms/registry-lexicons
@emdash-cms/sandbox-workerd
@emdash-cms/x402
@emdash-cms/plugin-ai-moderation
@emdash-cms/plugin-atproto
@emdash-cms/plugin-audit-log
@emdash-cms/plugin-color
@emdash-cms/plugin-embeds
@emdash-cms/plugin-field-kit
@emdash-cms/plugin-forms
@emdash-cms/plugin-webhook-notifier
commit: |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
emdash-playground | 3a9b2e7 | Jun 02 2026, 10:33 AM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
docs | 3a9b2e7 | Jun 02 2026, 10:31 AM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
emdash-demo-cache | 3a9b2e7 | Jun 02 2026, 10:32 AM |
Overlapping PRsThis PR modifies files that are also changed by other open PRs:
This may cause merge conflicts or duplicated work. A maintainer will coordinate. |
![emdashbot[bot]](https://avatars.githubusercontent.com/in/3255304?s=60&v=4){kind=small}
There was a problem hiding this comment.
This is the right fix for the stated problem: moving content-list search server-side so large collections are fully searchable, plus locale-aware composite indexes to keep list queries index-served. The implementation follows EmDash conventions well—Lingui-wrapped labels, sql.ref() / validateIdentifier() for SQL safety, a proper changeset, describeEachDialect integration tests, and admin component tests. The handler-resolved searchColumns approach keeps the repository generic while avoiding missing-column errors.
However, the PR introduces three regressions that should be fixed before merge:
-
Postgres index-name truncation / collision. The new index names
idx_{table}_deleted_locale_updated_idandidx_{table}_deleted_locale_created_iddiffer only after a 26-character suffix. Postgres truncates identifiers to 63 bytes, so for any collection slug ≥ 40 characters (ec_+ slug = table name ≥ 43), the distinguishing part is truncated away and both names collide. In the migrationIF NOT EXISTSsilently skips the second index; inSchemaRegistry(noIF NOT EXISTS) collection creation hard-fails with “relation already exists”. The names need to be shortened so the divergence survives truncation. -
Search box disappears on empty server-side results. The search
<Input>is guarded by{items.length > 0 && ...}. With server-side search, a query that returns zero matches makesitemsempty, unmounting the search box. The user then cannot edit or clear their query. -
Pagination count shows loaded page size instead of total during server search.
renderItemCountreceivesfilteredCount: filteredItems.length. In server-search modefilteredItemsis just the currently loaded API page, so the count text displays e.g. “20 items matching …” even when the server reports a total of 100+. The caller already passes the correcttotalprop; the count text should use it in server mode.
|
@scottbuscemi tell ya boy to use the PR template! |
…nt search
- Shorten locale list index names to idx_{table}_loc_upd / _loc_crt so the
updated/created discriminator survives Postgres's 63-byte identifier
truncation; the prior _deleted_locale_*_id names collided from 40-char slugs,
silently dropping one index (migration) or hard-failing collection creation
(registry). Names kept identical across migration 041 and schema/registry.ts.
- Add a pure-string distinctness unit test for the index names.
- Keep the ContentList search box mounted whenever server search is available
so a zero-match query can be cleared, and distinguish no-content from
no-matches in the empty state.
- Count server-search matches from the server total, not the loaded page.
github-actions
Bot
added
the
review/needs-rereview
There was a problem hiding this comment.
Pull request overview
This PR fixes large-collection admin content list search by moving filtering to the server (q query param) and improves i18n list navigation performance by adding locale-aware composite indexes on per-collection ec_* tables (migration 041 + registry parity).
Changes:
- Add server-side, case-insensitive substring search (
q) to the content list API/repository and wire it up in the admin UI with debounced search. - Add locale-aware
(deleted_at, locale, updated_at/created_at, id)indexes via a new migration and inSchemaRegistryfor new collections. - Add integration/unit/UI tests covering server search behavior and index-name truncation constraints.
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/core/tests/unit/database/migrations/041_content_locale_list_index.test.ts | Adds unit tests to guard index-name distinctness under Postgres identifier truncation. |
| packages/core/tests/integration/database/migrations.test.ts | Registers migration 041 in integration migration tests. |
| packages/core/tests/integration/content/content-list-search.test.ts | Adds integration coverage for server-side search (q) behavior. |
| packages/core/src/schema/registry.ts | Creates the new locale-aware indexes when creating new collections. |
| packages/core/src/emdash-runtime.ts | Extends runtime content list params to accept q. |
| packages/core/src/database/repositories/types.ts | Extends repository query options with q + searchColumns. |
| packages/core/src/database/repositories/content.ts | Implements the escaped, case-insensitive LIKE search filter in findMany/count. |
| packages/core/src/database/migrations/runner.ts | Registers migration 041 in the migration runner. |
| packages/core/src/database/migrations/041_content_locale_list_index.ts | Adds migration to create locale-aware composite indexes across ec_* tables. |
| packages/core/src/api/schemas/content.ts | Adds optional q to the content list query schema. |
| packages/core/src/api/handlers/content.ts | Resolves searchable columns per-collection and passes q + columns to the repository. |
| packages/admin/tests/components/ContentList.test.tsx | Adds UI tests for server-search mode (debounce, empty state, counts, no client filtering). |
| packages/admin/src/router.tsx | Adds searchTerm state and includes it in the infinite-query key and request params. |
| packages/admin/src/lib/api/content.ts | Sends q to the API when search is provided. |
| packages/admin/src/components/ContentList.tsx | Adds server-search mode via onSearchChange, debouncing, and updated empty/count logic. |
| .changeset/fix-content-list-search.md | Adds release notes for the behavior/index changes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // Seed 60 ordinary posts plus a "deep" needle far past the first page. | ||
| for (let i = 0; i < 60; i++) { | ||
| const created = await handleContentCreate(ctx.db, "posts", { | ||
| slug: `post-${String(i).padStart(3, "0")}`, | ||
| data: { title: `Ordinary Post ${i}` }, | ||
| }); | ||
| if (!created.success) throw new Error("seed failed"); | ||
| } | ||
| const needle = await handleContentCreate(ctx.db, "posts", { | ||
| slug: "the-needle-post", | ||
| data: { title: "zzz Needle Headline" }, | ||
| }); | ||
| if (!needle.success) throw new Error("needle seed failed"); | ||
|
|
| "@emdash-cms/admin": patch | ||
| --- | ||
|
|
||
| Make content list search work on large collections (#1219). The admin content list previously filtered only the rows already loaded on the current page, so an entry far back in a big collection could not be found until you navigated near it. The list endpoint now accepts a `q` parameter and performs a case-insensitive substring search across the collection's title/name/slug columns server-side (LIKE wildcards in the query are escaped), and the admin search box drives that query (debounced) instead of filtering in memory. Also adds locale-aware composite indexes (`idx_{table}_deleted_locale_updated_id` / `_created_id`) so locale-filtered content lists stay index-served on large, i18n-enabled tables. |
| // Locale-aware composite indexes for i18n content lists (see migration 041). | ||
| // Short `loc_upd`/`loc_crt` suffix keeps the updated/created discriminator | ||
| // inside Postgres's 63-byte identifier limit for long slugs; keep these | ||
| // names identical to migration 041. | ||
| await sql` | ||
| CREATE INDEX ${sql.ref(`idx_${tableName}_loc_upd`)} | ||
| ON ${sql.ref(tableName)} (deleted_at, locale, updated_at DESC, id DESC) | ||
| `.execute(conn); | ||
|
|
||
| await sql` | ||
| CREATE INDEX ${sql.ref(`idx_${tableName}_loc_crt`)} | ||
| ON ${sql.ref(tableName)} (deleted_at, locale, created_at DESC, id DESC) | ||
| `.execute(conn); |
| * Index names use a short `loc_upd`/`loc_crt` suffix rather than spelling out | ||
| * `deleted_locale_updated_id`: Postgres truncates identifiers to 63 bytes, and | ||
| * the longer form pushes the `updated`/`created` discriminator past byte 63 for | ||
| * slugs as short as 40 chars, making both names truncate to the same string. | ||
| * Keep these identical to the names in `schema/registry.ts`. | ||
| */ | ||
| export async function up(db: Kysely<unknown>): Promise<void> { | ||
| const tableNames = await listTablesLike(db, "ec_%"); | ||
|
|
||
| for (const tableName of tableNames) { | ||
| await sql` | ||
| CREATE INDEX IF NOT EXISTS ${sql.ref(`idx_${tableName}_loc_upd`)} | ||
| ON ${sql.ref(tableName)} (deleted_at, locale, updated_at DESC, id DESC) | ||
| `.execute(db); | ||
|
|
||
| await sql` | ||
| CREATE INDEX IF NOT EXISTS ${sql.ref(`idx_${tableName}_loc_crt`)} | ||
| ON ${sql.ref(tableName)} (deleted_at, locale, created_at DESC, id DESC) | ||
| `.execute(db); |
| it("stay <=63 bytes and distinct after Postgres truncation for a 47-char slug", () => { | ||
| const table = `ec_${"a".repeat(47)}`; | ||
| const updated = localeUpdatedIndexName(table); | ||
| const created = localeCreatedIndexName(table); | ||
|
|
||
| expect(Buffer.byteLength(updated, "utf8")).toBeLessThanOrEqual(63); | ||
| expect(Buffer.byteLength(created, "utf8")).toBeLessThanOrEqual(63); | ||
| expect(truncateToPostgresIdentifier(updated)).not.toBe(truncateToPostgresIdentifier(created)); | ||
| }); | ||
|
|
||
| it("stay distinct after truncation for every slug length up to 50 chars", () => { | ||
| for (let slugLength = 1; slugLength <= 50; slugLength++) { | ||
| const table = `ec_${"a".repeat(slugLength)}`; | ||
| const updated = truncateToPostgresIdentifier(localeUpdatedIndexName(table)); | ||
| const created = truncateToPostgresIdentifier(localeCreatedIndexName(table)); | ||
| expect(updated, `slug length ${slugLength}`).not.toBe(created); | ||
| } | ||
| }); |
ascorbic
added
bot:review
There was a problem hiding this comment.
The three issues from the prior review have been partially addressed. The search-box mount-on-empty-results and the server-search match-count bugs are both fixed and verified by new tests. Good.
However, the Postgres identifier truncation fix is incomplete. The shortened loc_upd / loc_crt suffixes keep the two index names distinct only for collection slugs up to 50 characters. SchemaRegistry.validateSlug permits slugs up to 63 characters, and for any slug of 51–63 characters the distinguishing suffix is truncated away by Postgres’s 63-byte identifier limit. Both names then collide to the same string: in the migration IF NOT EXISTS silently skips the second index, and in SchemaRegistry.createContentTable the plain CREATE INDEX hard-fails with “relation already exists”.
The unit test only iterates up to slug length 50, so it gives false confidence. To fix this robustly for all allowed slug lengths, move the discriminator earlier in the identifier — for example idx_loc_upd_{table} → idx_upd_loc_{table} and idx_loc_crt_{table} → idx_crt_loc_{table} — so the upd/crt difference lives near the start and survives truncation regardless of table-name length.
Apart from that, the implementation is solid: SQL is safely parameterized with sql.ref() / validateIdentifier(), Lingui wraps all user-facing strings, RTL-safe logical Tailwind is used throughout, locale filtering is preserved, the changeset is present, and the integration tests cover deep-entry search, case-insensitivity, slug matching, and wildcard escaping.
| function truncateToPostgresIdentifier(name: string): string { | ||
| return Buffer.from(name, "utf8").subarray(0, 63).toString("utf8"); | ||
| } | ||
|
|
There was a problem hiding this comment.
[needs fixing] The test asserts index-name distinctness only for slugs up to 50 characters, but SchemaRegistry.validateSlug allows collection slugs up to 63 characters. For slugs of 51–63 chars the loc_upd/loc_crt suffix is truncated away by Postgres’s 63-byte identifier limit and both names collide (idx_ec_{51+a’s}_loc_… truncates to the same prefix). The migration’s IF NOT EXISTS silently skips the second index; SchemaRegistry.createContentTable hard-fails with “relation already exists”.
Extend the loop to the maximum allowed slug length (63) — it will fail and prove the names still need shortening. A robust fix is to move the discriminator before the table name (e.g. idx_upd_loc_{table} and idx_crt_loc_{table}) so the upd/crt difference survives truncation for every legal slug.
| it("stay distinct after truncation for every slug length up to 63 chars", () => { | |
| for (let slugLength = 1; slugLength <= 63; slugLength++) { | |
| const table = `ec_${"a".repeat(slugLength)}`; | |
| const updated = truncateToPostgresIdentifier(localeUpdatedIndexName(table)); | |
| const created = truncateToPostgresIdentifier(localeCreatedIndexName(table)); | |
| expect(updated, `slug length ${slugLength}`).not.toBe(created); | |
| } | |
| }); |
github-actions
Bot
added
review/awaiting-author
There was a problem hiding this comment.
This is a solid re-review. All three blockers from the previous review have been correctly fixed:
- Postgres index-name collision — Names were shortened from
deleted_locale_updated_id/deleted_locale_created_idtoloc_upd/loc_crt, pushing the collision boundary from ≥40-char slugs to ≥51-char slugs. A string-level unit test verifies distinctness under truncation up to 50 chars, and the migration/registry names are kept identical. - Search box disappearing on empty results — The guard changed from
{items.length > 0 && ...}to{(serverSearch || items.length > 0) && ...}, and the empty state now distinguishes "no content" from "no matches" usingactiveSearch. - Pagination undercounting during server search —
renderItemCountnow uses the servertotalwhenserverSearchis active, so the count line reads e.g. "143 items matching …" instead of the loaded page size.
The implementation follows EmDash conventions: Lingui-wrapped strings, sql.ref() + validateIdentifier() for SQL safety, describeEachDialect integration tests, admin component tests, and a proper changeset. The debounced onSearchChange prop cleanly separates server-mode from legacy client-side filtering.
Two minor things remain:
- Changeset stale index names:
.changeset/fix-content-list-search.mdstill documents the oldidx_{table}_deleted_locale_updated_id/_created_idnames rather than the actualloc_upd/loc_crt. The release notes should match the code. - Index-name truncation edge case: For collection slugs of 51–63 characters (the validated maximum), the
loc_upd/loc_crtsuffix is fully truncated and both names collide to the same identifier. The prior existing indexes (deleted_updated_id/deleted_created_id) have the same limitation, so this is not a new regression, but further shortening the suffixes (or hashing long identifiers) would make the fix complete.
The changeset documented the old idx_{table}_deleted_locale_updated_id
/ _created_id names; the shipped migration and registry use the shorter
idx_{table}_loc_upd / idx_{table}_loc_crt names. Sync the release notes
to the code.
github-actions
Bot
added
review/needs-rereview
The core server-side content search (?q=) landed via emdash-cms#1226. Extend it to two surfaces that still post-filtered in memory: - ContentPickerModal now pushes its search box to the server (search option -> ?q=), so it finds entries anywhere in a large collection instead of only the rows already loaded. Uses keepPreviousData to avoid flashing to empty between keystrokes and keeps load-more available while searching. - MCP content_list gains a q parameter, so agents search server-side rather than post-filtering a page of results. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
What does this PR do?
Closes #1219.
On collections with 1000+ entries:
ContentListfiltereddata.pages.flatMap(...).filter(byTitle)client-side; the API client never sent a search term; the list query/handler/repository had no search filter.)Fix:
Server-side search
contentListQueryschema gains an optionalq(trimmed, 1–200 chars).ContentRepository.findMany/countapply a case-insensitive substring filter across handler-resolvedsearchColumns, OR'd.lower()on both sides for SQLite/Postgres parity; LIKE wildcards (% _ \) in the query are escaped withESCAPE '\'.handleContentListresolves the searchable columns from the collection's fields — alwaysslug, plustitle/namewhen defined — so collections without those columns don't error.fetchContentListsendsq;ContentListdebounces the search box (300ms) and reports it up via a newonSearchChangeprop. The search box now stays mounted on a zero-match query (so it can be cleared), the empty state distinguishes "no content" from "no matches", and in server mode the result count reflects the servertotalrather than the loaded page size. Client-side filtering is retained only as legacy behavior whenonSearchChangeisn't supplied.Navigation
041_content_locale_list_indexadds locale-aware composite list indexes across allec_*tables (idempotent, forward-only), mirrored inSchemaRegistryfor new collections. Index names are kept short enough that theupdated/createddiscriminator survives Postgres's 63-byte identifier truncation for long collection slugs (a collision the first cut of these indexes would have hit at slug length ≥ 40).Deliberately out of scope
The UI's page-number pagination over forward-only cursors is the bigger driver of deep-navigation latency; that's a larger UX refactor, deferred. The per-page COUNT is also intentionally kept — there's a regression guard requiring
totalon every page.Testing:
packages/core/tests/integration/content/content-list-search.test.ts(describeEachDialect): deep entry, case-insensitive, slug match, wildcard-escape, unfiltered fallback. AdminContentList.test.tsx: server-mode query reporting, empty-results search box, server-total count, plus a pure-string index-name distinctness test. Full core suite green;pnpm lint:jsonclean;emdash+@emdash-cms/admintypecheck pass. (Live-Postgres truncation not exercised locally — noPG_CONNECTION_STRING; modelled by the string test and runs against Postgres in CI.)Manual verification:
Type of change
Checklist
pnpm typecheckpassespnpm lintpassespnpm testpasses (or targeted tests for my change)pnpm formathas been runmessages.pochanges except in translation PRs — a workflow extracts catalogs on merge tomain.AI-generated code disclosure
Try this PR
Open a fresh playground →
A full working EmDash site, deployed from this branch. Each visit gets its own session-scoped sandbox: no login needed and no shared state. Try the admin, edit content, hit the public site.
Tracks
fix/1219-content-list-search. Updated automatically when the playground redeploys.