Add contributor + per-role operators to advanced search

[bendichter]

Stacked on top of #2821. Adds operators for filtering dandisets by their version-metadata contributors:

• contributor: — catch-all (any role); matches by name/email/identifier
• 9 role-specific operators: author, contact_person, data_collector, data_curator, data_manager, maintainer, project_leader, funder, sponsor. Each matches a contributor by name/email/identifier AND requires that contributor to hold the corresponding dandischema.RoleType role.
• affiliation: — queries the nested Person.affiliation[] field by organization name or ROR identifier. Structurally distinct from the role operators (see below).

Behavior

Query	Effect
contributor:"Doe, Jane"	dandisets where any contributor element matches "Doe, Jane" via name/email/identifier
author:Doe	dandisets where some contributor named Doe holds the Author role
data_curator:Doe	dandisets where some contributor named Doe holds the DataCurator role
funder:NIH	dandisets where some contributor named NIH holds the Funder role
author:Doe funder:NIH	dandisets where (some Doe is an Author) AND (some NIH is a Funder) — possibly different contributor entries
contributor:0000-0002-2990-9889	matches by ORCID
contributor:01cwqze88	matches the ROR URL https://ror.org/01cwqze88 via substring
data_curator:0000-0002-2990-9889	composes role + ORCID — that specific person must hold the DataCurator role
affiliation:Stanford	dandisets with any contributor affiliated with Stanford University
affiliation:00f54p054	ROR ID substring (matches Stanford's full https://ror.org/00f54p054 URL)
author:Doe affiliation:Stanford	composes — Doe is an Author AND someone is Stanford-affiliated, on the same version
data_curatr:Doe	400 with Did you mean "data_curator"?

Each role/contributor operator's lookup ORs across name, email, AND identifier (case-insensitive substring), so ORCIDs for Persons and ROR URLs for Organizations both work. Substring matching means bare-ID forms like 01cwqze88 work without typing the full URL prefix. Operator keys themselves are also case-insensitive (AUTHOR:Doe is the same as author:Doe).

The role list is deliberately concise — most RoleType values aren't operators users reach for. The catch-all contributor: still finds anyone in any role; the role-restricting shortcuts cover the common search intents (authors, curators, funders, etc.).

Affiliation is special

Affiliation exists in dandischema.RoleType, but in practice no real DANDI contributor uses dcite:Affiliation as a roleName. Affiliations live in a separate nested field — Person.affiliation[] — populated for ~all real Person contributors (dandiset 000409 alone has 40+ contributors each affiliated with their respective universities).

So affiliation: is not a per-role operator. It uses a dedicated jsonpath ($.contributor[*].affiliation[*]) that ORs across the affiliation's name and identifier. Implementation-wise it shares the same batch-AND-on-same-Version dispatch as the role operators, so it composes with them as expected.

Why per-role operators (Option D) over contributor: + role:

Considered combining a generic contributor: and a separate role: qualifier with same-element semantics, but per-role operators won on:

• Composability — author:Doe funder:NIH cleanly means two independent constraints (no ambiguity about which role: applies to which contributor: when both repeat).
• Consistency — mirrors Gmail's from:/to:/cc: and falls out of the same AND-across-keys semantics our other operators already have.
• Precedent — independent role operators is a normal shape; arbitrary "qualifier composition" semantics aren't.

Implementation cost is the same either way (one dict + one jsonpath builder).

Implementation notes

• A CONTRIBUTOR_ROLE_OPS dict in dandiapi/api/services/search/operators.py drives the parser allowlist + the role-based dispatch. Adding a future role is one new entry. Kept explicit (rather than auto-derived from dandischema.RoleType) so a schema-level rename or addition can't silently change our public search syntax — see discussion thread. A unit test (test_contributor_role_ops_match_actual_dandischema_roletype) pins each value against an actual RoleType.name so drift is caught explicitly.
• The contributor + affiliation jsonpath builders use SQL-time concatenation (to_jsonb(%s::text)::text) to inline the bound regex into the jsonpath text, because Postgres' jsonpath like_regex requires its pattern as a string literal (not a $variable). Same shape as the asset operators.
• _apply_contributor_filters() accumulates (where, params) pairs and chains them as Version.objects.extra(...) calls, so role-based and affiliation predicates share the same batch logic.
• All contributor predicates in one query AND on the same Version.metadata so a draft + published version with disjoint contributor lists never combine into a spurious match. Cross-operator AND on the same dandiset, but each operator may match a different contributor element within that version's array.

Files

• dandiapi/api/services/search/operators.py — new pure-Python module with all operator vocabulary + dispatch tables; OPERATOR_KEYS is now derived as the union (no more duplication between parser and filter)
• dandiapi/api/services/search/parser.py — imports OPERATOR_KEYS from operators.py; key matching is now case-insensitive
• dandiapi/api/services/search/filters.py — _contributor_where(), _affiliation_where(), _apply_contributor_filters()
• dandiapi/api/views/serializers.py — OpenAPI help text appended
• web/src/components/DandisetSearchField.vue — popover entries for the catch-all, 6 common roles, and affiliation:
• dandiapi/api/tests/test_dandiset.py — one consolidated test for catch-all/role/case-insensitivity/identifier/role-substring/composition (per @yarikoptic's "denser tests" preference); separate tests for affiliation (org name + ROR id + Person/Organization mix + composition) and the did-you-mean-on-typo path. Anonymous Doe placeholders.
• dandiapi/api/tests/test_search_parser.py — drift-guard test against dandischema.RoleType.

Test plan

• tox -e lint — clean
• tox -e test -- dandiapi/api/tests/test_dandiset.py dandiapi/api/tests/test_search_parser.py -k "search_parser or advanced_search or contributor or affiliation" — 53 pass
• Manual: curl 'http://localhost:8000/api/dandisets/?search=author:Doe', affiliation:Stanford, data_curator:0000-0002-2990-9889, etc.

Once #2821 merges, this PR's base will collapse to master automatically.

[yarikoptic]

my initial HI review (edit: actually it is "requesting changes")

Stacked on top of #2821 (which is itself stacked on #2814). Until those merge, ...

need to refresh you AI memory map of things -- #2814 is merged. For a number of "situations", I myself started to read/tune more of the AI output I give to folks for consumption and also "fold" some parts of the descriptions into <details>, like e.g. see freshish datalad/datalad#7852 ... on that end, started

• dandi/dandi-infrastructure#278

[yarikoptic]

I itch with duplication allergy so badly: are those all duplicates of what we have in the dandi-schema mapped from snail_case to PythonCase?

❯ git grep -p -e ContactPerson -e Conceptualization -- dandischema/models.py
dandischema/models.py=class RoleType(Enum):
dandischema/models.py:    #: Conceptualization
dandischema/models.py:    Conceptualization = "dcite:Conceptualization"
dandischema/models.py:    ContactPerson = "dcite:ContactPerson"
dandischema/models.py=class Contributor(DandiBaseModel):
dandischema/models.py:        if role_names is not None and RoleType.ContactPerson in role_names:
dandischema/models.py=class Dandiset(CommonModel):
dandischema/models.py:            if val.roleName and RoleType.ContactPerson in val.roleName:
dandischema/models.py:            raise ValueError("At least one contributor must have role ContactPerson")

I feel like should be programmatically mapped from that dandischema.models.RoleType!

[bendichter]

Two duplications conflated here, taking them separately:

Within our code (parser allowlist + filter dispatch tables listing the same operator names): real, will fix. Plan is to move the dispatch tables into a small pure-Python module both parser.py and filters.py import from, then construct OPERATOR_KEYS as the union of those tables. Adding a new operator becomes one entry in the dispatch table, automatically known to the parser.

Between our code and dandischema.RoleType: I considered deriving _CONTRIBUTOR_ROLE_OPS programmatically from RoleType with an exclusion list, but I think the explicit allowlist is the right call here:

• The dispatch dict is a translation (snake_case operator name → PascalCase role name). Even with auto-derivation, we'd still have an exclusion list and a snake-case transformer — it's the same lines moved into a for loop, not fewer.
• API stability: a future PR that adds a new RoleType to dandi-schema shouldn't silently expand our public search syntax. With derivation, the new operator just appears with no review of the operator name, UX, or whether anyone wants it.
• Renames are a footgun: if dandi-schema ever renames DataCurator → DataCuratorPerson, the auto-derivation would silently change data_curator: to data_curator_person: and break every saved/shared user query. The explicit allowlist catches that as a diff to review here.

What I'll add as a guard: a unit test that every value in _CONTRIBUTOR_ROLE_OPS is an actual RoleType.name — one assertion, catches typos against the schema without auto-tracking it.

[yarikoptic]

can exclude explicitly selected few if needed

[bendichter]

👍 see consolidated reply on the comment above — the exclusion list will live next to the dispatch dict (and a unit test will pin every entry against RoleType.name so drift is caught explicitly).

[yarikoptic]

"many more" is unclear here, of what more -- operators? or what it matches, or ...?

I feel that here we are getting into the land of "we need a UI" to support this variability. Here we do not even list all possible values I think from that earlier "duplicated" list, so how users would know? or is that the final list supported?

So I feel like we do need to figure out API to query for "available operators" and then "available values for the operator given current search query". Quick and dirty could be -- we could do smth like

• having "?" as an "operator" to trigger error listing all known operators, and we just present to user (with some minimal formatting tune up). ideally should return structured record of e.g. {name:str : { description:str, available values: list[str] }} or alike
• having "operator:?" returning similar record but now just for that operator.

In both cases -- would be with constraint given the rest of the query, e.g. "neuropixels species:?" would give error listing error like available 'species': "mus musculus", .... Then at least it becomes usable: referral to API is really not for user facing.

Then on top of that (in different PR) can bolt on nice frontend UI with dropbox for selection of values or operators to use for (which would inject "?" into query and run it, to render then result UI)

[bendichter]

Agreed the popover isn't the right surface for this many operators. Two-step plan:

This PR: replace the unhelpful "many more — see API docs" hand-wave with a real link to a docs page. The docs themselves will land in dandi/dandi-docs (separate repo); I'll add the page there and link from the popover here once it's published.

Future PR (your "?" idea): a discovery endpoint along the lines you sketched — ?search=? lists known operators; ?search=species:? lists candidate values, constrained by the rest of the query. That's a real improvement, but it's enough scope to deserve its own PR rather than getting bundled here.

On the user-doesn't-need-all-of-them point: agreed. 90%+ of contributor queries will be contributor: or author:. The 25 role-specific operators exist for the case where someone DOES want to find e.g. "dandisets with NIH as a Funder," and the docs page is the right place to enumerate them rather than dumping them in the popover.

[yarikoptic]

oy -- I think I saw those somewhere... duplication again? could be avoided?

[bendichter]

Same root cause as the role-list duplication thread above. Plan: move the dispatch tables into a small pure-Python module both parser.py and filters.py import from, and construct OPERATOR_KEYS as the union of those tables — one source of truth in our codebase. Adding a new operator becomes one entry; the parser automatically knows about it.

[bendichter]

Refreshed the description: dropped the stale "#2814 stacked" framing (since it merged) and folded the implementation-notes / files / test-plan sections into <details> per your suggestion. Inline replies on each thread are posted.

[bendichter]

here is the doc page on this that would be added into dandi-docs

advanced-search.md

pasted (?) content from above: click to expand

Advanced Search

The dandiset list's search box accepts a Gmail/GitHub-style syntax that lets you mix
free-text terms with structured key:value operators. Filter by creation date,
species, file type, contributor, role, owner, and more — all from the same input.

Quick examples

neuropixels species:mouse created_after:2023-01-01
author:"Doe, Jane" funder:NIH
data_curator:"Smith, Alice" published_after:2024-01-01
contributor:0000-0002-2990-9889 standard:nwb
affiliation:Stanford

Operators combine with AND. Quoted phrases ("like this") are treated as a single
value. Anything you type without a key: prefix is full-text matched against the
dandiset metadata, the same way the original search box worked.

---

How operators combine

• Operators describe the dandiset, not individual assets. Each operator is
  an independent constraint at the dandiset level. species:mouse species:rat
  returns dandisets that have at least one mouse asset AND at least one rat
  asset — they can be the same asset (multi-species recording) or two
  different assets (a comparative-species dandiset).
• Free text + operators: ANDed together. place cells species:mouse
  returns dandisets whose metadata contains "place" AND "cells" AND has at
  least one mouse asset.
• Multiple different operators: ANDed at the dandiset level. author:Doe funder:NIH returns dandisets where someone named Doe is an Author and
  someone named NIH is a Funder. They can be different contributor entries.
  species:mouse approach:electrophysiological returns dandisets that have
  some mouse data AND some electrophysiology data — possibly on different
  assets, possibly on the same one.
• Quoting: wrap multi-word values in double quotes, e.g.
  technique:"spike sorting". A whole token wrapped in quotes opts out of
  operator parsing — "author:Doe" searches for the literal text author:Doe
  rather than running the operator.

---

Operator reference

Dates

All take an ISO date in the form YYYY-MM-DD. Bounds are exclusive on
_before and inclusive on _after.

Operator	What it filters
created_before:YYYY-MM-DD	Dandiset's created timestamp before the date
created_after:YYYY-MM-DD	Dandiset's created timestamp on/after the date
modified_before:YYYY-MM-DD	Most recent version's modified timestamp before the date
modified_after:YYYY-MM-DD	Most recent version's modified timestamp on/after the date
published_before:YYYY-MM-DD	Most recent published version's created timestamp before the date (draft-only dandisets are excluded)
published_after:YYYY-MM-DD	Most recent published version's created timestamp on/after the date

created_after:2024-01-01                    # everything created since 2024
modified_after:2025-01-01 modified_before:2026-01-01   # changed during 2025
published_after:2023-01-01                  # published since 2023

Asset content

Substring matches (case-insensitive) against the dandiset's asset metadata.
A dandiset matches if at least one of its assets satisfies the predicate.
Multiple asset operators are AND'd at the dandiset level — each must be
satisfied by some asset, but not necessarily the same one. See
How operators combine above.

Operator	What it matches
species:VALUE	Substring against any wasAttributedTo[].species.name
approach:VALUE	Substring against any approach[].name
technique:VALUE	Substring against any measurementTechnique[].name
standard:VALUE	Substring against any dataStandard[].name
file_type:VALUE	encodingFormat startswith. Accepts the aliases nwb, image, text, video, or any MIME prefix (application/x-nwb, image/, ...)

species:mouse                          # House mouse, Mus musculus, etc.
species:"Mus musculus"                 # exact-ish phrase match
approach:electrophysiological          # any contributor's approach contains this
technique:"spike sorting"
standard:nwb
file_type:image                        # → image/* mime types
file_type:application/x-nwb            # explicit MIME prefix

Owner

Operator	What it matches
owner:VALUE	Dandisets owned by users matching VALUE (case-insensitive) against username, email, first_name, last_name, or "first_name last_name"

owner:alice
owner:alice@example.com
owner:Smith                            # any user named Smith
owner:"Jane Doe"                       # full display name

If a name matches multiple users (e.g. two Smiths), dandisets owned by any
of them are returned.

Contributors

The contributor operators search the dandiset's metadata.contributor[] list
(the same data shown in the "Contributors" section on the landing page). Each
operator matches a contributor by name, email, OR identifier —
which means ORCID for Person contributors (0000-0002-2990-9889) and ROR URL
for Organization contributors (https://ror.org/01cwqze88) both work. Bare-ID
substrings (01cwqze88) match the full URL.

Operator	Role constraint
contributor:VALUE	Any role (catch-all)
author:VALUE	Must hold the Author role
contact_person:VALUE	Must hold the ContactPerson role
data_collector:VALUE	Must hold the DataCollector role
data_curator:VALUE	Must hold the DataCurator role
data_manager:VALUE	Must hold the DataManager role
maintainer:VALUE	Must hold the Maintainer role
project_leader:VALUE	Must hold the ProjectLeader role
funder:VALUE	Must hold the Funder role
sponsor:VALUE	Must hold the Sponsor role

contributor:"Doe, Jane"                # any role
author:Doe                             # Doe specifically as an Author
data_curator:0000-0002-2990-9889       # this ORCID, must be a DataCurator
funder:NIH                             # NIH (or any string containing NIH) as Funder
funder:01cwqze88                       # by ROR id
author:Doe funder:NIH                  # both must hold (possibly different people)

The role-restricting operators map to the DANDI schema's RoleType
values. The catch-all contributor: covers any other role
(Conceptualization, Researcher, etc.); for those, filter by name and use the
landing page to check the specific role.

Affiliation

affiliation is special — affiliations live in a nested field
(contributor[].affiliation[]), not as a role on the contributor itself. The
operator queries that path:

Operator	What it matches
affiliation:VALUE	Substring against any contributor's affiliation name OR identifier (ROR URL)

affiliation:Stanford                       # any contributor affiliated with Stanford
affiliation:"University College London"
affiliation:00f54p054                      # Stanford's ROR id (substring of the URL)
author:Doe affiliation:Stanford            # Doe as author AND someone Stanford-affiliated

---

Recipes

Find recent NWB dandisets from a particular lab.

file_type:nwb affiliation:"University College London" published_after:2024-01-01

Find dandisets where I'm the contact person.

contact_person:"My Name"

Find dandisets funded by NIH with mouse data.

funder:NIH species:mouse

Find dandisets that cite a particular ORCID as an author.

author:0000-0002-2990-9889

Find your own dandisets in the listing.

owner:"Your Name"

(Or use the My Dandisets tab if you're signed in — it's the same set.)

---

Quoting rules

• Wrap a multi-word value in double quotes:
  technique:"spike sorting", contributor:"Doe, Jane",
  affiliation:"Cold Spring Harbor Laboratory".
• Wrap a whole token in double quotes to opt out of operator parsing —
  useful when the text you're searching for contains a colon:
  "foo:bar" searches for the literal text foo:bar.
• Unbalanced quotes return a 400 with a friendly error message.

---

Error messages

Invalid syntax doesn't fail silently. Common cases:

What you type	What you get back
specie:mouse	400 — Unknown search operator "specie". Did you mean "species"?
data_curatr:Doe	400 — Did you mean "data_curator"?
created_after:not-a-date	400 — Invalid date for "created_after"; Use YYYY-MM-DD.
hello "world	400 — Unbalanced quote in search query. Remove the stray quote...
owner: (empty value)	400 — Operator "owner" requires a value

Typo suggestions are produced by difflib.get_close_matches;
they're a hint, not authoritative.

---

Using from the API

The same syntax works against the REST API — the search string lives in the
?search= query parameter on /api/dandisets/:

curl 'https://api.dandiarchive.org/api/dandisets/?search=species:mouse+author:Doe'

import requests
r = requests.get(
    'https://api.dandiarchive.org/api/dandisets/',
    params={'search': 'species:mouse author:Doe', 'draft': 'true', 'empty': 'true'},
)
r.json()

The OpenAPI description on /swagger/ lists every operator inline.

---

Limitations and notes

• Substring, case-insensitive. species:mouse matches House mouse,
  Mus musculus, etc. There's no exact-match mode at the moment — use a longer
  substring to narrow.
• No OR or NOT. Operators always combine with AND. To express OR, run two
  queries (or wait for a future revision; see below).
• No nesting. (species:mouse OR species:rat) and similar grammar isn't
  supported.
• AND combines at the dandiset level for assets and contributors. Each
  asset operator filters dandisets independently — different operators may
  match different assets within the same dandiset. Contributor operators
  combine on the same version's contributor list (so a draft + published
  version with disjoint contributors don't combine into a spurious match);
  within that single version, different contributor operators may match
  different entries of contributor[].
• ?user=me (an existing query parameter) still works for "my dandisets";
  there's no owner:me magic alias in the operator syntax.
• Free-text and operators combine. The same ?search= parameter accepts
  both, so you don't need a different endpoint depending on whether you have
  operators.

[yarikoptic]

Thank you @bendichter . Great work -- I think we are converging. Could you please refine PR description to correspond to current changes, since I think 25 role-specific operators is no longer there and potentially other aspects? note that I also folded that extended paste on advanced search in a most recent comment.

edit: also rebase/merge master to get advantage of #2820 since now renders skinny

[yarikoptic]

overall looks great to my eye... let's see if more eyes could have a peek

[yarikoptic]

I feel we would really need a URL to docs there now ... best even not to delay but preempt location?

[bendichter]

dandi/dandi-docs#238

[candleindark]

The PR description is outdated at this point. A update can help conveying the intent of the PR more clearly. (For example, the trimming of the role-specific operators in 5240487 is not reflected in the description.)

[bendichter]

@yarikoptic @candleindark — refreshed the PR description: dropped the stale "25 role-specific operators" framing (now 9), called out the case-insensitive operator keys, the explicit-allowlist rationale, and the RoleType drift-guard test. Also rebased on master to pick up #2820 (skinny popover render) — both branches force-pushed. tox -e test is green locally (53 advanced-search + parser tests pass).

[candleindark]

It may be a good idea to add a test assure that there is no spurious match. Such a test can safeguard against a future that break this behavior.

[bendichter]

Good idea — added in 0cbdca7. The new test_advanced_search_contributor_operators_and_on_same_version seeds a dandiset whose draft has Author=Doe and whose published version has Funder=NIH (no contributor overlap), then asserts author:Doe funder:NIH rejects it (while a positive-control dandiset with both contributors on the same version still matches). A future change that ANDs per-operator subqueries against unrelated Version rows would let the spurious match through and trip the test.

All concerns I raised have been addressed, deferring to other reviewers for full approval.