Skip to content

Add specificity databases and immvoke annotate - #9

Open
ssnn-airr wants to merge 51 commits into
devfrom
annotation-dbs
Open

ssnn-airr wants to merge 51 commits into
devfrom
annotation-dbs

Conversation

@ssnn-airr

@ssnn-airr ssnn-airr commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds antigen specificity annotation to immvoke in two separate steps:

  1. Download public specificity databases (immvoke specificity).
  2. Annotate your own AIRR repertoire against one of them (immvoke annotate).

Usage is in Sources/Specificity/README.md; the matching rules and design are in idea.md. Those two files are the best place to start reviewing.

immvoke specificity: database downloads

  • Databases: IEDB, CoV-AbDab and ASD, via immvoke specificity <db> search|download, immvoke specificity all download, and immvoke specificity bcr as a shortcut for IEDB's BCR table.
  • IEDB BCR table: taken from IEDB's own AIRR export.
    • schema-validated, one record per chain, linked by cell_id
    • uses IEDB's calculated (IMGT standardized) gene calls
    • keeps human receptors only
  • Layout: each database gets its own folder, so several can share one --outdir:
    databases/iedb/
    ├── bcr.tsv                  # the reference for annotate
    ├── bcr.tsv.validation.txt
    ├── download_metadata.yml    # source, license, citation, time, command
    └── raw/
    

immvoke annotate

Matches a query AIRR table against a reference table:

Option Choices
--sequence-region CDR3H (default), VDJ, CDRs (CDR1-3 pooled)
--partition-by none, V, VJ, VJL (same gene, any allele; VJL also requires the same CDR3 length)
--identity-threshold exact by default, or a fraction in (0, 1]
--distance-metric hamming, levenshtein
--paired heavy and light chain must match the same reference antibody; --count-field picks one chain of each per cell
  • Matching engine: symscan finds the hits.
  • Rows beyond symscan's limits: these (over 14 differences, or very long sequences at loose thresholds) are scored directly in numpy instead, so no row misses a hit.
  • Gene calls: compared at gene level, any allele.
  • X residues: match any residue under exact and Hamming matching.
  • Blank values: never match.

Output: --outdir holds three files, each named after the reference's database and table:

annotation/F09_exact/
├── iedb_bcr_annotation.tsv            # one row per query (by its sequence_id): its hits
├── iedb_bcr_annotation_pairs.tsv      # one scored row per (query, reference) pair
└── iedb_bcr_annotation_metadata.yml   # options, inputs, reference database, counts, version, command
  • Exact runs: the pairs file lists the hits.
  • Hamming and Levenshtein runs: it lists every pair through the chosen filters, whatever its distance, with within_threshold marking the hits.

Dependencies

  • New: symscan>=0.8 (matching), pyarrow>=16 and gdown>=5.0 (for ASD's Parquet data on Google Drive).
  • Unchanged: the pandas and numpy floors.

Tests and CI

  • Test location: Specificity's tests live beside its source. CI runs them as a separate step, and they are excluded from the wheel.
  • Correctness: symscan's results and the direct scorer (Distances.py) are both checked against a brute-force matcher on seeded random tables covering X residues, blanks, duplicate IDs, gene and exact gates, pooled columns and fractional thresholds.
  • Real data: checked on one sample (F09, about 21k heavy chains) against IEDB.
  • CI: all checks pass (lint, docs, min-deps, Python 3.11–3.14, wheel).

Known limits

  • Reference tables: only IEDB's bcr table is AIRR so far. CoV-AbDab and ASD download in their own columns and can't be used as a reference yet.
  • Receptor types: antibodies only, no TCR. No BLOSUM scoring.
  • Query cdr3_aa: the query needs cdr3_aa in the AIRR definition (without the flanking C and W/F). Change-O clone-pass tables have only junction_aa, so cdr3_aa must be added before annotating.
  • Pairs file size: without --partition-by, a Hamming or Levenshtein pairs file covers every query against every reference: millions of rows per sample under Hamming, about 10⁸ under Levenshtein. --partition-by VJ keeps it manageable.

🤖 Generated with Claude Code

pramodsshinde and others added 27 commits August 14, 2026 11:02
Brings annotation-dbs up to date with dev, including the sourcerer ->
immvoke rename. Sources/Specificity and the iedb, covabdab and asd schema
snapshots move to src/immvoke, and their imports, CLI text and provenance
column names (sourcerer_* -> immvoke_*) follow the rename.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add Sources/Specificity/Symscan.py, ported from annotate_specificity:
exact, Hamming and Levenshtein matching through symscan's
symmetric-deletion index, returning the same result tables as the
existing CompAIRR and pure-Python functions. `immvoke annotate` gains
--backend {symscan,compairr,python}, defaulting to symscan for every mode
including --paired.

Unlike the original, rows symscan cannot compare are split off rather
than sending the whole run elsewhere: 'X' wildcard rows, and groups whose
deletion-variant index would exceed a fixed budget or whose distance
exceeds 14 (either of which aborts the process inside symscan), are
compared in Python and merged. Queries are grouped by distance (and by
length under Hamming) and chunked so each symscan call stays in budget.

runFuzzyAnnotation now resolves --identity through _allowedDifferences,
so float error can no longer drop its threshold by one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Specificity keeps its tests beside its source, which top level
`unittest discover` never reaches, so CI and the release workflow ran
none of them. Discover them in a second pass. Also sort the import
blocks ruff flagged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The README, NEWS and `immvoke annotate --help` now name only the symscan
and python backends. The compairr backend still runs --exact/--paired
when asked for by name, but --compairr-bin is hidden from help and an
invalid --backend is reported against symscan and python only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pyarrow wheels before 16 were built against numpy 1.x and fail to import
under numpy 2, which the min-deps CI job resolves to, so ASD's Parquet
reader test failed there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
IEDB recommends its calculated gene calls for analysis: they are always
valid IMGT nomenclature, while curated calls carry legacy names (TCRBV19)
and family level ones (IGHV3). CDRs keep preferring the curated value.
immvoke annotate is now a thin wrapper over Annotate.annotateSpecificity(),
following the flow in Specificity/idea.md: --paired, or heavy chain only ->
--sequence-region (CDR3, VDJ, CDRs) -> --partition-by (V, VJ, VJL) ->
--identity-threshold and --distance-metric (hamming, levenshtein). This
replaces --exact/--fuzzy/--identity/--levenshtein/--compare-cols.

- V/J-gene calls are parsed (Genes.py, Change-O's gene pattern) and compared
  at --gene-match level: allele, gene (default) or family. Several calls in
  one cell agree if any does; blank or unrecognized calls are unknown and
  agree with anything.
- symscan runs every mode, with no install check and no fallback; the pure
  Python engine runs only when chosen with --backend python.
- The reference must be an AIRR rearrangement table (every AIRR required
  field present); anything else is refused, as is a table left with no IGH
  rows, rather than returning an empty result.
- Remove the CompAIRR exact matcher and runSymscanExactAnnotation, no
  longer used.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
README.md now covers running annotate: options, examples and the output
columns. idea.md holds the workflow diagram, the AIRR input contract, the
matching rules and the engine notes, with nothing repeated between them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
IEDB publishes its receptors in AIRR format (airr_full.zip, refreshed
weekly), so bcr and tcr now read bcr_full_airr.tsv / tcr_full_airr.tsv from
it as is, adding only provenance columns. This drops the converter for the
receptor_full_v3 CSV (two-row header, both chains side by side). Gene calls
and CDRs are IEDB's calculated values; cell_id is per receptor and epitope.

`immvoke specificity bcr` no longer takes --strict-airr: the written table
always keeps IEDB's own columns (epitope, assay, ...) and every AIRR
required field, blank where IEDB lacks one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Remove the pure-Python matchers and --backend: symscan is the only
  engine. Tests now check it against a brute-force matcher (tests/
  BruteForce.py) instead.
- X matches any residue under exact and Hamming matching: reference X
  positions are cut out before symscan searches, query X's get slack, and
  pairs involving an X are re-scored.
- Both query and reference sides are chunked to symscan's budget; rows
  that cannot fit at all (over 14 differences, or a sequence too long to
  index) are skipped with a warning naming them, not compared in Python.
- A blank sequence value never matches, nor does a blank or unrecognized
  V/J-gene call; a gene-only call (IGHV1-2) never matches at allele level.
- A blank locus is filled from the row's V call, else its J call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Annotation matches through symscan only now; nothing imports rapidfuzz.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bcr and tcr keep a receptor when it has a human chain (locus_species
NCBITAXON:9606) and no chain of another species; a chain with no species
recorded stays with its human partner, so pairs are never split. Decided
per receptor in a first pass over cell_id and locus_species.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Remove --gene-match / gene_match and its allele and family levels: calls
now always agree on the same gene, whatever the allele (IGHV1-2*02 ==
IGHV1-2*04 == IGHV1-2). A cell with several calls agrees if any one does;
a blank or unrecognized call agrees with nothing, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- --sequence-region's CDR3 is now CDR3H (the default), naming it the heavy
  chain CDR3; it still compares AIRR's cdr3_aa on IGH rows.
- A blank locus is no longer filled from the gene calls: it stays as the
  reference has it, so such a row is neither heavy nor light.
- idea.md shows the workflow as a step table instead of a Mermaid diagram,
  and states that cdr3_aa must follow the AIRR definition (no flanking
  C/W), which annotate does not check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Downloads: every specificity download lands in <outdir>/<db>/ -- the
  normalized <table>.tsv at its top, raw/ beside it, its own
  download_metadata.yml -- so databases can share an outdir; the
  specificity/ subfolder is gone. `specificity bcr` defaults to
  --outdir databases.
- annotate: --outdir (outdir in Python) replaces -o/--out and writes
  annotation.tsv, annotation_pairs.tsv and annotation_metadata.yml. The
  metadata records the options, query, reference and its source database
  (read from the download_metadata.yml beside it), counts, skipped rows,
  version, time and command.
- annotation_pairs.tsv has one row per matched (query, reference) pair
  with its own mismatches and identity (plus mismatch_positions under
  Hamming); under --paired, cell_id, hit_id and matched_chains.
- --paired: --count-field (duplicate_count by default) keeps one heavy and
  one light chain per query cell, the highest count, the first row on a
  tie, a blank count or no such column.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- NEWS: replace the removed --exact/--fuzzy/--backend options with
  annotate's current ones, and note the per-database download folders and
  the --outdir bundle.
- idea.md: add the --count-field step and annotation_metadata.yml, and
  correct the VDJ limit to about 0.97 (4 differences on ~110-130 residues,
  measured), not 0.99.
- Docstrings: drop a reference to the removed runFuzzyAnnotation, describe
  the pairs table and outdir in Annotate's module and section notes, and
  say where each command writes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- annotation_metadata.yml: numpy option values (e.g. a threshold from
  numpy.arange) are written as plain values instead of failing, and the
  YAML is rendered before any file is written, so a failure leaves no
  partial bundle.
- --paired: rows with a blank cell_id are left out, with a warning,
  instead of being lumped together as one antibody (where --count-field
  would then drop all but one of their chains).
- Fill blanks as '' through object dtype: a run with no hit at all left
  hit_ids/mismatch_positions float, and pandas 4 refuses a str fill there
  (a Pandas4Warning today).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Output files are named after the reference's database and table, read
  off the download_metadata.yml beside it: iedb_bcr_annotation.tsv, ...;
  a reference without one by its file name, a DataFrame by 'reference'.
- The pairs file of a Hamming or Levenshtein run lists every pair through
  the chosen filters (gene gate, VJL length, same length under Hamming),
  whatever its distance, with within_threshold marking the hits. Scored
  directly in numpy (new Distances.py), streamed in query blocks, and
  checked against the brute-force matcher and symscan's hits. Its id
  column is now reference_id; identity is written to 4 decimals.
- Queries are named by their own sequence_id instead of query_row, in the
  result, the pairs and symscan's warning; query_row stays for a query
  without that column.
- Rows beyond symscan's limits are scored directly instead of skipped, so
  no row misses a hit (e.g. full VDJ at 0.9 identity); the metadata counts
  them as query_rows_scored_directly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@pramodsshinde pramodsshinde changed the title Annotation dbs Add specificity databases and immvoke annotate Oct 23, 2026
@pramodsshinde
pramodsshinde marked this pull request as ready for review October 23, 2026 15:19

@ggabernet ggabernet left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

General comments:

The immvoke specificity iedb downlaod all --outdir . is either haning or very slow. Could you fix that or investigate why is it slow? It's stuck at the tcr.tsv download.

  1. General comments for immvoke specificity download:
  • allow choosing species with additional param (air_species). If not specified then download all.
  • allow choosing assay outcomes with additional param. If not specified then download all.
  • allow choosing antigen type with:
    "immvoke bcr
    -allow choosing antigen species with (if applicable). Follow the same vocabulary as in iedb:
    "immvoke specificity iedb download --antigen-species "Influenza A"

@@ -0,0 +1,28 @@
source: iedb

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are these schemas for?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is consistent with what @ssnn-airr has for the base repo. I read it is more like a config file when reading the database in the download/search options.

Comment thread src/immvoke/Sources/Specificity/Iedb.py Resolved
Comment thread src/immvoke/Sources/Specificity/Iedb.py Resolved
Comment thread src/immvoke/Sources/Specificity/Iedb.py Resolved
Comment thread src/immvoke/Sources/Specificity/Iedb.py Resolved
Comment thread src/immvoke/Sources/Specificity/Iedb.py Resolved
pramodsshinde and others added 24 commits October 28, 2026 17:39
…on log

IEDB's raw/ folder (the bulk ZIP and the tables extracted from it) is only
an intermediate step, so a new keep_raw source attribute lets the
specificity download delete it once every table is written; IEDB turns it
off, every other source keeps raw/ as before.

The AIRR validation report is now plain sentences with an overall
PASS/FAIL verdict, named <table>.airr_validation.log rather than
<table>.tsv.validation.txt, so it is not mistaken for a download check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`immvoke specificity bcr` was a convenience alias from when BCR was the
only annotate reference: it called handleSpecificityDownload with
db='iedb', table='bcr' and a default --outdir, and offered fewer options
than the full command. Two ways to fetch one table is more to document
and maintain, so it goes; `immvoke specificity iedb download bcr --outdir
<dir>` is the one command, and the docs and annotate help now name it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The receptor tables were limited to human receptors. Now every species is
kept unless --air_species names one, from IEDB's own species names (43,
listed in RECEPTOR_SPECIES and the schema snapshot), mapped to the
locus_species codes the AIRR export records. A species includes its
strains: "Mus musculus" also keeps BALB/c, C57BL/6, Xenomouse, ...;
codes from 10000000 up are IEDB's strain codes, each linked to its parent.
A receptor is kept only when no chain is of another species, as before.

Schema-generated filter flags now keep the field's underscores
(--air_species, --qualitative_measure), and specificity tables list every
allowed value in --help, since `schema show` does not cover them.
NEWS no longer mentions the removed `specificity bcr` shortcut.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ggabernet asked for assay outcomes to be choosable, every outcome kept when
none is named. bcr and tcr take --outcome Positive|Negative, matched
case-insensitively against iedb_assay_outcome (IEDB writes it in lower
case); bcell's --qualitative_measure becomes --outcome with its five
grades, an exact match, so Positive does not take in Positive-High. Both
are applied client-side in _applyClientFilter, from one table -> column
map, and --outcome combines with --air_species.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`immvoke specificity <db> download <table>` and `specificity all
download` now take --dry_run, --no_resume and --strict_airr, matching the
schema-generated filter flags (--air_species, --outcome). annotate and the
OAS and germline downloads keep their dashed flags.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ggabernet asked to filter by antigen species in IEDB's own vocabulary.
--antigen_species takes any antigen organism name IEDB uses, ignoring case
and, when that leaves one organism, its bracketed common name. The AIRR
export records each row's epitope but not its organism, so when the flag
is given, the table's epitopes are looked up on IEDB's epitope_search in
batches of 300; its source_organism_iri_search lists every organism above
an epitope's, so "Influenza A virus" keeps every flu strain. An unknown
name fails with the closest IEDB names. About 15 s for bcr.

Schema fields gain open_values for a vocabulary the source checks itself
(written to YAML only where set, so other snapshots are unchanged); the
schema keeps a few names as examples for --help.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A table copied out of its database folder now still says where it came
from. annotate's output prefix stays iedb_bcr (not iedb_iedb_bcr), and an
older bcr.tsv still gets its database added. README and NEWS drop the raw/
folder IEDB no longer keeps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ggabernet asked for one combined BCR output rather than extra tables. Each
record of IEDB's AIRR export already carries its epitope, assay ids and
outcome, which is what bcr_to_bcell linked; bcell_search is every B-cell
assay in IEDB (~1.7M rows, mostly without a receptor), paged 2,000 at a
time. So both go, with the API table download, the JSON reader and the
assay path in normalizeChunk; the API is now used only for
--antigen_species.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ggabernet saw `iedb download all` hang at tcr.tsv. Normalizing IEDB's
338,000 row tcr table took 46 s, 45 of them in apply(_rowHash, axis=1),
which builds a Series per row. rowHashes (new RowHash.py, shared by IEDB,
CoV-AbDab and ASD in place of three copies) turns each column to text with
str() and joins them per row: the same key, so every hash is unchanged
(checked on all 357,146 bcr and tcr rows), in 0.8 s. tcr now normalizes in
under 2 s and writes, AIRR-validated, in about 11 s more.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…filters

`immvoke specificity <db> search <table>` printed the download URL and
ignored its filters. It now fetches the table into a temporary folder,
applies the filters and reports rows, distinct receptor groups and
epitopes, and the most common loci, receptor species and outcomes (--top,
10 by default), as each source lists them in search_columns; species codes
are named through labelValue. --out writes every count as a TSV. The code
is in the new Specificity/Search.py.

Filters on search and download are grouped in --help, each with a
placeholder and what it filters on (filter_help), values packed without
splitting a name, and no "(default: None)". --limit stays only on
`download all`, the one place it had an effect.

IEDB's bulk ZIP moves to ~/.cache/immvoke/iedb ($XDG_CACHE_HOME), reused
for a day, so search, download, and bcr then tcr share one copy; tests
point the cache at a temporary folder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ggabernet wants annotate to report each hit's antigen, epitope and antigen
source species. IEDB's AIRR export has the epitope (iedb_epitope_name) but
not its antigen, so bcr and tcr gain iedb_antigen and
iedb_antigen_source_species after it, from curated_source_antigens on
IEDB's epitope_search (several ' | ' joined). The table's epitopes are now
always looked up there (_epitopeRecords), and --antigen_species filters on
the same records. The row hash still covers IEDB's export only. search
breaks the table down by antigen species too. Adds about 15 s to a bcr
download or search.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t-matrix

<db>_<table>_annotation.tsv now has sequence_id, n_hits, identity, hit_ids,
antigen, epitope and antigen_source_species: the hits with the highest
identity, all of them on a tie, ';' between hits and ' | ' between the
epitopes one hit stands for. The pairs file is gone; --full-dist-matrix
writes <db>_<table>_annotation_full_matrix.txt with every hit within the
threshold (with --paired, each antibody's hits and the chains matched).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
IEDB's epitope structure_type (Linear peptide, Discontinuous peptide,
Discontinuous peptide on multi chain, Non-peptidic), the same choice as
IEDB's own Epitope Structure search, looked up in the existing
epitope_search call. 'Discontinuous peptide' includes its multi chain
kind; the filter combines with --antigen_species. Each record gains
iedb_antigen_structure_type, and search breaks rows down by it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A download now prints 'Download started' with the table and filters, the
source's own steps in plain words (IEDB: the cached export's age, the
epitopes looked up, what each filter kept), and 'Download completed in N
s' with rows kept, receptor groups, epitopes and the AIRR validation
result, then where the provenance went. Counts are taken as the table is
written (Specificity/Report.py). Status lines drop their 'INFO ' prefix;
warnings and errors keep their level.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A heavy chain of the asked species used to keep a light chain IEDB gave
no species, so search showed '(not recorded)' under receptor species.
Such chains are now dropped, leaving their partner alone; a receptor with
a chain of another species is still dropped whole. The status line says
how many were dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s share

IEDB's bcr and tcr filter on the same fields, so 'iedb search all
--air_species ...' now works instead of rejecting the flag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
--assay_type keeps records with an assay of that type (neutralization,
3D structure, ...) and --disease those with an assay studying that
disease or one under it; given both, one assay must satisfy both. They
look up the table's assays on bcell_search/tcell_search, only when given.
--paired_only keeps receptors with both chains of a pair (IGH with IGK
or IGL, TRB with TRA, TRD with TRG), naming a chain with no locus by its
V or J gene. It is an on/off flag outside the schema (filter_switches).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A rejected filter value tells the user to run 'immvoke schema show
--source <db> ...', but the schema commands only accepted the repertoire
registry, so that hint named a command that failed. Both --source flags
now take either registry, and an open_values field says 'any value (N
examples)' rather than listing its examples as the vocabulary.

Two bugs this uncovered: schema refresh called harvestCatalog on every
source although only OAS defines one, so it crashed for the other six;
it now ends after the snapshot when a source has no catalog. And the
iedb snapshot was the only one missing reported_totals, so a refresh
would have reformatted the whole file; a test now checks every packaged
snapshot round-trips.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
On a real repertoire almost every query row matches nothing: F09 against
IEDB at 0.7 identity gave 21,412 rows of which 17 had a hit. The table
now holds only the rows that matched something, so it reads at a glance;
the counts it used to carry stay in the metadata, which already recorded
query_rows_compared beside query_rows_with_hits. The full matrix and
--paired are unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A run writes the whole bundle, but a rerun into the same --outdir without
--full-dist-matrix used to leave the earlier matrix in place, beside a
table and a metadata record it had just overwritten: a 0.7 matrix next to
an exact-match table, describing a comparison that no longer happened.
The matrix is now removed in that case, and said so in the log.

Also pins the matrix's contract in a test: the table reduces a query to
its best hits, the matrix reduces nothing and keeps every hit the
threshold let through.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
annotate used to print its progress as terse technical notes and no
timing, which is thin company for a run that takes a minute. It now opens
with what it is about to compare, in words rather than raw option values
('a hit is: 70% identity or better, hamming'), indents what it does in
between, and closes with how long it took and how many query rows matched
something. Report.py gains annotationStarted/annotationCompleted beside
the download's pair, and humanDuration no longer calls sub-second work
'0 s'.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
--partition-by was applied only to the pairs symscan returned, so every
query CDR3 was distance-compared against references of every V/J
combination and the gate then threw nearly all of it away. The index cost
grows combinatorially with the tolerance, so that waste dominated at
loose thresholds: F09 against IEDB took 57 s at 0.7 identity, and 0.5 did
not finish in an hour.

The search now runs once per gene key. Two rows pass the gate when they
name a gene in common in every gene column, which is exactly when their
key sets intersect, so grouping never separates a pair the gate would
keep; a row calling several genes joins each of their groups, and an
unparseable call joins none, as the gate already had it. The gate still
runs afterwards and still decides what is kept, so the answer does not
depend on the partition being right.

F09 at 0.7: 57 s -> 3 s, output identical. 0.5: over an hour -> 24 s.
Smaller indexes also push fewer rows past symscan's limits (22 -> 1 at
0.7), so fewer fall back to direct scoring.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The annotation table answers what a query matched, so
--identity-threshold decides it. The matrix answers how close a query is
to everything it was compared with, which the threshold has no business
deciding: it now holds every pair the filters admit, whatever its
distance. --partition-by still applies, so V, VJ or VJL chooses what was
compared at all, and under hamming a pair of different CDR3 length has no
distance to report and is left out rather than given a blank.

Built on Distances.allPairDistances, which already scores every gated
pair, and streamed to disk a block at a time: without --partition-by this
is every same-length pair of both tables, 14.6 million rows for a
repertoire against IEDB where VJ gives 28,671, so that case now warns.
The run's own counts gain full_matrix_rows, and the timing is reported
after the matrix is written rather than before it is computed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
<db>_<table>_annotation.tsv had one row per query, its hits joined into
each cell with ';' and a hit's several epitopes with ' | '. Neither
separator is safe: IEDB's own epitope names contain the first (1,576 of
them, 'V: F43; W: Y71, ...') and its antigen names the second, so 367 of
6,348 rows in a real run could not be parsed back. The table is now long
-- one row per (query row, matching reference row) -- so nothing is
joined and nothing can be mis-separated. n_hits goes with it, being a
row count now, and hit_ids becomes hit_id.

Each run also writes <db>_<table>_annotation_summary.txt: what was
compared, what came of it and what each other file holds, in prose, so
the folder explains itself without reading the YAML. The run's counts
gain annotation_rows.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@pramodsshinde

Copy link
Copy Markdown

@ggabernet Thanks for going over the repo and adding comments. I resolved most of the comments and concerns you raised! Please have another look when you get a chance and let me know if I missed anything or anything is still confusing.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants