feat(panel): store concatenated panels and bump each source on read by elhb · Pull Request #478 · PixelgenTechnologies/pixelator

elhb · 2026-10-01T15:41:09Z

Description

Several --panel inputs are concatenated into one panel (marker_id, sequence_1, and sequence_2 stay unique across that panel).
Each input panel is one source, and each marker keeps a link to the source it came from.
A concatenated panel is stored in the .pxl file. It cannot be registered in the config. add_panel only accepts a single source panels.

New .pxl files store the panel in the panels and panel_sources tables.
Redundant storage of the same info is removed:

Panel columns are not written onto var, and uns['panel_metadata'] is no longer written. instead, the panel columns are joined onto the in-memory AnnData, on read.
Pixel-file metadata no longer stores panel_name or panel_version.

Files from pixelator 0.22.0 through 0.30.0 are still read from uns['panel_metadata'] and var. Denoise still reads panel_name for files from before 0.22.0.

When several files are read together, each source is bumped to the newest patch those files carry. A source is one input panel. Sources match on name, product, and major.minor. The bump renames markers in the in-memory edgelist, proximity, and layouts. The file on disk is not rewritten. Sample-called files keep hashing clones collapsed.

Denoise rebuilds the count matrix from the markers already in var, so hashing clones that sample calling removed do not come back. The full panel is still written to the tables.

from_adata is removed. It only rebuilt a panel from an AnnData that still carried the on-disk panel fields, and a loaded dataset no longer has those. Legacy .pxl files are still read directly. No caller in this repository used from_adata on an AnnData that was not the pxl file.

Fixes: PNA-3624

Type of change

New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

How Has This Been Tested?

Concatenate two panels and confirm marker ids and sequences must be unique.
Round-trip a new .pxl and confirm the panel comes from the tables, not from var or uns.
Open a 0.22.0–0.30.0 file and confirm the panel is rebuilt from uns and var.
Read two samples that share a source at different patch versions and confirm only that source is bumped, including edgelist, proximity, and layouts.
Run sample calling, then a patch bump, and confirm hashing clones stay collapsed and a missing non-hashing marker still fails.
Denoise a pre-0.22 file and confirm it still falls back to panel_name.
Denoise a sample-called file and confirm hashing clones removed from var do not come back.

PR checklist:

This comment contains a description of changes (with reason).
I have performed a self-review of my own code
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
If a new tool or package is included, I have updated dependencies in pyproject.toml and cited it properly
I have checked my code and documentation and corrected any misspellings
I have documented any significant changes to the code in CHANGELOG.md

Note

High Risk
Breaking .pxl panel storage and metadata, plus automatic patch alignment and hashing rename rules, affect denoise, sample calling, and multi-file analysis paths where marker identity must stay consistent.

Overview
This PR redefines how antibody panels are defined, stored, and reconciled across the PNA pipeline.

CLI and panel model: single-cell-pna demux, collapse, and graph now accept repeated --panel flags; inputs are concatenated into one panel (unique marker_id and sequences), with each CSV as a tracked source. PNAAntibodyPanel.from_adata is removed; panels come from the .pxl file or config via load_antibody_panels / read_dataset_panel.

On-disk format: New .pxl files persist panels in panels and panel_sources DuckDB tables. Panel columns are not duplicated on var, uns['panel_metadata'] is dropped, and file metadata no longer carries panel_name / panel_version. Rewriting legacy 0.22–0.30 files migrates panel data from uns/var into those tables.

Multi-sample reads: When several files are loaded together, align_panel_patches bumps each source to the newest patch shared across files (match on name, product, major.minor). Renames apply in memory to AnnData, edgelist, proximity, and layouts (session views); disks are unchanged. Sample-called files respect hashing_collapsed so hashing clones stay collapsed.

Pipeline fixes: Denoise reads the panel from the file and rebuilds counts using markers already in var, so clones removed by sample calling are not reintroduced. Sample calling uses structured hashing id rules and writes hashing_collapsed: true. Edgelist.to_record_batches now renames legacy marker1/marker2 like other exporters.

^{Reviewed by Cursor Bugbot for commit c818e23. Bugbot is set up for automated code reviews on this repo. Configure here.}