Skip to content
HareeshDaxtonPublic

About

๐Ÿ‘๏ธ Multi-agent AI that drafts glaucoma screening reports from eye photos. A doctor always reviews and signs. Screening support only.

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

ย 

History

529 Commits

Folders and files

NameName
Last commit message
Last commit date
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 

Repository files navigation

CONSILIUM banner Typing SVG



V1 phases pilot screening only
tests coverage mypy ruff import-linter vitest playwright gitleaks



What it does โ€ข Architecture โ€ข Golden rules โ€ข Evaluation โ€ข Quick start โ€ข API โ€ข Roadmap


Caution

CONSILIUM is screening support, not a diagnostic device. Every report carries the line "Screening support only; not a diagnosis." Nothing becomes final until a human clinician reviews and signs it. V1 is cleared for a supervised pilot on synthetic or consented research data only โ€” not for real patients.


๐Ÿงญ Table of Contents

Click to collapse
  1. Why CONSILIUM exists
  2. What it does
  3. Architecture
  4. The pipeline, stage by stage
  5. Who sees what โ€” the Context Router
  6. The verification loop
  7. The 14 golden rules
  8. Tech stack
  9. Evaluation โ€” honest numbers
  10. Privacy & security
  11. Quick start
  12. Configuration
  13. API
  14. Project structure
  15. Testing & quality gates
  16. Observability
  17. Differences from the research paper
  18. Roadmap & open items
  19. Documentation map
  20. Team

๐Ÿ’ก Why CONSILIUM exists

Glaucoma damages the optic nerve slowly and silently; by the time a patient notices, vision loss is permanent. Screening a fundus photograph for an enlarged optic cup relative to the disc is one of the cheapest early signals โ€” but turning a photo plus a doctor's note into a careful, multi-perspective report takes specialist time.

Multi-agent LLM systems look like an obvious fit. They have a known failure mode, though:

๐Ÿ” Consensus collapse

When several agents read the same context, they converge. They agree with each other, repeat each other's mistakes, and a minority opinion that might have mattered quietly vanishes.

๐Ÿงฑ CONSILIUM's structural answer

  • Different views: each specialist gets a role-specific slice of the facts, decided by fixed code, never by an LLM.
  • Hard checks: the combined draft is verified against the facts by rules and a separate critic model.
  • Human accountability: a clinician signs, and their edits are recorded with reason codes.

โœจ What it does

๐Ÿ“ท fundus photo  +  ๐Ÿ“ doctor's note (optional)  +  ๐Ÿ‘๏ธ which eye
                              โ”‚
                              โ–ผ
        โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
        โ”‚  measure  โ†’  route  โ†’  specialise  โ†’  draft โ”‚
        โ”‚        โ†’  verify  โ†’  critique  โ†’  revise    โ”‚
        โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                              โ”‚
                              โ–ผ
   ๐Ÿ“„ draft report  โ†’  ๐Ÿฉบ clinician review & sign  โ†’  โฌ‡๏ธ PDF / Word
๐Ÿ”ฌ
Measures
SwinV2 glaucoma probability + SegFormer disc/cup masks โ†’ vertical CDR, quality gate, risk tier
๐Ÿง‘โ€โš•๏ธ
Specialises
Ophthalmologist, Optometrist and Pharmacist agents reason in parallel over isolated views
๐Ÿ›ก๏ธ
Verifies
7+ deterministic checks, then an independent LLM critic, with a bounded 3-round revise loop
โœ๏ธ
Defers
Clinician approves, edits (with reason codes) or rejects. Approved reports are immutable

๐Ÿ— Architecture

System topology

flowchart LR
    subgraph LAPTOP["๐Ÿ’ป Application host (CPU)"]
        direction TB
        UI["๐ŸŒ React + Vite UI<br/>:5173"]
        API["โšก FastAPI API<br/>:8000"]
        WK["โš™๏ธ Worker<br/>LangGraph runner"]
        PG[("๐Ÿ˜ PostgreSQL 16<br/>11 tables")]
        OS[("๐Ÿ—‚๏ธ Object store<br/>content-addressed")]
        UI -- "/v1 proxy" --> API
        API -- "job queue<br/>SKIP LOCKED" --> WK
        API --> PG
        WK --> PG
        API --> OS
        WK --> OS
    end

    subgraph GPU["๐ŸŽฎ Vision server (NVIDIA GPU)"]
        VS["Deva/server.py<br/>SwinV2-tiny + SegFormer<br/>pinned commits"]
    end

    subgraph OAI["๐Ÿค– OpenAI"]
        M1["gpt-4o-mini<br/>extractor ยท specialists ยท Director"]
        M2["gpt-6-luna<br/>Critic"]
    end

    WK -- "HTTPS ยท image bytes + bearer<br/>POST /predict" --> VS
    VS -- "p_raw + 2 RLE masks + revision" --> WK
    WK -- "facts + REDACTED note only<br/>never the image" --> OAI

    classDef host fill:#0f2027,stroke:#00c9a7,color:#ffffff,stroke-width:2px;
    classDef gpu fill:#2c1a4d,stroke:#b388ff,color:#ffffff,stroke-width:2px;
    classDef llm fill:#3a1f1f,stroke:#ff8a65,color:#ffffff,stroke-width:2px;
    classDef store fill:#13315c,stroke:#64b5f6,color:#ffffff;
    class UI,API,WK host;
    class VS gpu;
    class M1,M2 llm;
    class PG,OS store;
Loading

The case pipeline (LangGraph state machine)

flowchart TD
    A([๐Ÿ“ค POST /v1/cases]):::intake --> B[precheck_image]:::intake
    B --> C[input_rail<br/>injection ยท length ยท PHI redaction]:::intake
    C -.->|queued โ†’ worker| D[extract_note<br/>gpt-4o-mini ยท T=0]:::ai
    D --> E[call_vision<br/>SwinV2 + SegFormer]:::vision
    E --> F[build_context<br/>named fact sheet]:::det
    F --> G[route_context<br/>routing.yaml allowlist]:::det
    G --> H1[๐Ÿ‘๏ธ Ophthalmologist]:::ai
    G --> H2[๐Ÿ”ญ Optometrist]:::ai
    G --> H3[๐Ÿ’Š Pharmacist]:::ai
    H1 & H2 & H3 --> I[draft<br/>Director]:::ai
    I --> J{audit<br/>verifier โ†’ critic}:::check
    J -- "โœ… approved" --> K[output_rail<br/>disclaimer ยท no dose ยท no PHI]:::check
    J -- "โŒ issues ยท rounds left" --> R[revise<br/>Director gets issue list only]:::ai
    R --> J
    J -- "โŒ still failing at round 3" --> U[[โš ๏ธ NEEDS_ATTENTION<br/>draft marked UNVERIFIED]]:::warn
    K --> P[[๐Ÿ•’ PENDING_REVIEW]]:::human
    U --> V{๐Ÿฉบ Clinician review}:::human
    P --> V
    V -- approve --> Z1([โœ… APPROVED]):::done
    V -- "edit + reason code" --> Z2([โœ๏ธ APPROVED_WITH_EDITS]):::done
    V -- "reject + reason code" --> Z3([โ›” REJECTED]):::stop

    classDef intake fill:#1e3a5f,stroke:#64b5f6,color:#fff;
    classDef vision fill:#2c1a4d,stroke:#b388ff,color:#fff;
    classDef det fill:#0f3d3e,stroke:#00c9a7,color:#fff;
    classDef ai fill:#4a2511,stroke:#ffb74d,color:#fff;
    classDef check fill:#1b4332,stroke:#52b788,color:#fff;
    classDef warn fill:#5c1f1f,stroke:#ff6b6b,color:#fff;
    classDef human fill:#3d2c5c,stroke:#e0aaff,color:#fff;
    classDef done fill:#14532d,stroke:#4ade80,color:#fff;
    classDef stop fill:#4a1d1d,stroke:#f87171,color:#fff;
Loading

Legend: ๐ŸŸฆ intake ยท ๐ŸŸช vision ยท ๐ŸŸฉ deterministic code ยท ๐ŸŸง LLM ยท ๐ŸŸข checks ยท ๐ŸŸฅ attention ยท ๐ŸŸฃ human

Crash-safe by construction. After every node, one database transaction writes the status, a state snapshot, last_node, any failure, an audit event, the domain rows and agent telemetry. A crashed case resumes at the successor of its last persisted node โ€” vision and extraction are never recomputed.


๐Ÿ” The pipeline, stage by stage

1 ยท ๐Ÿ“ค Upload & image pre-check
  • Accepts PNG / JPEG, โ‰ค 20 MB, short side โ‰ฅ 256 px.
  • The original filename is never read or stored (filenames often contain patient names). The browser renames uploads to fundus.<ext>.
  • Images are stored content-addressed by SHA-256.
  • Idempotency-Key header: re-submitting returns the same case.
  • A doctor's report can be dropped in as PDF / Word / photo โ€” text is extracted inside the browser (pdfjs-dist, mammoth, tesseract.js); only the reviewed text is sent.
  • Rejections: IMAGE_EMPTY ยท IMAGE_TOO_LARGE ยท UNSUPPORTED_IMAGE_FORMAT ยท IMAGE_UNDECODABLE ยท LOW_RESOLUTION
2 ยท ๐Ÿ“ Note intake: safety rail โ†’ de-identification โ†’ extraction

The doctor's note is untrusted input (it could carry a prompt injection).

Step What happens
๐Ÿงฑ Input rail Rejects notes > 4000 chars, odd control characters, bad charsets, and instruction-like text (multilingual, homoglyph- and zero-width-normalised).
๐Ÿ•ถ๏ธ De-identification Regex policy phi_patterns.yaml v2: Aadhaar, Indian mobile, phone, email, SSN, MRN / UHID / OP-IP No / Reg. No, date of birth, titled names โ†’ [REDACTED:<rule>]. Only the rule name is stored, never the removed value.
๐Ÿงพ Extraction gpt-4o-mini at T=0, structured output bound to ClinicalNote (age, IOP OD/OS, meds, intolerances, comorbidities, prior Dx, exam findings, free text). โ‰ค 2 repair retries, then LLM_OUTPUT_INVALID.
๐Ÿ“‰ Confidence 8 fields counted: โ‰ฅ 4 HIGH, 1โ€“3 MODERATE, 0 LOW. LOW adds a caveat fact the report must mention.
๐Ÿ” Storage Redacted note stored AES-256-GCM encrypted. The raw note is never persisted and never sent to OpenAI.
3 ยท ๐Ÿ”ฌ Vision: remote inference, local interpretation

Models (Apache-2.0, fine-tuned on REFUGE), pinned to exact commits:

Model Task Pin
pamixsun/swinv2_tiny_for_glaucoma_classification glaucoma probability a25a03d9ff23โ€ฆ
pamixsun/segformer_for_optic_disc_cup_segmentation disc / cup masks a0463fb6c2bfโ€ฆ

The GPU server returns only p_raw, two RLE masks, mask size and endpoint_revision. All interpretation happens in this repo:

RLE decode โ†’ largest component โ†’ fill holes โ†’ clip cup to disc
          โ†’ measurements โ†’ CDR โ†’ mask QC โ†’ quality gate โ†’ calibration โ†’ tier
  • Vertical CDR = cup vertical extent รท disc vertical extent (primary); sqrt(cup_area / disc_area) (secondary).
  • Quality gate (inside the field of view): UNDEREXPOSED < 0.12 ยท OVEREXPOSED > 0.90 ยท BLURRED Laplacian variance < 1e-4 ยท NOT_A_FUNDUS_IMAGE mean(R) โˆ’ mean(B) < 0.05 ยท OPTIC_DISC_NOT_VISIBLE disc fraction outside 0.005โ€“0.30.
  • Tiers on calibrated probability: โ‰ฅ 0.70 HIGH ยท โ‰ฅ 0.30 INTERMEDIATE ยท else LOW (placeholders, flagged CLINICAL-REVIEW).
  • Revision pin: any mismatch with configs/models.yaml โ†’ VISION_REVISION_MISMATCH.
  • Transient errors retried with exponential backoff + jitter, bounded at 120 s โ†’ VISION_UNAVAILABLE. Fail closed โ€” never a fake report.
4 ยท ๐Ÿ—‚๏ธ Fact sheet: agents can only cite, never invent

Every piece of case information becomes a named fact with a stable ID from a single source (context/facts.py):

cad.p_glaucoma.tier        cad.cdr.vertical        cad.quality.flags      note.iop.od
cad.p_glaucoma.calibrated  cad.cdr.area_based      cad.maskqc.ok          note.meds
cad.disc.vdiam_px          cad.laterality          cad.caveat.<flag>      note.caveat.extraction

Free-text facts are wrapped in [untrusted-data source="clinician-note"] blocks inside prompts. Warnings become facts too, so the report is forced to carry them.

5 ยท ๐Ÿง‘โ€โš•๏ธ Three specialists, in parallel, in isolation
Agent Focus Hard limits
๐Ÿ‘๏ธ Ophthalmologist Disc measurements, risk tier, consistency with the note, escalation Never sees the medication list
๐Ÿ”ญ Optometrist Next steps: visual field, OCT, gonioscopy, repeat photo, referral urgency, image quality Never sees area-based CDR or raw disc/cup sizes
๐Ÿ’Š Pharmacist Medication safety by drug class only (e.g. ฮฒ-blockers + asthma) No doses, no brands; always "subject to prescriber judgement"; never sees eye measurements

Each returns a SubReport: findings with evidence_refs + confidence, recommendations with triggered_by, uncertainties, and what it declined as out of scope. asyncio.gather; never see each other's output.

6 ยท ๐ŸŽฌ The Director

Writes one DirectorDraft: impression, numbered claims (each citing fact IDs and the roles that raised it), disagreements kept visible ("left to clinician"), recommendations, limitations. NUMERIC claims must carry numeric_value + numeric_ref.

Prompt v2 rules: read the note before writing "no VF/OCT data", always state the eye ("eye not specified" when unknown), don't invent disagreements from rounding, always include the disclaimer.

7 ยท ๐Ÿฉบ Clinician review & export
  • Review screen: fundus photo with toggleable disc (cyan) / cup (magenta) overlay and opacity slider, claims with evidence chips, red highlights on flagged issues, field-level edit diff.
  • Actions: Approve ยท Approve with edits ยท Reject โ€” edits and rejects require a reason code: factual CAD error ยท clinical interpretation ยท knowledge gap ยท unsafe recommendation ยท omission ยท tone/wording ยท scope violation ยท other.
  • Self-approval refused server-side (separation of duties). Stale revisions refused.
  • UNVERIFIED drafts can only be approved once every open issue is resolved and the edited draft passes the deterministic verifier.
  • Approved reports are immutable; amendments create a new linked version.
  • One server-built report, three renderings: screen ยท PDF (fpdf2) ยท Word (python-docx).

๐Ÿ” Who sees what โ€” the Context Router

Isolation is the research claim, so it is enforced three times: by the router, by the specialist output validator, and by the verifier's role-scope check. A Hypothesis property test (โ‰ฅ 1000 examples) proves no forbidden fact ID, label or numeric value ever reaches a role's prompt.

Fact ๐Ÿ‘๏ธ Ophthal. ๐Ÿ”ญ Optom. ๐Ÿ’Š Pharm.
cad.p_glaucoma.tier ๐ŸŸข ๐ŸŸข ๐ŸŸข
cad.p_glaucoma.calibrated, cad.cdr.vertical ๐ŸŸข ๐ŸŸข โšซ
cad.cdr.area_based, disc/cup areas & diameters ๐ŸŸข โšซ โšซ
cad.quality.*, cad.caveat.*, cad.maskqc.ok ๐ŸŸข ๐ŸŸข โšซ
cad.laterality, note.age ๐ŸŸข ๐ŸŸข ๐ŸŸข
note.iop.od/os, note.prior_dx ๐ŸŸข ๐ŸŸข ๐ŸŸข
note.exam_findings, note.free_text ๐ŸŸข ๐ŸŸข โšซ
note.meds, note.intolerances, note.comorbidities โšซ โšซ ๐ŸŸข
note.caveat.extraction ๐ŸŸข ๐ŸŸข ๐ŸŸข

Deny-by-default allowlist in configs/routing.yaml. routing_version = SHA-256 of the YAML, stamped on every routed context. Policy is pending clinician sign-off.


๐Ÿ›ก The verification loop

stateDiagram-v2
    direction LR
    [*] --> Verifier: draft (rev n)
    Verifier --> Critic: no deterministic issues
    Verifier --> Revise: issues found
    Critic --> OutputRail: approved
    Critic --> Revise: issues found
    Revise --> Verifier: rev n+1 (rounds left)
    Revise --> Unverified: round = 3
    OutputRail --> PendingReview
    Unverified --> PendingReview: highlighted for clinician
    PendingReview --> [*]
Loading

โš™๏ธ Deterministic verifier (no AI)

Code Catches
BAD_REFERENCE cites a fact that doesn't exist
NUMERIC_MISMATCH number off by more than tolerance (CDR 0.02, p 0.01)
UNSUPPORTED stray numbers with no fact behind them
OMITTED_FINDING missing eye, tier, vertical CDR or caveat
UNSAFE_LANGUAGE diagnosis / dose / staging wording
DISAGREEMENT_SUPPRESSED real disagreement dropped
OUT_OF_SCOPE role cites a fact it couldn't see

๐Ÿง  LLM critic

  • Separate, stronger model: gpt-6-luna, reasoning effort medium, prompt v4.
  • Runs only when the verifier is clean.
  • Sees claims, disagreements, impression, recommendations, limitations + the fact sheet โ€” never specialist prose.
  • The Director, in revise mode, gets only the issue list, never the critic's reasoning.
  • approved โ‡” no issues in both layers.

Output rail before any clinician sees it: DRAFT_DEFINITIVE_DIAGNOSIS ยท DRAFT_DOSE_OR_REGIMEN ยท DRAFT_STAGE_SEVERITY ยท DRAFT_PHI_ECHO ยท DRAFT_MISSING_DISCLAIMER.


๐Ÿ“œ The 14 golden rules

These invariants live in AGENTS.md ยง2, the project's highest authority. If a change seems to need breaking one, the change stops.

# Rule
I-1 ๐Ÿ”ข AI never sees the photo and never invents or alters measurement numbers.
I-2 ๐Ÿงญ Who-sees-what is fixed code + routing.yaml, never an LLM decision.
I-3 ๐Ÿ”— Every factual statement cites facts its author was allowed to see.
I-4 โœ๏ธ Nothing is final without a clinician; only the review service writes APPROVED.
I-5 ๐Ÿšช Fail closed: errors become FAILED / NEEDS_ATTENTION with a reason โ€” never a silently degraded report.
I-6 ๐Ÿ•ถ๏ธ Minimum personal data: opaque IDs; no names/DOB/MRN in prompts, logs, traces or tests; no upload filenames.
I-7 โ˜ฃ๏ธ The doctor's note is untrusted input.
I-8 ๐Ÿ“ Every AI input and output is strictly schema-validated.
I-9 ๐Ÿท๏ธ Prompts are versioned files; a change requires an evaluation run; the version is recorded on every call.
I-10 ๐Ÿšซ V1: no RAG, no serving embeddings, no pgvector, no LLM fine-tuning.
I-11 ๐Ÿ“ No performance claim without a measured, stored number.
I-12 ๐Ÿฉบ Screening-support language only: no diagnosis, no staging unless the note says so, no doses.
I-13 ๐Ÿงฑ Vision only behind VisionClient, never in the API process; all interpretation in-repo.
I-14 ๐Ÿ–ผ๏ธ The photo never reaches the LLM provider, logs, traces or prompts.

๐Ÿงฐ Tech stack

Backend & pipeline

AI & vision

Frontend

Ops & quality

Model assignments
Role Model Temp Prompt Notes
Note extractor gpt-4o-mini-2024-07-18 0 v1 structured output โ†’ ClinicalNote
Ophthalmologist / Optometrist / Pharmacist gpt-4o-mini-2024-07-18 0.2 v1 โ‰ค 2000 output tokens each
Director gpt-4o-mini-2024-07-18 0.2 v2 โ‰ค 4000 output tokens
Critic gpt-6-luna 1.0 v4 reasoning medium, verbosity low, โ‰ค 12000 tokens

Per-case token budget: 100 000 (raises, never truncates). Cost: the gpt-4o-mini portion of a real report was โ‰ˆ 0.3 US cents (~12k in / ~2k out), plus 2โ€“3 critic calls.


๐Ÿ“Š Evaluation โ€” honest numbers

The research claim is tested as a four-stage ablation on the same cases, the same recorded vision output and the same model snapshots, with โ‰ฅ 3 samples per case and case-level bootstrap 95% CIs.

flowchart LR
    S1["<b>S1</b><br/>single agent<br/>full context"]:::s1 --> S2["<b>S2</b><br/>multi-agent<br/>shared context"]:::s2 --> S3["<b>S3</b><br/>multi-agent<br/>routed context"]:::s3 --> S4["<b>S4</b><br/>routed +<br/>verification<br/><i>= CONSILIUM</i>"]:::s4
    classDef s1 fill:#4a1d1d,stroke:#f87171,color:#fff;
    classDef s2 fill:#4a3511,stroke:#fbbf24,color:#fff;
    classDef s3 fill:#1e3a5f,stroke:#60a5fa,color:#fff;
    classDef s4 fill:#14532d,stroke:#4ade80,color:#fff;
Loading

Stored run eval-live-20261017T140231Z-d6ed738d โ€” 6 synthetic cases ร— 3 samples, all gpt-4o-mini:

Claim Comparison Result Verdict
Routing reduces convergence S3 โˆ’ S2, S_inter (cosine similarity) โˆ’0.050 [โˆ’0.083, โˆ’0.024] โœ… supported
Verification reduces omissions S4 โˆ’ S3, omission rate โˆ’0.256 โœ… supported
Full system beats single agent on grounding S4 โˆ’ S1, unsupported-claim rate โˆ’0.081 โœ… supported
Routing increases unique-fact coverage โ€” not shown โšช not supported
Verification reduces unsupported claims vs S3 โ€” not shown โšช not supported
๐Ÿ”ง How the critic was fixed (and why the negative result is published)

The first live run surfaced a real problem: the critic rejected everything.

Critic Model S4 UNVERIFIED rate Clean-draft false rejections Fault recall (combined) Status
v1 gpt-4o-mini 1.00 7 / 7 โ€” โŒ over-rejects
v2 gpt-4o-mini 0.83 4 / 7 100% โš ๏ธ better
v3 gpt-4.1-mini 0.00 1 / 7 100% ๐Ÿšซ not adopted (owner decision)
v4 gpt-6-luna โ‰ˆ ยฝ of real cases go straight to review โ€” 7 / 7 per class โœ… released โ€” stored run still TODO

Fault injection plants known errors into clean drafts and measures recall per layer (deterministic, LLM, combined): numeric_gross ยท numeric_subtle ยท fabricated_with_ref ยท fabricated_no_ref ยท deleted_mandatory ยท dose_insertion ยท laterality_flip ยท suppressed_disagreement

Important

All evaluation so far uses synthetic cases. It says nothing about real patients. Vision accuracy has not been measured on held-out data โ€” the interim weights were trained on REFUGE, so REFUGE images can test the pipeline but cannot evaluate the models.


๐Ÿ”’ Privacy & security

flowchart LR
    N["๐Ÿ“ raw note"]:::raw -->|input rail| R["๐Ÿ•ถ๏ธ redacted note"]:::safe
    R -->|AES-256-GCM| DB[("๐Ÿ˜ Postgres")]:::store
    R -->|facts + redacted text| O["๐Ÿค– OpenAI"]:::ext
    I["๐Ÿ“ท image"]:::raw -->|bytes + bearer| G["๐ŸŽฎ GPU server<br/>does not store"]:::ext
    I -->|sha256 name| S[("๐Ÿ—‚๏ธ object store")]:::store
    I -.->|โŒ never| O
    F["๐Ÿ“› filename"]:::raw -.->|โŒ never read| DB
    classDef raw fill:#5c1f1f,stroke:#ff6b6b,color:#fff;
    classDef safe fill:#14532d,stroke:#4ade80,color:#fff;
    classDef store fill:#13315c,stroke:#64b5f6,color:#fff;
    classDef ext fill:#3d2c5c,stroke:#e0aaff,color:#fff;
Loading
  • Google sign-in โ†’ ID token verified (issuer, audience, expiry, verified email) โ†’ roles only from the GOOGLE_CLINICIANS / GOOGLE_SUBMITTERS allowlists โ†’ short-lived CONSILIUM JWT (HS256, 480 min). OIDC-ready (RS256/ES256).
  • Session token held in memory, not localStorage.
  • Every /v1 route except the two sign-in endpoints returns 401 anonymously โ€” enforced by a route-matrix test.
  • audit_log, verdicts, reviews are append-only (UPDATE/DELETE revoked for the app role).
  • Structured JSON logs with PHI-key redaction; spans and Langfuse carry metadata only.
  • RFC 7807 errors never echo input.
  • STRIDE threat model in doc/threat_model.md; CI runs pip-audit, npm audit and gitleaks over full history.
  • Right-to-erasure and retention sweep: scripts/retention.py.

๐Ÿš€ Quick start

Prerequisites

1 ยท Install

git clone https://github.com/HareeshDaxton/CONSILIUM.git
cd CONSILIUM
uv sync --all-groups
cd frontend && npm ci && cd ..

2 ยท Configure

cp .env.example .env
# Generate a field-encryption key:
python -c "import base64,os; print(base64.urlsafe_b64encode(os.urandom(32)).decode())"

Fill in OPENAI_API_KEY, JWT_SECRET (โ‰ฅ 32 chars), FIELD_ENCRYPTION_KEY, the Google settings, and the vision server's VISION_ENDPOINT_URL + VISION_ENDPOINT_TOKEN.

3a ยท ๐ŸŽญ Offline demo (no GPU, no OpenAI)

uv run python scripts/dev_ui_stack.py --reset

Canned AI responses, fake vision, seeded cases โ€” the real API, worker and Postgres.

3b ยท ๐Ÿ”ด Real stack

# Terminal 1 โ€” database + API + worker (one process)
docker compose up -d postgres
uv run python scripts/run_app.py
#   โœ” vision status : OK on cuda
#   โœ” Google sign-in: ON

# Terminal 2 โ€” frontend
cd frontend && npm run dev

Open http://localhost:5173 โ†’ sign in with Google โ†’ New screening โ†’ upload a photo โ†’ choose the eye โ†’ add a note (made-up data only) โ†’ Generate report. A report takes about 20โ€“60 s.

Tip

You cannot sign a case you submitted. To try the full review flow alone, sign in with a second Google account listed in GOOGLE_CLINICIANS.

๐ŸŽฎ Running the GPU vision server (Deva/)

On the machine with the NVIDIA GPU:

cd Deva
./run.sh        # Linux
run.bat         # Windows

First run creates a venv, installs CUDA PyTorch, downloads both pinned models, opens an ngrok HTTPS tunnel and prints two lines (URL + token) to paste into the main .env. The server returns raw outputs only and never saves images. A free ngrok URL can change on restart โ€” update .env when it does.

๐Ÿฉน Troubleshooting
Symptom Fix
vision status NOT REACHABLE / 404 or cases fail VISION_UNAVAILABLE GPU server is off or its ngrok URL changed โ†’ update URL + token in .env, restart
DLL load failed โ€ฆ Application Control (Windows) Windows App Control blocks SQLAlchemy .pyd files โ†’ allow .venv in Windows Security
Docker API pipe not found Start Docker Desktop
Case FAILED with BLURRED / UNDEREXPOSED Quality gate working as intended โ†’ use a sharper / brighter photo
Report marked UNVERIFIED A real issue survived 3 rounds; it's highlighted for the clinician to fix and sign
SELF_APPROVAL Use a second clinician account
Report says "eye not specified" Choose the eye at upload
Integration tests skipped docker compose up -d postgres
make: command not found Use the uv run โ€ฆ commands directly

โš™๏ธ Configuration

Secrets and machine settings live in .env (never committed). Clinical and safety numbers live in versioned YAML so every change is reviewable โ€” values a clinician must approve are tagged # CLINICAL-REVIEW:.

File Holds
configs/routing.yaml Role allowlists for the Context Router
configs/thresholds.yaml Screening tiers, image pre-check, quality gate, numeric tolerances, stray-number whitelist, disagreement topics
configs/models.yaml Endpoint pin, model IDs + commits, calibration temperature
configs/eval.yaml Bootstrap settings, paired comparisons, price table
configs/guardrails/ NeMo config, Colang rails, input_rail.yaml, output_rail.yaml, phi_patterns.yaml
Key .env variables
# LLMs
OPENAI_API_KEY=...
LLM_MODEL_SPECIALIST=gpt-4o-mini-2024-07-18
LLM_MODEL_DIRECTOR=gpt-4o-mini-2024-07-18
LLM_MODEL_CRITIC=gpt-6-luna
LLM_CASE_TOKEN_BUDGET=100000
CRITIC_MAX_ROUNDS=3

# Storage
DATABASE_URL=postgresql+psycopg://...
OBJECT_STORE_URI=file://./data/objects
FIELD_ENCRYPTION_KEY=...

# Vision
VISION_CLIENT_MODE=http            # http | fake
VISION_ENDPOINT_URL=https://<ngrok>.ngrok-free.app
VISION_ENDPOINT_TOKEN=...
VISION_TIMEOUT_S=120

# Auth
JWT_SECRET=...                     # >= 32 chars
GOOGLE_CLIENT_ID=...
GOOGLE_CLINICIANS=doc@example.com
GOOGLE_SUBMITTERS=clinic@example.com

# Safety switches
RAILS_ENGINE=passthrough           # nemo | passthrough
INTAKE_FROZEN=false                # true โ†’ 503 on new submissions (rollback switch)

๐Ÿ”Œ API

All routes are under /v1 and return RFC 7807 problem+json errors.

Method Endpoint Who Purpose
GET /healthz ยท /readyz ยท /metrics public liveness, readiness, Prometheus
GET /v1/auth/config public Google sign-in on/off
POST /v1/auth/google public Google ID token โ†’ session token
GET /v1/me any caller identity + roles
POST /v1/cases submitter submit case (202, Idempotency-Key)
GET /v1/cases any list / queue filters
GET /v1/cases/{id} any status, CAD summary, progress
GET /v1/cases/{id}/draft clinician draft + open issues + evidence map
GET /v1/cases/{id}/overlay any disc/cup overlay PNG
GET /v1/cases/{id}/timeline any audit events
GET /v1/cases/{id}/facts any fact sheet
GET /v1/cases/{id}/report-document any structured report
GET /v1/cases/{id}/export?format=pdf|docx any download
POST /v1/cases/{id}/review clinician approve / edit / reject
GET /v1/cases/{id}/report any approved report only
GET /v1/meta/reason-codes any closed reason-code list
GET /v1/meta/status any vision reachability, LLM configured
๐Ÿ“ Case lifecycle
flowchart LR
    subgraph P["โณ In progress"]
        direction LR
        RECEIVED --> IMAGE_PRECHECKED --> INPUT_RAIL_PASSED --> NOTE_EXTRACTED --> CAD_COMPLETE --> CONTEXT_BUILT --> ROUTED --> SPECIALISTS_DONE --> DRAFTED --> AUDITING
        AUDITING <--> REVISING
        AUDITING --> OUTPUT_RAIL_PASSED
    end
    OUTPUT_RAIL_PASSED --> PENDING_REVIEW:::wait
    AUDITING --> NEEDS_ATTENTION:::warn
    PENDING_REVIEW --> APPROVED:::ok
    PENDING_REVIEW --> APPROVED_WITH_EDITS:::ok
    PENDING_REVIEW --> REJECTED:::bad
    NEEDS_ATTENTION --> APPROVED_WITH_EDITS
    IMAGE_PRECHECKED -.-> REJECTED_INPUT:::bad
    CAD_COMPLETE -.-> FAILED:::bad
    classDef wait fill:#3d2c5c,stroke:#e0aaff,color:#fff;
    classDef warn fill:#5c3d1f,stroke:#fbbf24,color:#fff;
    classDef ok fill:#14532d,stroke:#4ade80,color:#fff;
    classDef bad fill:#4a1d1d,stroke:#f87171,color:#fff;
Loading

Any node can fail closed to FAILED or NEEDS_ATTENTION with a reason code; dotted edges show the most common ones.


๐Ÿ—‚ Project structure

CONSILIUM/
โ”œโ”€โ”€ ๐Ÿ“œ AGENTS.md                 โ† the contract: invariants, design, rules (highest authority)
โ”œโ”€โ”€ ๐Ÿ“˜ README.md
โ”œโ”€โ”€ โš™๏ธ configs/                  โ† routing, thresholds, models, eval, guardrails
โ”œโ”€โ”€ ๐Ÿ’ฌ prompts/                  โ† versioned prompt files (vN.md, old versions kept)
โ”‚   โ”œโ”€โ”€ note_extractor/  ophthalmologist/  optometrist/  pharmacist/
โ”‚   โ”œโ”€โ”€ director/ (v1, v2โœ”)      critic/ (v1โ€“v4โœ”)
โ”‚   โ””โ”€โ”€ single_agent/  judge/    โ† evaluation only
โ”œโ”€โ”€ ๐Ÿ src/consilium/
โ”‚   โ”œโ”€โ”€ core/          settings, errors, JSON logging with PHI redaction, IDs, clock
โ”‚   โ”œโ”€โ”€ schemas/       ALL cross-component Pydantic models (imports nothing)
โ”‚   โ”œโ”€โ”€ vision/        client protocol, HTTP/fake clients, RLE, postprocess, CDR, quality, calibration, overlay
โ”‚   โ”œโ”€โ”€ intake/        de-identification, LLM note extraction
โ”‚   โ”œโ”€โ”€ context/       fact IDs, fact-sheet builder, allowlist router
โ”‚   โ”œโ”€โ”€ llm/           OpenAI client, retry/repair, token budget, prompt loader
โ”‚   โ”œโ”€โ”€ agents/        3 specialists, Director, Critic, prompt templates
โ”‚   โ”œโ”€โ”€ verify/        deterministic verifier, lexicon, critic facade, verdict merge
โ”‚   โ”œโ”€โ”€ guardrails/    input/output checks, normalisation, NeMo wiring
โ”‚   โ”œโ”€โ”€ graph/         LangGraph state, nodes, edges, runner, DI container
โ”‚   โ”œโ”€โ”€ persistence/   11 tables, encryption, object store, erasure, Alembic
โ”‚   โ”œโ”€โ”€ jobs/          Postgres job queue, worker, retention
โ”‚   โ”œโ”€โ”€ review/        review service, diff, reason codes, feedback store, analytics
โ”‚   โ”œโ”€โ”€ reporting/     report document, PDF + Word rendering
โ”‚   โ”œโ”€โ”€ api/           FastAPI app, auth, Google auth, routes
โ”‚   โ”œโ”€โ”€ observability/ Prometheus, OTel, Langfuse sink
โ”‚   โ””โ”€โ”€ eval/          ablation harness, metrics, diversity, fault injection, judge, report
โ”œโ”€โ”€ โš›๏ธ frontend/                 โ† React 18 + Vite + TS review UI and 3D landing page
โ”œโ”€โ”€ ๐ŸŽฎ Deva/                     โ† standalone GPU vision server (never imported by src/)
โ”œโ”€โ”€ ๐Ÿ‹๏ธ training/                 โ† vision training code (never imported by src/)
โ”œโ”€โ”€ ๐Ÿšข deploy/                   โ† HF endpoint handler, Prometheus/Grafana provisioning
โ”œโ”€โ”€ ๐Ÿ› ๏ธ scripts/                  โ† run_app, dev_ui_stack, dev_token, fake vision, load test, retention
โ”œโ”€โ”€ ๐Ÿงช tests/                    โ† unit ยท contract ยท integration ยท adversarial ยท e2e ยท live
โ””โ”€โ”€ ๐Ÿ“š doc/                      โ† ARCHITECTURE, SKILLS, threat model, DR, load test, model card, eval runs

Import boundaries (importlinter.ini, 8 contracts): schemas imports nothing ยท vision never imports llm/agents/graph ยท agents never import vision ยท only llm/ touches openai ยท only persistence/ touches sqlalchemy ยท only vision/client_http touches httpx ยท only graph/ imports langgraph ยท training/ and deploy/ are never imported.


๐Ÿงช Testing & quality gates

964

Python tests passing

~92%

coverage

23

schema snapshots

โ‰ฅ 1000

router-leak examples

83 + 8

vitest + Playwright
Suite Covers
unit/ every module, incl. Hypothesis router-leak and verifier-boundary properties
contract/ JSON-Schema snapshots of all 23 models; frontend fixtures validated against real API schemas
integration/ graph, failures, crash-resume, API, persistence, review, reports, retention (real Postgres)
adversarial/ prompt-injection notes, input rail, output rail with obfuscations
e2e/ vertical slice: submit โ†’ worker โ†’ review โ†’ approved report
live/ real OpenAI calls, manual only (-m live)

Rule: the default suite never calls a real LLM or the network โ€” fake LLM and fake vision clients answer deterministically.

uv run ruff check . && uv run ruff format --check .
uv run mypy --strict src/
uv run lint-imports
uv run pytest -m "not live"
cd frontend && npm run typecheck && npm test && npm run e2e

CI (.github/workflows/ci.yml): gates (ruff, mypy, import-linter, pytest, schema check) ยท security (pip-audit, npm audit, gitleaks) ยท frontend (typecheck, vitest, build). CI has no Postgres service, so DB-backed tests run locally before pushing.

Load test: 50 cases, concurrency 10, emulated latencies โ†’ 0 failures, p95 6523 ms (target 8250 ms). DR rehearsal: pg_dump + object-store restore recovered all 11 tables with identical row counts.


๐Ÿ“ˆ Observability

docker compose --profile observability up -d --no-deps prometheus grafana
# Grafana โ†’ http://localhost:3001   ยท   Prometheus โ†’ http://localhost:9090
  • Metrics: node latency, cases by status, LLM calls/tokens/latency, vision latency / cold starts / revision mismatches, critic issues by type and layer, rounds to approval, review actions.
  • Tracing: OTel spans per request, node, LLM call and vision call (OTLP/HTTP).
  • Langfuse: metadata-only sink.
  • Alerts (7): VisionFailureSpike ยท VisionRevisionMismatch ยท VisionColdStartSpike ยท CriticRoundOneApprovalNear100 ยท NeedsAttentionDrift ยท ReviewBacklog ยท LLMErrorSpike

Only numbers and IDs are recorded โ€” never note text or images.


๐Ÿ“ Differences from the research paper

Topic Paper CONSILIUM V1 Why
CDR sqrt(cup / (cup + disc)) vertical CDR (primary) + sqrt(cup_area / disc_area) nested masks would double-count the cup
Convergence metric distance cosine similarity (higher = more convergent), fixed eval-only lexical embedder unambiguous direction
Verification LLM critic deterministic verifier first, then critic over CAD and note facts cheap exact checks catch most errors
Feedback memory pgvector relational storage of clinician corrections no embeddings in V1 serving
Vision serving โ€” remote behind VisionClient; interpretation in-repo swappable, testable
Reporting โ€” every figure from a stored run; null and negative results reported invariant I-11

๐Ÿ—บ Roadmap & open items

๐Ÿ”ง Engineering

  • Stored eval-live run with the final config (critic v4 + Director v2)
  • Detect laterality flips when laterality is UNKNOWN
  • Re-measure minority-opinion preservation with Director v2
  • Fit calibration on labelled non-REFUGE images (T is 1.0 today)
  • Measure vision accuracy on held-out data
  • Install the NeMo model rail (passthrough today)
  • Self-host Langfuse; deploy an OTel collector
  • Validate routing.yaml entries against facts.py
  • Replace ngrok with a private vision endpoint before any real patient data

๐Ÿฉบ Clinical & governance

  • Clinician sign-off on routing.yaml
  • Clinician review of specialist prompts
  • Approve the extraction-confidence rule
  • Tier cut-offs and the 11 CLINICAL-REVIEW values
  • Clinician rubric scoring (accuracy, completeness, safety, usefulness)
  • Retention period + privileged retention role
  • Record provider data terms (OpenAI, Hugging Face, ngrok)
  • Intended-use statement + medical-device determination

๐Ÿ”ฎ V2 direction

Clinician edits are stored with reason codes behind a FeedbackStore seam. review/analytics.py computes reason-code shares that will gate the decision on whether retrieval (RAG) is worth adding in V2 โ€” the move happens only if the data says the answers need it.

๐Ÿ•ฐ๏ธ Build history โ€” 14 phases
Phase Delivered
P0 Scaffold: uv, CI gates, Makefile, Docker, settings, logging
P1 Schemas, fact IDs, JSON-Schema snapshots
P2 Vision: REFUGE tooling, training code, calibration, tiers, quality, CDR, VisionClient
P3 Note intake: de-identification, LLM client, extraction, adversarial notes
P4 Context builder, router, leak tests
P5 LLM hardening (retry / repair / budget), prompt loader, 3 specialists
P6 Director draft + revise, disagreement preservation
P7 Deterministic verifier, lexicon, tolerances, seeded recall
P8 LLM critic, merge, bounded loop, UNVERIFIED contract, fault injection
P9 Persistence, graph, persistence-per-node, crash-resume
P10 Guardrails: input/output checks, NeMo wiring
P11 FastAPI, auth, job queue, review service, e2e
P12 React review UI, 3D landing, Google sign-in, PDF/Word export, GPU server over ngrok
P13 Evaluation harness, ablation, critic evaluation, first live run
P14 Observability, alerts, threat model, security scans, erasure, load test, DR, release checklist, model card
14.5โ€“14.7 UI redesign, Indian-format PHI patterns, critic v4 on gpt-6-luna + Director v2

๐Ÿ“š Documentation map

Document What's inside
AGENTS.md The contract โ€” invariants, design, stack, rules for coding agents
doc/ARCHITECTURE.md Modules, schemas, API, state machine, failure matrix, algorithms
doc/SKILLS.md 22 step-by-step playbooks (schema change, routing change, prompt change, PHI review, โ€ฆ)
doc/release_checklist.md V1 release items, accepted exceptions, rollback plan, CLINICAL-REVIEW register
doc/threat_model.md Assets, trust boundaries, STRIDE, scan results
doc/MODEL_CARD.md Vision model card โ€” intended use and limits
doc/dr.md ยท doc/load_test.md Backup/restore rehearsal ยท load test method and results
doc/eval_runs/ Committed evaluation run + generated report
training/data/DATASETS.md Dataset registry and allowed uses
Deva/README.md GPU vision server setup
Kimi.md Section-by-section progress log with verification evidence

๐Ÿ‘ฅ Team

HareeshDaxton
Hareesh Kumar K
@HareeshDaxton
Owner ยท engineering lead ยท research co-author
Deva
Deva
Vision inference host

๐Ÿ™ Acknowledgements

  • Vision models by pamixsun on Hugging Face (Apache-2.0), fine-tuned on the REFUGE challenge dataset.
  • The team's earlier MedChat work, which served as the baseline this system was rebuilt from.

๐Ÿ“„ License

License to be decided. Third-party models keep their own licenses; dataset terms are tracked in training/data/DATASETS.md.


โš•๏ธ Screening support only; not a diagnosis.

Built with discipline: fail closed ยท cite everything ยท a human signs.

footer

About

๐Ÿ‘๏ธ Multi-agent AI that drafts glaucoma screening reports from eye photos. A doctor always reviews and signs. Screening support only.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages