Caution
CONSILIUM is screening support, not a diagnostic device. Every report carries the line "Screening support only; not a diagnosis." Nothing becomes final until a human clinician reviews and signs it. V1 is cleared for a supervised pilot on synthetic or consented research data only โ not for real patients.
Click to collapse
- Why CONSILIUM exists
- What it does
- Architecture
- The pipeline, stage by stage
- Who sees what โ the Context Router
- The verification loop
- The 14 golden rules
- Tech stack
- Evaluation โ honest numbers
- Privacy & security
- Quick start
- Configuration
- API
- Project structure
- Testing & quality gates
- Observability
- Differences from the research paper
- Roadmap & open items
- Documentation map
- Team
Glaucoma damages the optic nerve slowly and silently; by the time a patient notices, vision loss is permanent. Screening a fundus photograph for an enlarged optic cup relative to the disc is one of the cheapest early signals โ but turning a photo plus a doctor's note into a careful, multi-perspective report takes specialist time.
Multi-agent LLM systems look like an obvious fit. They have a known failure mode, though:
|
When several agents read the same context, they converge. They agree with each other, repeat each other's mistakes, and a minority opinion that might have mattered quietly vanishes. |
|
๐ท fundus photo + ๐ doctor's note (optional) + ๐๏ธ which eye
โ
โผ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ measure โ route โ specialise โ draft โ
โ โ verify โ critique โ revise โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ
โผ
๐ draft report โ ๐ฉบ clinician review & sign โ โฌ๏ธ PDF / Word
| ๐ฌ Measures SwinV2 glaucoma probability + SegFormer disc/cup masks โ vertical CDR, quality gate, risk tier |
๐งโโ๏ธ Specialises Ophthalmologist, Optometrist and Pharmacist agents reason in parallel over isolated views |
๐ก๏ธ Verifies 7+ deterministic checks, then an independent LLM critic, with a bounded 3-round revise loop |
โ๏ธ Defers Clinician approves, edits (with reason codes) or rejects. Approved reports are immutable |
flowchart LR
subgraph LAPTOP["๐ป Application host (CPU)"]
direction TB
UI["๐ React + Vite UI<br/>:5173"]
API["โก FastAPI API<br/>:8000"]
WK["โ๏ธ Worker<br/>LangGraph runner"]
PG[("๐ PostgreSQL 16<br/>11 tables")]
OS[("๐๏ธ Object store<br/>content-addressed")]
UI -- "/v1 proxy" --> API
API -- "job queue<br/>SKIP LOCKED" --> WK
API --> PG
WK --> PG
API --> OS
WK --> OS
end
subgraph GPU["๐ฎ Vision server (NVIDIA GPU)"]
VS["Deva/server.py<br/>SwinV2-tiny + SegFormer<br/>pinned commits"]
end
subgraph OAI["๐ค OpenAI"]
M1["gpt-4o-mini<br/>extractor ยท specialists ยท Director"]
M2["gpt-6-luna<br/>Critic"]
end
WK -- "HTTPS ยท image bytes + bearer<br/>POST /predict" --> VS
VS -- "p_raw + 2 RLE masks + revision" --> WK
WK -- "facts + REDACTED note only<br/>never the image" --> OAI
classDef host fill:#0f2027,stroke:#00c9a7,color:#ffffff,stroke-width:2px;
classDef gpu fill:#2c1a4d,stroke:#b388ff,color:#ffffff,stroke-width:2px;
classDef llm fill:#3a1f1f,stroke:#ff8a65,color:#ffffff,stroke-width:2px;
classDef store fill:#13315c,stroke:#64b5f6,color:#ffffff;
class UI,API,WK host;
class VS gpu;
class M1,M2 llm;
class PG,OS store;
flowchart TD
A([๐ค POST /v1/cases]):::intake --> B[precheck_image]:::intake
B --> C[input_rail<br/>injection ยท length ยท PHI redaction]:::intake
C -.->|queued โ worker| D[extract_note<br/>gpt-4o-mini ยท T=0]:::ai
D --> E[call_vision<br/>SwinV2 + SegFormer]:::vision
E --> F[build_context<br/>named fact sheet]:::det
F --> G[route_context<br/>routing.yaml allowlist]:::det
G --> H1[๐๏ธ Ophthalmologist]:::ai
G --> H2[๐ญ Optometrist]:::ai
G --> H3[๐ Pharmacist]:::ai
H1 & H2 & H3 --> I[draft<br/>Director]:::ai
I --> J{audit<br/>verifier โ critic}:::check
J -- "โ
approved" --> K[output_rail<br/>disclaimer ยท no dose ยท no PHI]:::check
J -- "โ issues ยท rounds left" --> R[revise<br/>Director gets issue list only]:::ai
R --> J
J -- "โ still failing at round 3" --> U[[โ ๏ธ NEEDS_ATTENTION<br/>draft marked UNVERIFIED]]:::warn
K --> P[[๐ PENDING_REVIEW]]:::human
U --> V{๐ฉบ Clinician review}:::human
P --> V
V -- approve --> Z1([โ
APPROVED]):::done
V -- "edit + reason code" --> Z2([โ๏ธ APPROVED_WITH_EDITS]):::done
V -- "reject + reason code" --> Z3([โ REJECTED]):::stop
classDef intake fill:#1e3a5f,stroke:#64b5f6,color:#fff;
classDef vision fill:#2c1a4d,stroke:#b388ff,color:#fff;
classDef det fill:#0f3d3e,stroke:#00c9a7,color:#fff;
classDef ai fill:#4a2511,stroke:#ffb74d,color:#fff;
classDef check fill:#1b4332,stroke:#52b788,color:#fff;
classDef warn fill:#5c1f1f,stroke:#ff6b6b,color:#fff;
classDef human fill:#3d2c5c,stroke:#e0aaff,color:#fff;
classDef done fill:#14532d,stroke:#4ade80,color:#fff;
classDef stop fill:#4a1d1d,stroke:#f87171,color:#fff;
Legend: ๐ฆ intake ยท ๐ช vision ยท ๐ฉ deterministic code ยท ๐ง LLM ยท ๐ข checks ยท ๐ฅ attention ยท ๐ฃ human
Crash-safe by construction. After every node, one database transaction writes the status, a state snapshot, last_node, any failure, an audit event, the domain rows and agent telemetry. A crashed case resumes at the successor of its last persisted node โ vision and extraction are never recomputed.
1 ยท ๐ค Upload & image pre-check
- Accepts PNG / JPEG, โค 20 MB, short side โฅ 256 px.
- The original filename is never read or stored (filenames often contain patient names). The browser renames uploads to
fundus.<ext>. - Images are stored content-addressed by SHA-256.
Idempotency-Keyheader: re-submitting returns the same case.- A doctor's report can be dropped in as PDF / Word / photo โ text is extracted inside the browser (
pdfjs-dist,mammoth,tesseract.js); only the reviewed text is sent. - Rejections:
IMAGE_EMPTYยทIMAGE_TOO_LARGEยทUNSUPPORTED_IMAGE_FORMATยทIMAGE_UNDECODABLEยทLOW_RESOLUTION
2 ยท ๐ Note intake: safety rail โ de-identification โ extraction
The doctor's note is untrusted input (it could carry a prompt injection).
| Step | What happens |
|---|---|
| ๐งฑ Input rail | Rejects notes > 4000 chars, odd control characters, bad charsets, and instruction-like text (multilingual, homoglyph- and zero-width-normalised). |
| ๐ถ๏ธ De-identification | Regex policy phi_patterns.yaml v2: Aadhaar, Indian mobile, phone, email, SSN, MRN / UHID / OP-IP No / Reg. No, date of birth, titled names โ [REDACTED:<rule>]. Only the rule name is stored, never the removed value. |
| ๐งพ Extraction | gpt-4o-mini at T=0, structured output bound to ClinicalNote (age, IOP OD/OS, meds, intolerances, comorbidities, prior Dx, exam findings, free text). โค 2 repair retries, then LLM_OUTPUT_INVALID. |
| ๐ Confidence | 8 fields counted: โฅ 4 HIGH, 1โ3 MODERATE, 0 LOW. LOW adds a caveat fact the report must mention. |
| ๐ Storage | Redacted note stored AES-256-GCM encrypted. The raw note is never persisted and never sent to OpenAI. |
3 ยท ๐ฌ Vision: remote inference, local interpretation
Models (Apache-2.0, fine-tuned on REFUGE), pinned to exact commits:
| Model | Task | Pin |
|---|---|---|
pamixsun/swinv2_tiny_for_glaucoma_classification |
glaucoma probability | a25a03d9ff23โฆ |
pamixsun/segformer_for_optic_disc_cup_segmentation |
disc / cup masks | a0463fb6c2bfโฆ |
The GPU server returns only p_raw, two RLE masks, mask size and endpoint_revision. All interpretation happens in this repo:
RLE decode โ largest component โ fill holes โ clip cup to disc
โ measurements โ CDR โ mask QC โ quality gate โ calibration โ tier
- Vertical CDR = cup vertical extent รท disc vertical extent (primary);
sqrt(cup_area / disc_area)(secondary). - Quality gate (inside the field of view):
UNDEREXPOSED< 0.12 ยทOVEREXPOSED> 0.90 ยทBLURREDLaplacian variance < 1e-4 ยทNOT_A_FUNDUS_IMAGEmean(R) โ mean(B) < 0.05 ยทOPTIC_DISC_NOT_VISIBLEdisc fraction outside 0.005โ0.30. - Tiers on calibrated probability: โฅ 0.70 HIGH ยท โฅ 0.30 INTERMEDIATE ยท else LOW (placeholders, flagged
CLINICAL-REVIEW). - Revision pin: any mismatch with
configs/models.yamlโVISION_REVISION_MISMATCH. - Transient errors retried with exponential backoff + jitter, bounded at 120 s โ
VISION_UNAVAILABLE. Fail closed โ never a fake report.
4 ยท ๐๏ธ Fact sheet: agents can only cite, never invent
Every piece of case information becomes a named fact with a stable ID from a single source (context/facts.py):
cad.p_glaucoma.tier cad.cdr.vertical cad.quality.flags note.iop.od
cad.p_glaucoma.calibrated cad.cdr.area_based cad.maskqc.ok note.meds
cad.disc.vdiam_px cad.laterality cad.caveat.<flag> note.caveat.extraction
Free-text facts are wrapped in [untrusted-data source="clinician-note"] blocks inside prompts. Warnings become facts too, so the report is forced to carry them.
5 ยท ๐งโโ๏ธ Three specialists, in parallel, in isolation
| Agent | Focus | Hard limits |
|---|---|---|
| ๐๏ธ Ophthalmologist | Disc measurements, risk tier, consistency with the note, escalation | Never sees the medication list |
| ๐ญ Optometrist | Next steps: visual field, OCT, gonioscopy, repeat photo, referral urgency, image quality | Never sees area-based CDR or raw disc/cup sizes |
| ๐ Pharmacist | Medication safety by drug class only (e.g. ฮฒ-blockers + asthma) | No doses, no brands; always "subject to prescriber judgement"; never sees eye measurements |
Each returns a SubReport: findings with evidence_refs + confidence, recommendations with triggered_by, uncertainties, and what it declined as out of scope. asyncio.gather; never see each other's output.
6 ยท ๐ฌ The Director
Writes one DirectorDraft: impression, numbered claims (each citing fact IDs and the roles that raised it), disagreements kept visible ("left to clinician"), recommendations, limitations. NUMERIC claims must carry numeric_value + numeric_ref.
Prompt v2 rules: read the note before writing "no VF/OCT data", always state the eye ("eye not specified" when unknown), don't invent disagreements from rounding, always include the disclaimer.
7 ยท ๐ฉบ Clinician review & export
- Review screen: fundus photo with toggleable disc (cyan) / cup (magenta) overlay and opacity slider, claims with evidence chips, red highlights on flagged issues, field-level edit diff.
- Actions: Approve ยท Approve with edits ยท Reject โ edits and rejects require a reason code:
factual CAD errorยทclinical interpretationยทknowledge gapยทunsafe recommendationยทomissionยทtone/wordingยทscope violationยทother. - Self-approval refused server-side (separation of duties). Stale revisions refused.
- UNVERIFIED drafts can only be approved once every open issue is resolved and the edited draft passes the deterministic verifier.
- Approved reports are immutable; amendments create a new linked version.
- One server-built report, three renderings: screen ยท PDF (fpdf2) ยท Word (python-docx).
Isolation is the research claim, so it is enforced three times: by the router, by the specialist output validator, and by the verifier's role-scope check. A Hypothesis property test (โฅ 1000 examples) proves no forbidden fact ID, label or numeric value ever reaches a role's prompt.
| Fact | ๐๏ธ Ophthal. | ๐ญ Optom. | ๐ Pharm. |
|---|---|---|---|
cad.p_glaucoma.tier |
๐ข | ๐ข | ๐ข |
cad.p_glaucoma.calibrated, cad.cdr.vertical |
๐ข | ๐ข | โซ |
cad.cdr.area_based, disc/cup areas & diameters |
๐ข | โซ | โซ |
cad.quality.*, cad.caveat.*, cad.maskqc.ok |
๐ข | ๐ข | โซ |
cad.laterality, note.age |
๐ข | ๐ข | ๐ข |
note.iop.od/os, note.prior_dx |
๐ข | ๐ข | ๐ข |
note.exam_findings, note.free_text |
๐ข | ๐ข | โซ |
note.meds, note.intolerances, note.comorbidities |
โซ | โซ | ๐ข |
note.caveat.extraction |
๐ข | ๐ข | ๐ข |
Deny-by-default allowlist in configs/routing.yaml. routing_version = SHA-256 of the YAML, stamped on every routed context. Policy is pending clinician sign-off.
stateDiagram-v2
direction LR
[*] --> Verifier: draft (rev n)
Verifier --> Critic: no deterministic issues
Verifier --> Revise: issues found
Critic --> OutputRail: approved
Critic --> Revise: issues found
Revise --> Verifier: rev n+1 (rounds left)
Revise --> Unverified: round = 3
OutputRail --> PendingReview
Unverified --> PendingReview: highlighted for clinician
PendingReview --> [*]
|
|
Output rail before any clinician sees it: DRAFT_DEFINITIVE_DIAGNOSIS ยท DRAFT_DOSE_OR_REGIMEN ยท DRAFT_STAGE_SEVERITY ยท DRAFT_PHI_ECHO ยท DRAFT_MISSING_DISCLAIMER.
These invariants live in AGENTS.md ยง2, the project's highest authority. If a change seems to need breaking one, the change stops.
| # | Rule |
|---|---|
| I-1 | ๐ข AI never sees the photo and never invents or alters measurement numbers. |
| I-2 | ๐งญ Who-sees-what is fixed code + routing.yaml, never an LLM decision. |
| I-3 | ๐ Every factual statement cites facts its author was allowed to see. |
| I-4 | โ๏ธ Nothing is final without a clinician; only the review service writes APPROVED. |
| I-5 | ๐ช Fail closed: errors become FAILED / NEEDS_ATTENTION with a reason โ never a silently degraded report. |
| I-6 | ๐ถ๏ธ Minimum personal data: opaque IDs; no names/DOB/MRN in prompts, logs, traces or tests; no upload filenames. |
| I-7 | โฃ๏ธ The doctor's note is untrusted input. |
| I-8 | ๐ Every AI input and output is strictly schema-validated. |
| I-9 | ๐ท๏ธ Prompts are versioned files; a change requires an evaluation run; the version is recorded on every call. |
| I-10 | ๐ซ V1: no RAG, no serving embeddings, no pgvector, no LLM fine-tuning. |
| I-11 | ๐ No performance claim without a measured, stored number. |
| I-12 | ๐ฉบ Screening-support language only: no diagnosis, no staging unless the note says so, no doses. |
| I-13 | ๐งฑ Vision only behind VisionClient, never in the API process; all interpretation in-repo. |
| I-14 | ๐ผ๏ธ The photo never reaches the LLM provider, logs, traces or prompts. |
Model assignments
| Role | Model | Temp | Prompt | Notes |
|---|---|---|---|---|
| Note extractor | gpt-4o-mini-2024-07-18 |
0 | v1 | structured output โ ClinicalNote |
| Ophthalmologist / Optometrist / Pharmacist | gpt-4o-mini-2024-07-18 |
0.2 | v1 | โค 2000 output tokens each |
| Director | gpt-4o-mini-2024-07-18 |
0.2 | v2 | โค 4000 output tokens |
| Critic | gpt-6-luna |
1.0 | v4 | reasoning medium, verbosity low, โค 12000 tokens |
Per-case token budget: 100 000 (raises, never truncates). Cost: the gpt-4o-mini portion of a real report was โ 0.3 US cents (~12k in / ~2k out), plus 2โ3 critic calls.
The research claim is tested as a four-stage ablation on the same cases, the same recorded vision output and the same model snapshots, with โฅ 3 samples per case and case-level bootstrap 95% CIs.
flowchart LR
S1["<b>S1</b><br/>single agent<br/>full context"]:::s1 --> S2["<b>S2</b><br/>multi-agent<br/>shared context"]:::s2 --> S3["<b>S3</b><br/>multi-agent<br/>routed context"]:::s3 --> S4["<b>S4</b><br/>routed +<br/>verification<br/><i>= CONSILIUM</i>"]:::s4
classDef s1 fill:#4a1d1d,stroke:#f87171,color:#fff;
classDef s2 fill:#4a3511,stroke:#fbbf24,color:#fff;
classDef s3 fill:#1e3a5f,stroke:#60a5fa,color:#fff;
classDef s4 fill:#14532d,stroke:#4ade80,color:#fff;
Stored run eval-live-20261017T140231Z-d6ed738d โ 6 synthetic cases ร 3 samples, all gpt-4o-mini:
| Claim | Comparison | Result | Verdict |
|---|---|---|---|
| Routing reduces convergence | S3 โ S2, S_inter (cosine similarity) | โ0.050 [โ0.083, โ0.024] | โ supported |
| Verification reduces omissions | S4 โ S3, omission rate | โ0.256 | โ supported |
| Full system beats single agent on grounding | S4 โ S1, unsupported-claim rate | โ0.081 | โ supported |
| Routing increases unique-fact coverage | โ | not shown | โช not supported |
| Verification reduces unsupported claims vs S3 | โ | not shown | โช not supported |
๐ง How the critic was fixed (and why the negative result is published)
The first live run surfaced a real problem: the critic rejected everything.
| Critic | Model | S4 UNVERIFIED rate | Clean-draft false rejections | Fault recall (combined) | Status |
|---|---|---|---|---|---|
| v1 | gpt-4o-mini | 1.00 | 7 / 7 | โ | โ over-rejects |
| v2 | gpt-4o-mini | 0.83 | 4 / 7 | 100% | |
| v3 | gpt-4.1-mini | 0.00 | 1 / 7 | 100% | ๐ซ not adopted (owner decision) |
| v4 | gpt-6-luna | โ ยฝ of real cases go straight to review | โ | 7 / 7 per class | โ released โ stored run still TODO |
Fault injection plants known errors into clean drafts and measures recall per layer (deterministic, LLM, combined):
numeric_gross ยท numeric_subtle ยท fabricated_with_ref ยท fabricated_no_ref ยท deleted_mandatory ยท dose_insertion ยท laterality_flip ยท suppressed_disagreement
Important
All evaluation so far uses synthetic cases. It says nothing about real patients. Vision accuracy has not been measured on held-out data โ the interim weights were trained on REFUGE, so REFUGE images can test the pipeline but cannot evaluate the models.
flowchart LR
N["๐ raw note"]:::raw -->|input rail| R["๐ถ๏ธ redacted note"]:::safe
R -->|AES-256-GCM| DB[("๐ Postgres")]:::store
R -->|facts + redacted text| O["๐ค OpenAI"]:::ext
I["๐ท image"]:::raw -->|bytes + bearer| G["๐ฎ GPU server<br/>does not store"]:::ext
I -->|sha256 name| S[("๐๏ธ object store")]:::store
I -.->|โ never| O
F["๐ filename"]:::raw -.->|โ never read| DB
classDef raw fill:#5c1f1f,stroke:#ff6b6b,color:#fff;
classDef safe fill:#14532d,stroke:#4ade80,color:#fff;
classDef store fill:#13315c,stroke:#64b5f6,color:#fff;
classDef ext fill:#3d2c5c,stroke:#e0aaff,color:#fff;
- Google sign-in โ ID token verified (issuer, audience, expiry, verified email) โ roles only from the
GOOGLE_CLINICIANS/GOOGLE_SUBMITTERSallowlists โ short-lived CONSILIUM JWT (HS256, 480 min). OIDC-ready (RS256/ES256). - Session token held in memory, not
localStorage. - Every
/v1route except the two sign-in endpoints returns 401 anonymously โ enforced by a route-matrix test. audit_log,verdicts,reviewsare append-only (UPDATE/DELETE revoked for the app role).- Structured JSON logs with PHI-key redaction; spans and Langfuse carry metadata only.
- RFC 7807 errors never echo input.
- STRIDE threat model in
doc/threat_model.md; CI runspip-audit,npm auditandgitleaksover full history. - Right-to-erasure and retention sweep:
scripts/retention.py.
git clone https://github.com/HareeshDaxton/CONSILIUM.git
cd CONSILIUM
uv sync --all-groups
cd frontend && npm ci && cd ..cp .env.example .env
# Generate a field-encryption key:
python -c "import base64,os; print(base64.urlsafe_b64encode(os.urandom(32)).decode())"Fill in OPENAI_API_KEY, JWT_SECRET (โฅ 32 chars), FIELD_ENCRYPTION_KEY, the Google settings, and the vision server's VISION_ENDPOINT_URL + VISION_ENDPOINT_TOKEN.
uv run python scripts/dev_ui_stack.py --resetCanned AI responses, fake vision, seeded cases โ the real API, worker and Postgres.
# Terminal 1 โ database + API + worker (one process)
docker compose up -d postgres
uv run python scripts/run_app.py
# โ vision status : OK on cuda
# โ Google sign-in: ON
# Terminal 2 โ frontend
cd frontend && npm run devOpen http://localhost:5173 โ sign in with Google โ New screening โ upload a photo โ choose the eye โ add a note (made-up data only) โ Generate report. A report takes about 20โ60 s.
Tip
You cannot sign a case you submitted. To try the full review flow alone, sign in with a second Google account listed in GOOGLE_CLINICIANS.
๐ฎ Running the GPU vision server (Deva/)
On the machine with the NVIDIA GPU:
cd Deva
./run.sh # Linux
run.bat # WindowsFirst run creates a venv, installs CUDA PyTorch, downloads both pinned models, opens an ngrok HTTPS tunnel and prints two lines (URL + token) to paste into the main .env. The server returns raw outputs only and never saves images. A free ngrok URL can change on restart โ update .env when it does.
๐ฉน Troubleshooting
| Symptom | Fix |
|---|---|
vision status NOT REACHABLE / 404 or cases fail VISION_UNAVAILABLE |
GPU server is off or its ngrok URL changed โ update URL + token in .env, restart |
DLL load failed โฆ Application Control (Windows) |
Windows App Control blocks SQLAlchemy .pyd files โ allow .venv in Windows Security |
Docker API pipe not found |
Start Docker Desktop |
Case FAILED with BLURRED / UNDEREXPOSED |
Quality gate working as intended โ use a sharper / brighter photo |
| Report marked UNVERIFIED | A real issue survived 3 rounds; it's highlighted for the clinician to fix and sign |
SELF_APPROVAL |
Use a second clinician account |
| Report says "eye not specified" | Choose the eye at upload |
| Integration tests skipped | docker compose up -d postgres |
make: command not found |
Use the uv run โฆ commands directly |
Secrets and machine settings live in .env (never committed). Clinical and safety numbers live in versioned YAML so every change is reviewable โ values a clinician must approve are tagged # CLINICAL-REVIEW:.
| File | Holds |
|---|---|
configs/routing.yaml |
Role allowlists for the Context Router |
configs/thresholds.yaml |
Screening tiers, image pre-check, quality gate, numeric tolerances, stray-number whitelist, disagreement topics |
configs/models.yaml |
Endpoint pin, model IDs + commits, calibration temperature |
configs/eval.yaml |
Bootstrap settings, paired comparisons, price table |
configs/guardrails/ |
NeMo config, Colang rails, input_rail.yaml, output_rail.yaml, phi_patterns.yaml |
Key .env variables
# LLMs
OPENAI_API_KEY=...
LLM_MODEL_SPECIALIST=gpt-4o-mini-2024-07-18
LLM_MODEL_DIRECTOR=gpt-4o-mini-2024-07-18
LLM_MODEL_CRITIC=gpt-6-luna
LLM_CASE_TOKEN_BUDGET=100000
CRITIC_MAX_ROUNDS=3
# Storage
DATABASE_URL=postgresql+psycopg://...
OBJECT_STORE_URI=file://./data/objects
FIELD_ENCRYPTION_KEY=...
# Vision
VISION_CLIENT_MODE=http # http | fake
VISION_ENDPOINT_URL=https://<ngrok>.ngrok-free.app
VISION_ENDPOINT_TOKEN=...
VISION_TIMEOUT_S=120
# Auth
JWT_SECRET=... # >= 32 chars
GOOGLE_CLIENT_ID=...
GOOGLE_CLINICIANS=doc@example.com
GOOGLE_SUBMITTERS=clinic@example.com
# Safety switches
RAILS_ENGINE=passthrough # nemo | passthrough
INTAKE_FROZEN=false # true โ 503 on new submissions (rollback switch)All routes are under /v1 and return RFC 7807 problem+json errors.
๐ Case lifecycle
flowchart LR
subgraph P["โณ In progress"]
direction LR
RECEIVED --> IMAGE_PRECHECKED --> INPUT_RAIL_PASSED --> NOTE_EXTRACTED --> CAD_COMPLETE --> CONTEXT_BUILT --> ROUTED --> SPECIALISTS_DONE --> DRAFTED --> AUDITING
AUDITING <--> REVISING
AUDITING --> OUTPUT_RAIL_PASSED
end
OUTPUT_RAIL_PASSED --> PENDING_REVIEW:::wait
AUDITING --> NEEDS_ATTENTION:::warn
PENDING_REVIEW --> APPROVED:::ok
PENDING_REVIEW --> APPROVED_WITH_EDITS:::ok
PENDING_REVIEW --> REJECTED:::bad
NEEDS_ATTENTION --> APPROVED_WITH_EDITS
IMAGE_PRECHECKED -.-> REJECTED_INPUT:::bad
CAD_COMPLETE -.-> FAILED:::bad
classDef wait fill:#3d2c5c,stroke:#e0aaff,color:#fff;
classDef warn fill:#5c3d1f,stroke:#fbbf24,color:#fff;
classDef ok fill:#14532d,stroke:#4ade80,color:#fff;
classDef bad fill:#4a1d1d,stroke:#f87171,color:#fff;
Any node can fail closed to FAILED or NEEDS_ATTENTION with a reason code; dotted edges show the most common ones.
CONSILIUM/
โโโ ๐ AGENTS.md โ the contract: invariants, design, rules (highest authority)
โโโ ๐ README.md
โโโ โ๏ธ configs/ โ routing, thresholds, models, eval, guardrails
โโโ ๐ฌ prompts/ โ versioned prompt files (vN.md, old versions kept)
โ โโโ note_extractor/ ophthalmologist/ optometrist/ pharmacist/
โ โโโ director/ (v1, v2โ) critic/ (v1โv4โ)
โ โโโ single_agent/ judge/ โ evaluation only
โโโ ๐ src/consilium/
โ โโโ core/ settings, errors, JSON logging with PHI redaction, IDs, clock
โ โโโ schemas/ ALL cross-component Pydantic models (imports nothing)
โ โโโ vision/ client protocol, HTTP/fake clients, RLE, postprocess, CDR, quality, calibration, overlay
โ โโโ intake/ de-identification, LLM note extraction
โ โโโ context/ fact IDs, fact-sheet builder, allowlist router
โ โโโ llm/ OpenAI client, retry/repair, token budget, prompt loader
โ โโโ agents/ 3 specialists, Director, Critic, prompt templates
โ โโโ verify/ deterministic verifier, lexicon, critic facade, verdict merge
โ โโโ guardrails/ input/output checks, normalisation, NeMo wiring
โ โโโ graph/ LangGraph state, nodes, edges, runner, DI container
โ โโโ persistence/ 11 tables, encryption, object store, erasure, Alembic
โ โโโ jobs/ Postgres job queue, worker, retention
โ โโโ review/ review service, diff, reason codes, feedback store, analytics
โ โโโ reporting/ report document, PDF + Word rendering
โ โโโ api/ FastAPI app, auth, Google auth, routes
โ โโโ observability/ Prometheus, OTel, Langfuse sink
โ โโโ eval/ ablation harness, metrics, diversity, fault injection, judge, report
โโโ โ๏ธ frontend/ โ React 18 + Vite + TS review UI and 3D landing page
โโโ ๐ฎ Deva/ โ standalone GPU vision server (never imported by src/)
โโโ ๐๏ธ training/ โ vision training code (never imported by src/)
โโโ ๐ข deploy/ โ HF endpoint handler, Prometheus/Grafana provisioning
โโโ ๐ ๏ธ scripts/ โ run_app, dev_ui_stack, dev_token, fake vision, load test, retention
โโโ ๐งช tests/ โ unit ยท contract ยท integration ยท adversarial ยท e2e ยท live
โโโ ๐ doc/ โ ARCHITECTURE, SKILLS, threat model, DR, load test, model card, eval runs
Import boundaries (importlinter.ini, 8 contracts): schemas imports nothing ยท vision never imports llm/agents/graph ยท agents never import vision ยท only llm/ touches openai ยท only persistence/ touches sqlalchemy ยท only vision/client_http touches httpx ยท only graph/ imports langgraph ยท training/ and deploy/ are never imported.
| Python tests passing | coverage | schema snapshots | router-leak examples | vitest + Playwright |
| Suite | Covers |
|---|---|
unit/ |
every module, incl. Hypothesis router-leak and verifier-boundary properties |
contract/ |
JSON-Schema snapshots of all 23 models; frontend fixtures validated against real API schemas |
integration/ |
graph, failures, crash-resume, API, persistence, review, reports, retention (real Postgres) |
adversarial/ |
prompt-injection notes, input rail, output rail with obfuscations |
e2e/ |
vertical slice: submit โ worker โ review โ approved report |
live/ |
real OpenAI calls, manual only (-m live) |
Rule: the default suite never calls a real LLM or the network โ fake LLM and fake vision clients answer deterministically.
uv run ruff check . && uv run ruff format --check .
uv run mypy --strict src/
uv run lint-imports
uv run pytest -m "not live"
cd frontend && npm run typecheck && npm test && npm run e2eCI (.github/workflows/ci.yml): gates (ruff, mypy, import-linter, pytest, schema check) ยท security (pip-audit, npm audit, gitleaks) ยท frontend (typecheck, vitest, build). CI has no Postgres service, so DB-backed tests run locally before pushing.
Load test: 50 cases, concurrency 10, emulated latencies โ 0 failures, p95 6523 ms (target 8250 ms).
DR rehearsal: pg_dump + object-store restore recovered all 11 tables with identical row counts.
docker compose --profile observability up -d --no-deps prometheus grafana
# Grafana โ http://localhost:3001 ยท Prometheus โ http://localhost:9090- Metrics: node latency, cases by status, LLM calls/tokens/latency, vision latency / cold starts / revision mismatches, critic issues by type and layer, rounds to approval, review actions.
- Tracing: OTel spans per request, node, LLM call and vision call (OTLP/HTTP).
- Langfuse: metadata-only sink.
- Alerts (7):
VisionFailureSpikeยทVisionRevisionMismatchยทVisionColdStartSpikeยทCriticRoundOneApprovalNear100ยทNeedsAttentionDriftยทReviewBacklogยทLLMErrorSpike
Only numbers and IDs are recorded โ never note text or images.
| Topic | Paper | CONSILIUM V1 | Why |
|---|---|---|---|
| CDR | sqrt(cup / (cup + disc)) |
vertical CDR (primary) + sqrt(cup_area / disc_area) |
nested masks would double-count the cup |
| Convergence metric | distance | cosine similarity (higher = more convergent), fixed eval-only lexical embedder | unambiguous direction |
| Verification | LLM critic | deterministic verifier first, then critic over CAD and note facts | cheap exact checks catch most errors |
| Feedback memory | pgvector | relational storage of clinician corrections | no embeddings in V1 serving |
| Vision serving | โ | remote behind VisionClient; interpretation in-repo |
swappable, testable |
| Reporting | โ | every figure from a stored run; null and negative results reported | invariant I-11 |
|
|
Clinician edits are stored with reason codes behind a FeedbackStore seam. review/analytics.py computes reason-code shares that will gate the decision on whether retrieval (RAG) is worth adding in V2 โ the move happens only if the data says the answers need it.
๐ฐ๏ธ Build history โ 14 phases
| Phase | Delivered |
|---|---|
| P0 | Scaffold: uv, CI gates, Makefile, Docker, settings, logging |
| P1 | Schemas, fact IDs, JSON-Schema snapshots |
| P2 | Vision: REFUGE tooling, training code, calibration, tiers, quality, CDR, VisionClient |
| P3 | Note intake: de-identification, LLM client, extraction, adversarial notes |
| P4 | Context builder, router, leak tests |
| P5 | LLM hardening (retry / repair / budget), prompt loader, 3 specialists |
| P6 | Director draft + revise, disagreement preservation |
| P7 | Deterministic verifier, lexicon, tolerances, seeded recall |
| P8 | LLM critic, merge, bounded loop, UNVERIFIED contract, fault injection |
| P9 | Persistence, graph, persistence-per-node, crash-resume |
| P10 | Guardrails: input/output checks, NeMo wiring |
| P11 | FastAPI, auth, job queue, review service, e2e |
| P12 | React review UI, 3D landing, Google sign-in, PDF/Word export, GPU server over ngrok |
| P13 | Evaluation harness, ablation, critic evaluation, first live run |
| P14 | Observability, alerts, threat model, security scans, erasure, load test, DR, release checklist, model card |
| 14.5โ14.7 | UI redesign, Indian-format PHI patterns, critic v4 on gpt-6-luna + Director v2 |
| Document | What's inside |
|---|---|
AGENTS.md |
The contract โ invariants, design, stack, rules for coding agents |
doc/ARCHITECTURE.md |
Modules, schemas, API, state machine, failure matrix, algorithms |
doc/SKILLS.md |
22 step-by-step playbooks (schema change, routing change, prompt change, PHI review, โฆ) |
doc/release_checklist.md |
V1 release items, accepted exceptions, rollback plan, CLINICAL-REVIEW register |
doc/threat_model.md |
Assets, trust boundaries, STRIDE, scan results |
doc/MODEL_CARD.md |
Vision model card โ intended use and limits |
doc/dr.md ยท doc/load_test.md |
Backup/restore rehearsal ยท load test method and results |
doc/eval_runs/ |
Committed evaluation run + generated report |
training/data/DATASETS.md |
Dataset registry and allowed uses |
Deva/README.md |
GPU vision server setup |
Kimi.md |
Section-by-section progress log with verification evidence |
![]() Hareesh Kumar K @HareeshDaxtonOwner ยท engineering lead ยท research co-author |
Deva Vision inference host |
- Vision models by
pamixsunon Hugging Face (Apache-2.0), fine-tuned on the REFUGE challenge dataset. - The team's earlier MedChat work, which served as the baseline this system was rebuilt from.
License to be decided. Third-party models keep their own licenses; dataset terms are tracked in training/data/DATASETS.md.
