Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,11 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.

<small>[Compare with v1.0.0](https://github.com/WDGPH/ImmuKnow/compare/v1.0.0...v1.1.0)</small>

### Added

- Notice versioning (assignment manifest mode): a new optional mode that maps each client to a specific notice version and language via a JSON assignment manifest. When `--notice-assignments <file>` is provided and `config/notice_versions.yaml` exists, the pipeline dispatches each client to the correct template and language individually - enabling a single run to produce overdue, affirmative, and informational notices in mixed languages. When the catalog file is absent, the pipeline behaves identically to before.
- Example `config/notice_versions.yaml` with `overdue_standard_v1` and `affirmative_schedule_v1`.

### Changed

- Switch input schema from fuzzy matching / data normalization to enforced Frictionless schema validation. Add schema page to mkdocs.
Expand Down
49 changes: 43 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ The `pipeline/` package is organized by pipeline function, not by layer. Each st
| 8 | `bundle_pdfs.py` | PDF bundling & grouping (optional) |
| 9 | `cleanup.py` | Intermediate file cleanup |

**Supporting modules:** `orchestrator.py` (orchestrator), `config_loader.py`, `data_models.py`, `enums.py`, `utils.py`.
**Supporting modules:** `orchestrator.py` (orchestrator), `config_loader.py`, `data_models.py`, `enums.py`, `utils.py`, `notice_versioning.py`, `assignment_manifest.py`.

**Template modules** (in `templates/` package): `en_template.py`, `fr_template.py` (Typst template rendering). For module structure questions, see `docs/CODE_ANALYSIS_STANDARDS.md`.

Expand Down Expand Up @@ -115,13 +115,13 @@ The main pipeline orchestrator (`orchestrator.py`) automates the end-to-end work
Prepares the output directory, optionally removing existing contents while preserving logs.

2. **Preprocessing** (`preprocess.py`)
Cleans, validates, and structures input data into a normalized JSON artifact (`preprocessed_clients_<run_id>.json`). Optionally validates school/daycare names against the PHIX reference mapping (see [PHIX School Validation](./config/README.md#phix-school-validation)).
Cleans, validates, and structures input data into a normalized JSON artifact (`preprocessed_clients_<run_id>.json`). Optionally validates school/daycare names against the PHIX reference mapping (see [PHIX School Validation](./config/README.md#phix-school-validation)). In manifest mode, also reconciles the assignment manifest against the client list and runs a preflight gate before any PDF is generated.

3. **Generating QR Codes** (`generate_qr_codes.py`, optional)
Generates QR code PNG files from templated payloads. Skipped if `qr.enabled: false` in `parameters.yaml`.

4. **Generating Notices** (`generate_notices.py`)
Renders Typst templates (`.typ` files) for each client from the preprocessed artifact, with QR code references.
Renders Typst templates (`.typ` files) for each client from the preprocessed artifact, with QR code references. In manifest mode, dispatches each client to the template specified by their assigned notice version and language.

5. **Compiling Notices** (`compile_notices.py`)
Compiles Typst templates into individual PDF notices using the `typst` command-line tool.
Expand All @@ -143,18 +143,19 @@ The main pipeline orchestrator (`orchestrator.py`) automates the end-to-end work

**Usage Example:**
```bash
uv run viper <input_file> <language> [--output PATH]
uv run viper <input_file> [language] [--output PATH]
```

**Required Arguments:**
- `<input_file>`: Name of the input file (e.g., `students.xlsx`)
- `<language>`: Language code (`en` or `fr`)
- `[language]`: Language code (`en` or `fr`). Required in fixed mode; omit when using `--notice-assignments`.

**Optional Arguments:**
- `--input PATH`: Input directory (default: ../input)
- `--output PATH`: Output directory (default: ../output)
- `--config PATH`: Configuration directory (default: ../config)
- `--template NAME`: PHU template name within `phu_templates/` (e.g., `wdgph`); defaults to built-in `templates/` when omitted
- `--notice-assignments PATH`: JSON file mapping client IDs to notice versions (enables manifest mode; requires `config/notice_versions.yaml`)

**Configuration:**
See the complete configuration reference and examples in `config/README.md`:
Expand All @@ -163,19 +164,23 @@ See the complete configuration reference and examples in `config/README.md`:
- PDF Validation settings (rule-based quality checks)
- PDF encryption settings (password templating)
- Disease/chart/translation files
- Notice versioning catalog and assignment manifest

Direct link: [Configuration Reference](./config/README.md)

**Examples:**
```bash
# Basic usage
# Basic usage (fixed mode — all clients get the same language and template)
uv run viper students.xlsx en

# Override output directory
uv run viper students.xlsx en --output /tmp/output

# Use a PHU-specific template (from phu_templates/my_phu/)
uv run viper students.xlsx en --template my_phu

# Manifest mode — per-client notice version and language from assignment file
uv run viper students.xlsx --notice-assignments assignments.json --template my_phu
```

### Using PHU-Specific Templates
Expand Down Expand Up @@ -303,11 +308,43 @@ The preprocessed artifact contains:
}
```

In manifest mode, `assignment_mode` is `"manifest"`, `default_version` holds the catalog default version ID, and each client's `metadata` includes a `resolved_notice` object:

```json
{
"run_id": "20251023T200355",
"language": "en",
"assignment_mode": "manifest",
"default_version": "overdue_standard_v1",
"total_clients": 5,
"clients": [
{
"sequence": "00001",
"client_id": "1009876545",
"language": "fr",
"metadata": {
"recipient": "...",
"over_16": false,
"resolved_notice": {
"notice_version": "overdue_standard_v1",
"notice_kind": "overdue",
"language": "fr",
"experiment_id": null,
"experiment_arm": null,
"assignment_source": "manifest"
}
}
}
]
}
```

## Configuration quick links

- PHIX school validation: see [PHIX School Validation](./config/README.md#phix-school-validation)
- QR Code settings: see [QR Code Configuration](./config/README.md#qr-code-configuration)
- PDF Encryption settings: see [PDF Encryption Configuration](./config/README.md#pdf-encryption-configuration)
- Notice versioning catalog: see [Notice Versioning](./config/README.md#notice-versioning)
## Changelog

See [CHANGELOG.md](./CHANGELOG.md) for details of each release.
119 changes: 118 additions & 1 deletion config/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ This directory contains all configuration files for the immunization pipeline. E
- [QR Code Configuration](#qr-code-configuration)
- [PDF Validation Configuration](#pdf-validation-configuration)
- [PDF Encryption Configuration](#pdf-encryption-configuration)
- [Notice Versioning](#notice-versioning)
- [🏷️ Template Field Reference](#template-field-reference)
- [Adding New Configurations](#adding-new-configurations)

Expand All @@ -32,14 +33,17 @@ Raw Input (from CSV/Excel)
├─ disease_normalization.json → normalize variants
├─ vaccine_reference.json → expand vaccines to diseases
├─ parameters.yaml.chart_diseases_header → filter diseases not in chart → "Other"
└─ Emit artifact with filtered disease names
├─ notice_versions.yaml (optional) → load notice version catalog
├─ assignment manifest (optional) → reconcile per-client version/language assignments
└─ Emit artifact with filtered disease names (+ resolved_notice per client in manifest mode)
Artifact JSON (canonical English disease names, filtered by chart config)
[generate_notices.py]
├─ parameters.yaml.chart_diseases_header → load chart disease list
├─ translations/{lang}_diseases_chart.json → translate each disease name
├─ translations/{lang}_diseases_overdue.json → translate vaccines_due list
├─ (manifest mode) build template registry from per-version subdirectories
└─ Inject translated diseases into Typst template
Typst Files (with localized, filtered disease names)
Expand Down Expand Up @@ -436,6 +440,119 @@ All templates are validated at runtime to catch configuration errors early and p

---

## Notice Versioning

The notice versioning feature allows a single pipeline run to send different notice types (overdue, affirmative, informational) in different languages, by mapping each client to a specific notice version via a JSON assignment manifest. The feature is entirely **opt-in**: it is disabled when `config/notice_versions.yaml` is absent, and the pipeline behaves byte-for-byte identically to the fixed-mode default.

### `notice_versions.yaml`

**Purpose**: Catalog of notice version IDs and their eligibility kinds.

**Location**: `config/notice_versions.yaml`

**Format**:

```yaml
schema_version: 1
default_version: overdue_standard_v1
default_language: en

versions:
overdue_standard_v1:
kind: overdue
affirmative_schedule_v1:
kind: affirmative
informational_v1:
kind: informational
```

**Fields**:

| Field | Type | Description |
|-------|------|-------------|
| `schema_version` | int | Must be `1` |
| `default_version` | str | Version ID used for clients absent from the manifest when `allow_unassigned: true` |
| `default_language` | str | Language used for unassigned clients and as a fallback when a manifest row omits `language` |
| `versions` | map | Version ID → `{kind}` definition |

**Notice kinds**:

| Kind | Description | Eligibility rule |
|------|-------------|-----------------|
| `overdue` | Standard overdue notice | Client must have at least one vaccine due |
| `affirmative` | Notice for up-to-date clients | Client must have no vaccines due |
| `informational` | General informational notice | No eligibility constraint |

Eligibility conflicts (e.g., assigning an `affirmative` notice to a client with vaccines due) are caught at preflight and halt the pipeline before any PDF is generated.

### Assignment manifest format

The assignment manifest is a JSON array passed via `--notice-assignments`. Each entry maps a client ID to a version and language:

```json
[
{"client_id": "1009876545", "notice_version": "overdue_standard_v1", "language": "en"},
{"client_id": "2001234567", "notice_version": "affirmative_schedule_v1", "language": "fr"},
{"client_id": "3009876543", "notice_version": "overdue_standard_v1"}
]
```

**Fields**:

| Field | Required | Description |
|-------|----------|-------------|
| `client_id` | Yes | Must match a client ID in the input file |
| `notice_version` | Yes | Must match a version ID in `notice_versions.yaml` |
| `language` | No | ISO 639-1 language code; falls back to `default_language` when omitted |
| `experiment_id` | No | Optional experiment identifier (passed through to assignment metadata) |
| `experiment_arm` | No | Optional experiment arm (passed through to assignment metadata) |

### `parameters.yaml` — `notice_versioning` section

Optional behavior controls for manifest mode:

```yaml
notice_versioning:
allow_unassigned: false # true: unassigned clients use catalog defaults; false: error (default)
extra_manifest_rows: error # "error" or "warn" for manifest rows with no matching client
```

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| `allow_unassigned` | bool | `false` | When `true`, clients with no manifest row receive the catalog's `default_version` and `default_language` |
| `extra_manifest_rows` | str | `"error"` | When `"error"`, manifest rows for clients not in the input file halt the pipeline; when `"warn"`, they are logged and skipped |

### CLI usage

```bash
# Manifest mode — omit language, provide assignment file and catalog
uv run viper students.xlsx --notice-assignments assignments.json --template my_phu

# If --template is omitted in manifest mode, built-in templates/ is used
# (requires a subdirectory per version ID in templates/)
```

The `language` argument is **not required** in manifest mode. If supplied alongside `--notice-assignments`, it is ignored with a warning.

### Template directory layout for manifest mode

Each notice version must have its own subdirectory within the template directory:

```
phu_templates/my_phu/
├── overdue_standard_v1/
│ ├── en_template.py
│ └── fr_template.py
├── affirmative_schedule_v1/
│ ├── en_template.py
│ └── fr_template.py
└── conf.typ
```

The pipeline validates all required `(version_id, language)` pairs exist before rendering any client. Missing template paths are reported together so all gaps can be fixed in one pass.

---

## Adding New Configurations

### Adding a New Disease
Expand Down
10 changes: 2 additions & 8 deletions config/input_schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -86,19 +86,13 @@
"name": "overdue_disease",
"description": "Comma-separated list of overdue diseases for client",
"type": "string",
"stripWhitespace": true,
"constraints": {
"required": true
}
"stripWhitespace": true
},
{
"name": "overdue_agent",
"description": "Comma-separated list of overdue agents for client",
"type": "string",
"stripWhitespace": true,
"constraints": {
"required": true
}
"stripWhitespace": true
},
{
"name": "imms_given",
Expand Down
11 changes: 11 additions & 0 deletions config/notice_versions.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
schema_version: 1
default_version: overdue_standard_v1
default_language: en

versions:
overdue_standard_v1:
kind: overdue
requires: has_overdue
affirmative_schedule_v1:
kind: affirmative
requires: no_overdue
3 changes: 3 additions & 0 deletions config/parameters.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -54,3 +54,6 @@ qr:
typst:
bin: typst
font_path: /usr/share/fonts/truetype/freefont/
notice_versioning:
extra_manifest_rows: warn
allow_unassigned: False
18 changes: 18 additions & 0 deletions docs/reference/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,24 @@ Auto-generated from NumPy-format docstrings in the `pipeline/` package.

---

## Notice Versioning

::: pipeline.notice_versioning
options:
show_root_heading: true
show_root_full_path: true

---

## Assignment Manifest

::: pipeline.assignment_manifest
options:
show_root_heading: true
show_root_full_path: true

---

## Data Models

::: pipeline.data_models
Expand Down
11 changes: 8 additions & 3 deletions docs/reference/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,9 @@ Steps shown with dashed borders are optional — they are skipped when disabled
| Step | Module | Key Inputs | Key Outputs |
|------|--------|-----------|-------------|
| 1 | `prepare_output.py` | Config flags | Clean `output/` directory |
| 2 | `preprocess.py` | Excel file, `vaccine_reference.json`, `disease_normalization.json` | `preprocessed_clients_<run_id>.json` |
| 2 | `preprocess.py` | Excel file, `vaccine_reference.json`, `disease_normalization.json`, optional `notice_versions.yaml` + assignment manifest | `preprocessed_clients_<run_id>.json`; `notice_assignments_<run_id>.json` (manifest mode) |
| 3 | `generate_qr_codes.py` | Preprocessed JSON, QR config | PNG files in `output/artifacts/qr_codes/` |
| 4 | `generate_notices.py` | Preprocessed JSON, Typst templates | `.typ` files in `output/artifacts/typst/` |
| 4 | `generate_notices.py` | Preprocessed JSON, Typst templates (per-version subdirectories in manifest mode) | `.typ` files in `output/artifacts/typst/` |
| 5 | `compile_notices.py` | `.typ` files | PDF files in `output/pdf_individual/` |
| 6 | `validate_pdfs.py` | PDFs, artifact JSON | Console summary, `output/metadata/<lang>_validation_<run_id>.json` |
| 7 | `encrypt_notice.py` | Individual PDFs, encryption config | Encrypted PDFs in `output/pdf_individual/` |
Expand All @@ -51,7 +51,10 @@ Each step reads its inputs from disk and writes outputs to disk. The orchestrato
Preprocessing produces a single `preprocessed_clients_<run_id>.json` artifact that serves as the canonical source of truth for all downstream steps. Client records are deterministically ordered by school → last name → first name → client ID, and each client receives a stable sequence number (`00001`, `00002`, etc.) that persists through all downstream operations.

**Bilingual support**
Both English and French are first-class concerns. Disease names, notice text, and date formatting are all localized before being passed to Typst. The `language` argument selects the full rendering path; both languages share the same pipeline steps and configuration file.
Both English and French are first-class concerns. Disease names, notice text, and date formatting are all localized before being passed to Typst. In fixed mode the `language` argument selects a single rendering path shared by all clients. In manifest mode each client carries its own resolved language from the assignment manifest, enabling mixed-language runs.

**Notice versioning (manifest mode)**
When `config/notice_versions.yaml` is present and `--notice-assignments` is supplied, the pipeline enters manifest mode. Each client is mapped to a specific notice version (e.g., `overdue_standard_v1`, `affirmative_schedule_v1`) and language. A preflight gate after preprocessing catches missing clients, unknown version IDs, and eligibility conflicts before any PDF is generated. When the catalog file is absent, the pipeline behaves identically to fixed mode.

**Fail-fast vs. per-item recovery**
Critical steps (Preprocessing, Notice Generation, Compilation, PDF Validation) implement fail-fast: any error halts the pipeline immediately. Optional steps (QR Codes, Encryption, Bundling) implement per-item recovery: individual item failures are logged and skipped, and the pipeline continues processing remaining items.
Expand All @@ -75,6 +78,8 @@ pipeline/
├── enums.py # Language, BundleStrategy, TemplateField enums
├── translation_helpers.py # Disease name normalization and translation
├── validate_phix.py # PHIX school name validation (called from preprocess)
├── notice_versioning.py # Notice version catalog loader and eligibility validation
├── assignment_manifest.py # Assignment manifest loader, reconciliation, and preflight summary
└── utils.py # Template rendering and context building utilities

templates/ # Built-in Typst templates (EN/FR)
Expand Down
Loading