diff --git a/CHANGELOG.md b/CHANGELOG.md index 2ad8218..4969c74 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,11 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0. [Compare with v1.0.0](https://github.com/WDGPH/ImmuKnow/compare/v1.0.0...v1.1.0) +### Added + +- Notice versioning (assignment manifest mode): a new optional mode that maps each client to a specific notice version and language via a JSON assignment manifest. When `--notice-assignments ` is provided and `config/notice_versions.yaml` exists, the pipeline dispatches each client to the correct template and language individually - enabling a single run to produce overdue, affirmative, and informational notices in mixed languages. When the catalog file is absent, the pipeline behaves identically to before. +- Example `config/notice_versions.yaml` with `overdue_standard_v1` and `affirmative_schedule_v1`. + ### Changed - Switch input schema from fuzzy matching / data normalization to enforced Frictionless schema validation. Add schema page to mkdocs. diff --git a/README.md b/README.md index aaa5efe..42f7fb6 100644 --- a/README.md +++ b/README.md @@ -64,7 +64,7 @@ The `pipeline/` package is organized by pipeline function, not by layer. Each st | 8 | `bundle_pdfs.py` | PDF bundling & grouping (optional) | | 9 | `cleanup.py` | Intermediate file cleanup | -**Supporting modules:** `orchestrator.py` (orchestrator), `config_loader.py`, `data_models.py`, `enums.py`, `utils.py`. +**Supporting modules:** `orchestrator.py` (orchestrator), `config_loader.py`, `data_models.py`, `enums.py`, `utils.py`, `notice_versioning.py`, `assignment_manifest.py`. **Template modules** (in `templates/` package): `en_template.py`, `fr_template.py` (Typst template rendering). For module structure questions, see `docs/CODE_ANALYSIS_STANDARDS.md`. @@ -115,13 +115,13 @@ The main pipeline orchestrator (`orchestrator.py`) automates the end-to-end work Prepares the output directory, optionally removing existing contents while preserving logs. 2. **Preprocessing** (`preprocess.py`) - Cleans, validates, and structures input data into a normalized JSON artifact (`preprocessed_clients_.json`). Optionally validates school/daycare names against the PHIX reference mapping (see [PHIX School Validation](./config/README.md#phix-school-validation)). + Cleans, validates, and structures input data into a normalized JSON artifact (`preprocessed_clients_.json`). Optionally validates school/daycare names against the PHIX reference mapping (see [PHIX School Validation](./config/README.md#phix-school-validation)). In manifest mode, also reconciles the assignment manifest against the client list and runs a preflight gate before any PDF is generated. 3. **Generating QR Codes** (`generate_qr_codes.py`, optional) Generates QR code PNG files from templated payloads. Skipped if `qr.enabled: false` in `parameters.yaml`. 4. **Generating Notices** (`generate_notices.py`) - Renders Typst templates (`.typ` files) for each client from the preprocessed artifact, with QR code references. + Renders Typst templates (`.typ` files) for each client from the preprocessed artifact, with QR code references. In manifest mode, dispatches each client to the template specified by their assigned notice version and language. 5. **Compiling Notices** (`compile_notices.py`) Compiles Typst templates into individual PDF notices using the `typst` command-line tool. @@ -143,18 +143,19 @@ The main pipeline orchestrator (`orchestrator.py`) automates the end-to-end work **Usage Example:** ```bash -uv run viper [--output PATH] +uv run viper [language] [--output PATH] ``` **Required Arguments:** - ``: Name of the input file (e.g., `students.xlsx`) -- ``: Language code (`en` or `fr`) +- `[language]`: Language code (`en` or `fr`). Required in fixed mode; omit when using `--notice-assignments`. **Optional Arguments:** - `--input PATH`: Input directory (default: ../input) - `--output PATH`: Output directory (default: ../output) - `--config PATH`: Configuration directory (default: ../config) - `--template NAME`: PHU template name within `phu_templates/` (e.g., `wdgph`); defaults to built-in `templates/` when omitted +- `--notice-assignments PATH`: JSON file mapping client IDs to notice versions (enables manifest mode; requires `config/notice_versions.yaml`) **Configuration:** See the complete configuration reference and examples in `config/README.md`: @@ -163,12 +164,13 @@ See the complete configuration reference and examples in `config/README.md`: - PDF Validation settings (rule-based quality checks) - PDF encryption settings (password templating) - Disease/chart/translation files +- Notice versioning catalog and assignment manifest Direct link: [Configuration Reference](./config/README.md) **Examples:** ```bash -# Basic usage +# Basic usage (fixed mode β€” all clients get the same language and template) uv run viper students.xlsx en # Override output directory @@ -176,6 +178,9 @@ uv run viper students.xlsx en --output /tmp/output # Use a PHU-specific template (from phu_templates/my_phu/) uv run viper students.xlsx en --template my_phu + +# Manifest mode β€” per-client notice version and language from assignment file +uv run viper students.xlsx --notice-assignments assignments.json --template my_phu ``` ### Using PHU-Specific Templates @@ -303,11 +308,43 @@ The preprocessed artifact contains: } ``` +In manifest mode, `assignment_mode` is `"manifest"`, `default_version` holds the catalog default version ID, and each client's `metadata` includes a `resolved_notice` object: + +```json +{ + "run_id": "20251023T200355", + "language": "en", + "assignment_mode": "manifest", + "default_version": "overdue_standard_v1", + "total_clients": 5, + "clients": [ + { + "sequence": "00001", + "client_id": "1009876545", + "language": "fr", + "metadata": { + "recipient": "...", + "over_16": false, + "resolved_notice": { + "notice_version": "overdue_standard_v1", + "notice_kind": "overdue", + "language": "fr", + "experiment_id": null, + "experiment_arm": null, + "assignment_source": "manifest" + } + } + } + ] +} +``` + ## Configuration quick links - PHIX school validation: see [PHIX School Validation](./config/README.md#phix-school-validation) - QR Code settings: see [QR Code Configuration](./config/README.md#qr-code-configuration) - PDF Encryption settings: see [PDF Encryption Configuration](./config/README.md#pdf-encryption-configuration) +- Notice versioning catalog: see [Notice Versioning](./config/README.md#notice-versioning) ## Changelog See [CHANGELOG.md](./CHANGELOG.md) for details of each release. diff --git a/config/README.md b/config/README.md index b4463ab..91a8321 100644 --- a/config/README.md +++ b/config/README.md @@ -19,6 +19,7 @@ This directory contains all configuration files for the immunization pipeline. E - [QR Code Configuration](#qr-code-configuration) - [PDF Validation Configuration](#pdf-validation-configuration) - [PDF Encryption Configuration](#pdf-encryption-configuration) +- [Notice Versioning](#notice-versioning) - [🏷️ Template Field Reference](#template-field-reference) - [Adding New Configurations](#adding-new-configurations) @@ -32,7 +33,9 @@ Raw Input (from CSV/Excel) β”œβ”€ disease_normalization.json β†’ normalize variants β”œβ”€ vaccine_reference.json β†’ expand vaccines to diseases β”œβ”€ parameters.yaml.chart_diseases_header β†’ filter diseases not in chart β†’ "Other" - └─ Emit artifact with filtered disease names + β”œβ”€ notice_versions.yaml (optional) β†’ load notice version catalog + β”œβ”€ assignment manifest (optional) β†’ reconcile per-client version/language assignments + └─ Emit artifact with filtered disease names (+ resolved_notice per client in manifest mode) ↓ Artifact JSON (canonical English disease names, filtered by chart config) ↓ @@ -40,6 +43,7 @@ Artifact JSON (canonical English disease names, filtered by chart config) β”œβ”€ parameters.yaml.chart_diseases_header β†’ load chart disease list β”œβ”€ translations/{lang}_diseases_chart.json β†’ translate each disease name β”œβ”€ translations/{lang}_diseases_overdue.json β†’ translate vaccines_due list + β”œβ”€ (manifest mode) build template registry from per-version subdirectories └─ Inject translated diseases into Typst template ↓ Typst Files (with localized, filtered disease names) @@ -436,6 +440,119 @@ All templates are validated at runtime to catch configuration errors early and p --- +## Notice Versioning + +The notice versioning feature allows a single pipeline run to send different notice types (overdue, affirmative, informational) in different languages, by mapping each client to a specific notice version via a JSON assignment manifest. The feature is entirely **opt-in**: it is disabled when `config/notice_versions.yaml` is absent, and the pipeline behaves byte-for-byte identically to the fixed-mode default. + +### `notice_versions.yaml` + +**Purpose**: Catalog of notice version IDs and their eligibility kinds. + +**Location**: `config/notice_versions.yaml` + +**Format**: + +```yaml +schema_version: 1 +default_version: overdue_standard_v1 +default_language: en + +versions: + overdue_standard_v1: + kind: overdue + affirmative_schedule_v1: + kind: affirmative + informational_v1: + kind: informational +``` + +**Fields**: + +| Field | Type | Description | +|-------|------|-------------| +| `schema_version` | int | Must be `1` | +| `default_version` | str | Version ID used for clients absent from the manifest when `allow_unassigned: true` | +| `default_language` | str | Language used for unassigned clients and as a fallback when a manifest row omits `language` | +| `versions` | map | Version ID β†’ `{kind}` definition | + +**Notice kinds**: + +| Kind | Description | Eligibility rule | +|------|-------------|-----------------| +| `overdue` | Standard overdue notice | Client must have at least one vaccine due | +| `affirmative` | Notice for up-to-date clients | Client must have no vaccines due | +| `informational` | General informational notice | No eligibility constraint | + +Eligibility conflicts (e.g., assigning an `affirmative` notice to a client with vaccines due) are caught at preflight and halt the pipeline before any PDF is generated. + +### Assignment manifest format + +The assignment manifest is a JSON array passed via `--notice-assignments`. Each entry maps a client ID to a version and language: + +```json +[ + {"client_id": "1009876545", "notice_version": "overdue_standard_v1", "language": "en"}, + {"client_id": "2001234567", "notice_version": "affirmative_schedule_v1", "language": "fr"}, + {"client_id": "3009876543", "notice_version": "overdue_standard_v1"} +] +``` + +**Fields**: + +| Field | Required | Description | +|-------|----------|-------------| +| `client_id` | Yes | Must match a client ID in the input file | +| `notice_version` | Yes | Must match a version ID in `notice_versions.yaml` | +| `language` | No | ISO 639-1 language code; falls back to `default_language` when omitted | +| `experiment_id` | No | Optional experiment identifier (passed through to assignment metadata) | +| `experiment_arm` | No | Optional experiment arm (passed through to assignment metadata) | + +### `parameters.yaml` β€” `notice_versioning` section + +Optional behavior controls for manifest mode: + +```yaml +notice_versioning: + allow_unassigned: false # true: unassigned clients use catalog defaults; false: error (default) + extra_manifest_rows: error # "error" or "warn" for manifest rows with no matching client +``` + +| Key | Type | Default | Description | +|-----|------|---------|-------------| +| `allow_unassigned` | bool | `false` | When `true`, clients with no manifest row receive the catalog's `default_version` and `default_language` | +| `extra_manifest_rows` | str | `"error"` | When `"error"`, manifest rows for clients not in the input file halt the pipeline; when `"warn"`, they are logged and skipped | + +### CLI usage + +```bash +# Manifest mode β€” omit language, provide assignment file and catalog +uv run viper students.xlsx --notice-assignments assignments.json --template my_phu + +# If --template is omitted in manifest mode, built-in templates/ is used +# (requires a subdirectory per version ID in templates/) +``` + +The `language` argument is **not required** in manifest mode. If supplied alongside `--notice-assignments`, it is ignored with a warning. + +### Template directory layout for manifest mode + +Each notice version must have its own subdirectory within the template directory: + +``` +phu_templates/my_phu/ +β”œβ”€β”€ overdue_standard_v1/ +β”‚ β”œβ”€β”€ en_template.py +β”‚ └── fr_template.py +β”œβ”€β”€ affirmative_schedule_v1/ +β”‚ β”œβ”€β”€ en_template.py +β”‚ └── fr_template.py +└── conf.typ +``` + +The pipeline validates all required `(version_id, language)` pairs exist before rendering any client. Missing template paths are reported together so all gaps can be fixed in one pass. + +--- + ## Adding New Configurations ### Adding a New Disease diff --git a/config/input_schema.json b/config/input_schema.json index c660b73..2d147c0 100644 --- a/config/input_schema.json +++ b/config/input_schema.json @@ -86,19 +86,13 @@ "name": "overdue_disease", "description": "Comma-separated list of overdue diseases for client", "type": "string", - "stripWhitespace": true, - "constraints": { - "required": true - } + "stripWhitespace": true }, { "name": "overdue_agent", "description": "Comma-separated list of overdue agents for client", "type": "string", - "stripWhitespace": true, - "constraints": { - "required": true - } + "stripWhitespace": true }, { "name": "imms_given", diff --git a/config/notice_versions.yaml b/config/notice_versions.yaml new file mode 100644 index 0000000..a6647e6 --- /dev/null +++ b/config/notice_versions.yaml @@ -0,0 +1,11 @@ +schema_version: 1 +default_version: overdue_standard_v1 +default_language: en + +versions: + overdue_standard_v1: + kind: overdue + requires: has_overdue + affirmative_schedule_v1: + kind: affirmative + requires: no_overdue diff --git a/config/parameters.yaml b/config/parameters.yaml index bd8071e..b3805f0 100644 --- a/config/parameters.yaml +++ b/config/parameters.yaml @@ -54,3 +54,6 @@ qr: typst: bin: typst font_path: /usr/share/fonts/truetype/freefont/ +notice_versioning: + extra_manifest_rows: warn + allow_unassigned: False diff --git a/docs/reference/api.md b/docs/reference/api.md index 43d301e..562d0d9 100644 --- a/docs/reference/api.md +++ b/docs/reference/api.md @@ -26,6 +26,24 @@ Auto-generated from NumPy-format docstrings in the `pipeline/` package. --- +## Notice Versioning + +::: pipeline.notice_versioning + options: + show_root_heading: true + show_root_full_path: true + +--- + +## Assignment Manifest + +::: pipeline.assignment_manifest + options: + show_root_heading: true + show_root_full_path: true + +--- + ## Data Models ::: pipeline.data_models diff --git a/docs/reference/architecture.md b/docs/reference/architecture.md index 9be1347..a796d92 100644 --- a/docs/reference/architecture.md +++ b/docs/reference/architecture.md @@ -33,9 +33,9 @@ Steps shown with dashed borders are optional β€” they are skipped when disabled | Step | Module | Key Inputs | Key Outputs | |------|--------|-----------|-------------| | 1 | `prepare_output.py` | Config flags | Clean `output/` directory | -| 2 | `preprocess.py` | Excel file, `vaccine_reference.json`, `disease_normalization.json` | `preprocessed_clients_.json` | +| 2 | `preprocess.py` | Excel file, `vaccine_reference.json`, `disease_normalization.json`, optional `notice_versions.yaml` + assignment manifest | `preprocessed_clients_.json`; `notice_assignments_.json` (manifest mode) | | 3 | `generate_qr_codes.py` | Preprocessed JSON, QR config | PNG files in `output/artifacts/qr_codes/` | -| 4 | `generate_notices.py` | Preprocessed JSON, Typst templates | `.typ` files in `output/artifacts/typst/` | +| 4 | `generate_notices.py` | Preprocessed JSON, Typst templates (per-version subdirectories in manifest mode) | `.typ` files in `output/artifacts/typst/` | | 5 | `compile_notices.py` | `.typ` files | PDF files in `output/pdf_individual/` | | 6 | `validate_pdfs.py` | PDFs, artifact JSON | Console summary, `output/metadata/_validation_.json` | | 7 | `encrypt_notice.py` | Individual PDFs, encryption config | Encrypted PDFs in `output/pdf_individual/` | @@ -51,7 +51,10 @@ Each step reads its inputs from disk and writes outputs to disk. The orchestrato Preprocessing produces a single `preprocessed_clients_.json` artifact that serves as the canonical source of truth for all downstream steps. Client records are deterministically ordered by school β†’ last name β†’ first name β†’ client ID, and each client receives a stable sequence number (`00001`, `00002`, etc.) that persists through all downstream operations. **Bilingual support** -Both English and French are first-class concerns. Disease names, notice text, and date formatting are all localized before being passed to Typst. The `language` argument selects the full rendering path; both languages share the same pipeline steps and configuration file. +Both English and French are first-class concerns. Disease names, notice text, and date formatting are all localized before being passed to Typst. In fixed mode the `language` argument selects a single rendering path shared by all clients. In manifest mode each client carries its own resolved language from the assignment manifest, enabling mixed-language runs. + +**Notice versioning (manifest mode)** +When `config/notice_versions.yaml` is present and `--notice-assignments` is supplied, the pipeline enters manifest mode. Each client is mapped to a specific notice version (e.g., `overdue_standard_v1`, `affirmative_schedule_v1`) and language. A preflight gate after preprocessing catches missing clients, unknown version IDs, and eligibility conflicts before any PDF is generated. When the catalog file is absent, the pipeline behaves identically to fixed mode. **Fail-fast vs. per-item recovery** Critical steps (Preprocessing, Notice Generation, Compilation, PDF Validation) implement fail-fast: any error halts the pipeline immediately. Optional steps (QR Codes, Encryption, Bundling) implement per-item recovery: individual item failures are logged and skipped, and the pipeline continues processing remaining items. @@ -75,6 +78,8 @@ pipeline/ β”œβ”€β”€ enums.py # Language, BundleStrategy, TemplateField enums β”œβ”€β”€ translation_helpers.py # Disease name normalization and translation β”œβ”€β”€ validate_phix.py # PHIX school name validation (called from preprocess) +β”œβ”€β”€ notice_versioning.py # Notice version catalog loader and eligibility validation +β”œβ”€β”€ assignment_manifest.py # Assignment manifest loader, reconciliation, and preflight summary └── utils.py # Template rendering and context building utilities templates/ # Built-in Typst templates (EN/FR) diff --git a/docs/reference/pipeline_steps.md b/docs/reference/pipeline_steps.md index 5e63c86..c5b2529 100644 --- a/docs/reference/pipeline_steps.md +++ b/docs/reference/pipeline_steps.md @@ -45,6 +45,8 @@ Reads the raw Excel input, validates the schema, normalizes all client and vacci | `phix_validation.unmatched_behavior` | str | Action on unmatched schools: `warn`, `error`, or `skip` | | `date_notice_delivery` | ISO 8601 | Reference date for age-based eligibility (16+ threshold) | | `date_data_cutoff` | ISO 8601 | Date the source data was extracted from Panorama | +| `notice_versioning.allow_unassigned` | bool | When `true`, clients absent from the manifest receive catalog defaults (manifest mode only) | +| `notice_versioning.extra_manifest_rows` | str | How to treat manifest rows with no matching client: `"error"` or `"warn"` (manifest mode only) | **Inputs:** @@ -52,6 +54,8 @@ Reads the raw Excel input, validates the schema, normalizes all client and vacci - `config/vaccine_reference.json` β€” maps vaccine codes to disease names - `config/disease_normalization.json` β€” normalizes raw disease name variants - `config/phix_mapping.json` β€” PHU-keyed school name β†’ PHIX facility ID mapping (when PHIX validation enabled) +- `config/notice_versions.yaml` β€” notice version catalog (manifest mode only; feature is off when absent) +- Assignment manifest JSON (`--notice-assignments`) β€” per-client version and language assignments (manifest mode only) **Outputs:** @@ -60,6 +64,7 @@ Reads the raw Excel input, validates the schema, normalizes all client and vacci - `output/incomplete_addresses.csv` β€” records dropped due to missing address fields (written when any are found) - `output/incomplete_clients.csv` β€” records with missing required client fields, retained in processing (written when any are found) - `phix_exact.csv`, `phix_inexact.csv`, `phix_no_match.csv` β€” school match audit CSVs (when PHIX validation enabled) +- `output/metadata/notice_assignments_.json` β€” per-client assignment record, no PII (manifest mode only) **Processing:** @@ -75,6 +80,16 @@ Reads the raw Excel input, validates the schema, normalizes all client and vacci 10. Assigns stable sequence numbers (`00001`, `00002`, …) 11. Synthesizes missing school/board identifiers where needed 12. Writes the canonical JSON artifact +13. *(Manifest mode only)* Reconciles the assignment manifest against the client list; runs preflight checks (missing clients, unknown version IDs, eligibility conflicts); halts pipeline if any fatal issues are found; writes the assignment metadata file + +**Manifest preflight checks** (manifest mode only β€” all checked before any PDF is generated): + +| Check | Fatal? | Description | +|-------|--------|-------------| +| Missing clients | Always | Clients in the input with no manifest row (when `allow_unassigned: false`) | +| Unknown versions | Always | Manifest rows referencing a version ID not in `notice_versions.yaml` | +| Eligibility conflicts | Always | e.g., affirmative notice assigned to a client with vaccines due | +| Extra manifest rows | Configurable | Manifest rows with no matching client (`extra_manifest_rows: error` or `warn`) | --- @@ -125,6 +140,23 @@ Templates are loaded dynamically at runtime. The `--template` CLI argument selec Disease names are translated into the target language in Python before being passed to Typst β€” no runtime lookups occur in the Typst templates themselves. +**Manifest mode template layout** + +When running in manifest mode, each notice version requires its own template subdirectory: + +``` +phu_templates/my_phu/ +β”œβ”€β”€ overdue_standard_v1/ +β”‚ β”œβ”€β”€ en_template.py +β”‚ └── fr_template.py +β”œβ”€β”€ affirmative_schedule_v1/ +β”‚ β”œβ”€β”€ en_template.py +β”‚ └── fr_template.py +└── conf.typ +``` + +The pipeline builds a registry of all `(version_id, language)` pairs needed by the client list, verifies every required template exists before generating any file, and then dispatches each client to its resolved template. Missing templates are reported as a single error listing all absent paths. + --- ## Step 5 β€” PDF Compilation diff --git a/docs/user_guide/configuration.md b/docs/user_guide/configuration.md index f5e622d..3357284 100644 --- a/docs/user_guide/configuration.md +++ b/docs/user_guide/configuration.md @@ -19,6 +19,7 @@ This directory contains all configuration files for the immunization pipeline. E - [QR Code Configuration](#qr-code-configuration) - [PDF Validation Configuration](#pdf-validation-configuration) - [PDF Encryption Configuration](#pdf-encryption-configuration) +- [Notice Versioning](#notice-versioning) - [🏷️ Template Field Reference](#template-field-reference) - [Adding New Configurations](#adding-new-configurations) @@ -32,7 +33,9 @@ Raw Input (from CSV/Excel) β”œβ”€ disease_normalization.json β†’ normalize variants β”œβ”€ vaccine_reference.json β†’ expand vaccines to diseases β”œβ”€ parameters.yaml.chart_diseases_header β†’ filter diseases not in chart β†’ "Other" - └─ Emit artifact with filtered disease names + β”œβ”€ notice_versions.yaml (optional) β†’ load notice version catalog + β”œβ”€ assignment manifest (optional) β†’ reconcile per-client version/language assignments + └─ Emit artifact with filtered disease names (+ resolved_notice per client in manifest mode) ↓ Artifact JSON (canonical English disease names, filtered by chart config) ↓ @@ -40,6 +43,7 @@ Artifact JSON (canonical English disease names, filtered by chart config) β”œβ”€ parameters.yaml.chart_diseases_header β†’ load chart disease list β”œβ”€ translations/{lang}_diseases_chart.json β†’ translate each disease name β”œβ”€ translations/{lang}_diseases_overdue.json β†’ translate vaccines_due list + β”œβ”€ (manifest mode) build template registry from per-version subdirectories └─ Inject translated diseases into Typst template ↓ Typst Files (with localized, filtered disease names) @@ -452,6 +456,119 @@ All templates are validated at runtime to catch configuration errors early and p --- +## Notice Versioning + +The notice versioning feature allows a single pipeline run to send different notice types (overdue, affirmative, informational) in different languages, by mapping each client to a specific notice version via a JSON assignment manifest. The feature is entirely **opt-in**: it is disabled when `config/notice_versions.yaml` is absent, and the pipeline behaves byte-for-byte identically to the fixed-mode default. + +### `notice_versions.yaml` + +**Purpose**: Catalog of notice version IDs and their eligibility kinds. + +**Location**: `config/notice_versions.yaml` + +**Format**: + +```yaml +schema_version: 1 +default_version: overdue_standard_v1 +default_language: en + +versions: + overdue_standard_v1: + kind: overdue + affirmative_schedule_v1: + kind: affirmative + informational_v1: + kind: informational +``` + +**Fields**: + +| Field | Type | Description | +|-------|------|-------------| +| `schema_version` | int | Must be `1` | +| `default_version` | str | Version ID used for clients absent from the manifest when `allow_unassigned: true` | +| `default_language` | str | Language used for unassigned clients and as a fallback when a manifest row omits `language` | +| `versions` | map | Version ID β†’ `{kind}` definition | + +**Notice kinds**: + +| Kind | Description | Eligibility rule | +|------|-------------|-----------------| +| `overdue` | Standard overdue notice | Client must have at least one vaccine due | +| `affirmative` | Notice for up-to-date clients | Client must have no vaccines due | +| `informational` | General informational notice | No eligibility constraint | + +Eligibility conflicts (e.g., assigning an `affirmative` notice to a client with vaccines due) are caught at preflight and halt the pipeline before any PDF is generated. + +### Assignment manifest format + +The assignment manifest is a JSON array passed via `--notice-assignments`. Each entry maps a client ID to a version and language: + +```json +[ + {"client_id": "1009876545", "notice_version": "overdue_standard_v1", "language": "en"}, + {"client_id": "2001234567", "notice_version": "affirmative_schedule_v1", "language": "fr"}, + {"client_id": "3009876543", "notice_version": "overdue_standard_v1"} +] +``` + +**Fields**: + +| Field | Required | Description | +|-------|----------|-------------| +| `client_id` | Yes | Must match a client ID in the input file | +| `notice_version` | Yes | Must match a version ID in `notice_versions.yaml` | +| `language` | No | ISO 639-1 language code; falls back to `default_language` when omitted | +| `experiment_id` | No | Optional experiment identifier (passed through to assignment metadata) | +| `experiment_arm` | No | Optional experiment arm (passed through to assignment metadata) | + +### `parameters.yaml` β€” `notice_versioning` section + +Optional behavior controls for manifest mode: + +```yaml +notice_versioning: + allow_unassigned: false # true: unassigned clients use catalog defaults; false: error (default) + extra_manifest_rows: error # "error" or "warn" for manifest rows with no matching client +``` + +| Key | Type | Default | Description | +|-----|------|---------|-------------| +| `allow_unassigned` | bool | `false` | When `true`, clients with no manifest row receive the catalog's `default_version` and `default_language` | +| `extra_manifest_rows` | str | `"error"` | When `"error"`, manifest rows for clients not in the input file halt the pipeline; when `"warn"`, they are logged and skipped | + +### CLI usage + +```bash +# Manifest mode β€” omit language, provide assignment file and catalog +uv run viper students.xlsx --notice-assignments assignments.json --template my_phu + +# If --template is omitted in manifest mode, built-in templates/ is used +# (requires a subdirectory per version ID in templates/) +``` + +The `language` argument is **not required** in manifest mode. If supplied alongside `--notice-assignments`, it is ignored with a warning. + +### Template directory layout for manifest mode + +Each notice version must have its own subdirectory within the template directory: + +``` +phu_templates/my_phu/ +β”œβ”€β”€ overdue_standard_v1/ +β”‚ β”œβ”€β”€ en_template.py +β”‚ └── fr_template.py +β”œβ”€β”€ affirmative_schedule_v1/ +β”‚ β”œβ”€β”€ en_template.py +β”‚ └── fr_template.py +└── conf.typ +``` + +The pipeline validates all required `(version_id, language)` pairs exist before rendering any client. Missing template paths are reported together so all gaps can be fixed in one pass. + +--- + ## Adding New Configurations ### Adding a New Disease diff --git a/docs/user_guide/phu_templates.md b/docs/user_guide/phu_templates.md index 3c82ea0..8a4d169 100644 --- a/docs/user_guide/phu_templates.md +++ b/docs/user_guide/phu_templates.md @@ -95,6 +95,52 @@ If an asset referenced by your template is missing: FileNotFoundError: Logo not found: /path/to/phu_templates/my_phu/assets/logo.png ``` +## Notice Versioning (Manifest Mode) Templates + +When running in manifest mode (`--notice-assignments`), each notice version requires its own subdirectory inside the PHU template directory. The directory name must match the `version_id` from `config/notice_versions.yaml`. + +### Directory structure + +``` +phu_templates/my_phu/ +β”œβ”€β”€ overdue_standard_v1/ +β”‚ β”œβ”€β”€ en_template.py (required if any client gets overdue_standard_v1 in English) +β”‚ └── fr_template.py (required if any client gets overdue_standard_v1 in French) +β”œβ”€β”€ affirmative_schedule_v1/ +β”‚ β”œβ”€β”€ en_template.py +β”‚ └── fr_template.py +β”œβ”€β”€ conf.typ (shared Typst configuration β€” still at the top level) +└── assets/ (optional β€” shared across all versions) + β”œβ”€β”€ logo.png + └── signature.png +``` + +Each language template within a version subdirectory must define the same `render_notice()` function as fixed-mode templates. + +### Preflight template validation + +Before rendering any client, the pipeline checks that every `(version_id, language)` pair needed by the assignment manifest has a corresponding template file. All missing paths are reported in a single error so you can fix all gaps in one pass: + +``` +FileNotFoundError: Missing notice templates: + phu_templates/my_phu/affirmative_schedule_v1/fr_template.py + phu_templates/my_phu/informational_v1/en_template.py +``` + +### Fixed mode is unchanged + +Existing templates at the top level of the PHU directory (`en_template.py`, `fr_template.py`) continue to work for fixed-mode runs. No migration is needed unless you want to adopt manifest mode. + +### Example: adding a new version template + +```bash +mkdir -p phu_templates/my_phu/affirmative_schedule_v1 +cp phu_templates/my_phu/en_template.py phu_templates/my_phu/affirmative_schedule_v1/en_template.py +# Customize the template for the affirmative notice layout +``` + +--- + ## Git Considerations **Important:** PHU-specific templates are excluded from version control via `.gitignore`. diff --git a/input/rodent_dataset.csv b/input/rodent_dataset.csv index 0d1e725..c0a70da 100644 --- a/input/rodent_dataset.csv +++ b/input/rodent_dataset.csv @@ -1,8 +1,9 @@ ο»Ώschool_name,client_id,first_name,last_name,date_of_birth,street_address_line_1,street_address_line_2,city,province,postal_code,overdue_disease,overdue_agent,imms_given,Disease(s)/Agent(s),Imms History by Agent -WHISKER ELEMENTARY-1009876543,1009876543,Squeak,McCheese,2013-06-15,14 Burrow Lane,,Cheddarville,Ontario,M1C3E5,"Varicella, HPV, Hepatitis B","Var, HPV-9, Men-C-ACYW-135,","Aug 20, 2013 - DTaP-IPV-Hib; Aug 20, 2013 - Pneu-C-13; Aug 20, 2013 - rota-unspecified; Nov 18, 2013 - DTaP-IPV-Hib; Nov 18, 2013 - Pneu-C-13; Jan 25, 2014 - DTaP-IPV-Hib; May 12, 2014 - MMR; May 12, 2014 - Men-C-C; Oct 3, 2014 - Var; Apr 14, 2024 - Tdap-IPV;",Varicella (Var),"[2013 AUG 20: DTaP-IPV-Hib, Pneu-C-13, rota-unspecified] [2013 NOV 18: DTaP-IPV-Hib, Pneu-C-13] [2014 JAN 25: DTaP-IPV-Hib] [2014 MAY 12: MMR, Men-C-C] [2014 OCT 03: Var] [2024 APR 14: Tdap-IPV, MMR-Var]" -CHEESE WHEEL ACADEMY-1009876544,1009876544,Nibble,Sharpcheddar,2014-04-22,22 Gouda St,,Fromage City,Ontario,C3H3Z9,"Measles,","MMR,","Jul 10, 2014 - DTaP-IPV-Hib; Jul 10, 2014 - Pneu-C-13; Sep 15, 2014 - DTaP-IPV-Hib; Nov 20, 2014 - rota-unspecified; Mar 2, 2015 - MMR; Mar 2, 2015 - Men-C-C; Aug 7, 2015 - Var; Oct 1, 2015 - DTaP-IPV-Hib; May 19, 2024 - Tdap-IPV;",Measles (MMR),"[2014 JUL 10: DTaP-IPV-Hib, Pneu-C-13] [2014 SEP 15: DTaP-IPV-Hib] [2014 NOV 20: rota-unspecified] [2015 MAR 02: MMR, Men-C-C] [2015 AUG 07: Var] [2015 OCT 01: DTaP-IPV-Hib] [2024 MAY 19: Tdap-IPV]" -BURROW PUBLIC SCHOOL-1009876545,1009876545,Scurry,Nutcracker,2012-11-30,7 Tunnel Road,Unit 2,Gnawtown,Ontario,G9N8R2,"Hepatitis B,","HB,","Jan 5, 2013 - DTaP-IPV-Hib; Jan 5, 2013 - rota-unspecified; Mar 7, 2013 - Pneu-C-13; May 9, 2013 - DTaP-IPV-Hib; Jun 11, 2013 - MMR; Oct 23, 2013 - Men-C-C; Feb 2, 2014 - Var; May 6, 2014 - Pneu-C-13; Sep 12, 2014 - DTaP-IPV-Hib; May 1, 2024 - Tdap-IPV;",Hepatitis B (HB),"[2013 JAN 05: DTaP-IPV-Hib, rota-unspecified] [2013 MAR 07: Pneu-C-13] [2013 MAY 09: DTaP-IPV-Hib] [2013 JUN 11: MMR] [2013 OCT 23: Men-C-C] [2014 FEB 02: Var] [2014 MAY 06: Pneu-C-13] [2014 SEP 12: DTaP-IPV-Hib] [2024 MAY 01: Tdap-IPV]" -TUNNEL ACADEMY-1009876546,1009876546,Whiskers,Greyfur,2013-09-10,88 Haystack Drive,,Burrowville,Ontario,H8Y6T5,"Mumps,","MMR,","Oct 15, 2013 - DTaP-IPV-Hib; Dec 12, 2013 - rota-unspecified; Jan 17, 2014 - Pneu-C-13; Apr 8, 2014 - DTaP-IPV-Hib; Jun 19, 2014 - MMR; Oct 22, 2014 - Men-C-C; Feb 4, 2015 - Var; Sep 9, 2015 - DTaP-IPV-Hib; Apr 10, 2024 - Tdap-IPV;",Mumps (MMR),[2013 OCT 15: DTaP-IPV-Hib] [2013 DEC 12: rota-unspecified] [2014 JAN 17: Pneu-C-13] [2014 APR 08: DTaP-IPV-Hib] [2014 JUN 19: MMR] [2014 OCT 22: Men-C-C] [2015 FEB 04: Var] [2015 SEP 09: DTaP-IPV-Hib] [2024 APR 10: Tdap-IPV] -NUTCRACKER ACADEMY-1009876547,1009876547,Chisel,Teetherson,2014-02-28,3 Acorn Ave,Suite 1,Hazelton,Ontario,N4U2L1,"HPV,","HPV-9,","Mar 12, 2014 - DTaP-IPV-Hib; Mar 12, 2014 - rota-unspecified; May 14, 2014 - Pneu-C-13; Jul 19, 2014 - DTaP-IPV-Hib; Sep 21, 2014 - MMR; Nov 25, 2014 - Men-C-C; Apr 17, 2015 - Var; Sep 13, 2015 - DTaP-IPV-Hib; May 5, 2024 - Tdap-IPV;",HPV (HPV-9),"[2014 MAR 12: DTaP-IPV-Hib, rota-unspecified] [2014 MAY 14: Pneu-C-13] [2014 JUL 19: DTaP-IPV-Hib] [2014 SEP 21: MMR] [2014 NOV 25: Men-C-C] [2015 APR 17: Var] [2015 SEP 13: DTaP-IPV-Hib] [2024 MAY 05: Tdap-IPV]" -NUTCRACKER ACADEMY-1009876547,1009876548,Ratty,Teetherson,2009-02-28,,,Hazelton,Ontario,N4U2L1,"HPV,","HPV-9,","Mar 12, 2014 - DTaP-IPV-Hib; Mar 12, 2014 - rota-unspecified; May 14, 2014 - Pneu-C-13; Jul 19, 2014 - DTaP-IPV-Hib; Sep 21, 2014 - MMR; Nov 25, 2014 - Men-C-C; Apr 17, 2015 - Var; Sep 13, 2015 - DTaP-IPV-Hib; May 5, 2024 - Tdap-IPV;",HPV (HPV-9),"[2014 MAR 12: DTaP-IPV-Hib, rota-unspecified] [2014 MAY 14: Pneu-C-13] [2014 JUL 19: DTaP-IPV-Hib] [2014 SEP 21: MMR] [2014 NOV 25: Men-C-C] [2015 APR 17: Var] [2015 SEP 13: DTaP-IPV-Hib] [2024 MAY 05: Tdap-IPV]" -TUNNEL ACADEMY-1009876550,1009876550,Cheddarina,Swiftpaws,2014-09-14,44 Hayloft Road,,Burrowville,Ontario,H8Y6T6,MMR,MMR,"Jan 10, 2015 - DTaP-IPV-Hib; Jan 29, 2015 - Pneu-C-13; Feb 18, 2015 - rota-unspecified; Mar 07, 2015 - DTaP-IPV-Hib; Mar 28, 2015 - MMR; Apr 15, 2015 - Men-C-C; May 02, 2015 - Var; May 27, 2015 - DTaP-IPV-Hib; Jun 16, 2015 - Pneu-C-13; Jul 09, 2015 - Influenza (IIV4); Aug 01, 2015 - Influenza (IIV4); Aug 29, 2015 - MMR; Sep 22, 2015 - Var; Oct 11, 2015 - DTaP-IPV-Hib; Nov 05, 2015 - Pneu-C-13; Dec 03, 2015 - Men-C-C; Jan 14, 2016 - MMR; Feb 06, 2016 - Influenza (IIV4); Mar 12, 2016 - Hep A; Apr 04, 2016 - Hep A booster; May 18, 2016 - Yellow Fever; Jun 07, 2016 - Rabies (pre-exposure); Jun 30, 2016 - Rabies (pre-exposure) dose 2; Jul 23, 2016 - Rabies (pre-exposure) dose 3; Aug 15, 2016 - Var; Sep 08, 2016 - DTaP-IPV-Hib; Oct 01, 2016 - Pneu-C-13; Oct 27, 2016 - Influenza (IIV4); Nov 19, 2016 - MMR; Dec 14, 2016 - Men-C-C; Jan 09, 2017 - Var; Feb 03, 2017 - DTaP-IPV-Hib; Mar 01, 2017 - Pneu-C-13; Mar 29, 2017 - MMR; Apr 18, 2017 - Influenza (IIV4); May 10, 2017 - COVID-19 (Pfizer Pediatric); Jun 02, 2017 - COVID-19 (Pfizer Pediatric) dose 2; Jun 28, 2017 - COVID-19 Booster; Jul 20, 2017 - Var; Aug 12, 2017 - Men-C-C; Sep 03, 2017 - Influenza (IIV4); Oct 25, 2017 - DTaP-IPV-Hib; Nov 16, 2017 - Pneu-C-13; Dec 08, 2017 - MMR; May 02, 2023 - Tdap; Jan 18, 2024 - Men-C-ACYW-135; May 01, 2024 - Tdap-IPV",Measles (MMR),[2015 JAN 10: DTaP-IPV-Hib] [2015 JAN 29: Pneu-C-13] [2015 FEB 18: rota-unspecified] [2015 MAR 07: DTaP-IPV-Hib] [2015 MAR 28: MMR] [2015 APR 15: Men-C-C] [2015 MAY 02: Var] [2015 MAY 27: DTaP-IPV-Hib] [2015 JUN 16: Pneu-C-13] [2015 JUL 09: Influenza (IIV4)] [2015 AUG 01: Influenza (IIV4)] [2015 AUG 29: MMR] [2015 SEP 22: Var] [2015 OCT 11: DTaP-IPV-Hib] [2015 NOV 05: Pneu-C-13] [2015 DEC 03: Men-C-C] [2016 JAN 14: MMR] [2016 FEB 06: Influenza (IIV4)] [2016 MAR 12: Hep A] [2016 APR 04: Hep A booster] [2016 MAY 18: Yellow Fever] [2016 JUN 07: Rabies (pre-exposure)] [2016 JUN 30: Rabies (pre-exposure) dose 2] [2016 JUL 23: Rabies (pre-exposure) dose 3] [2016 AUG 15: Var] [2016 SEP 08: DTaP-IPV-Hib] [2016 OCT 01: Pneu-C-13] [2016 OCT 27: Influenza (IIV4)] [2016 NOV 19: MMR] [2016 DEC 14: Men-C-C] [2017 JAN 09: Var] [2017 FEB 03: DTaP-IPV-Hib] [2017 MAR 01: Pneu-C-13] [2017 MAR 29: MMR] [2017 APR 18: Influenza (IIV4)] [2017 MAY 10: COVID-19 (Pfizer Pediatric)] [2017 JUN 02: COVID-19 (Pfizer Pediatric) dose 2] [2017 JUN 28: COVID-19 Booster] [2017 JUL 20: Var] [2017 AUG 12: Men-C-C] [2017 SEP 03: Influenza (IIV4)] [2017 OCT 25: DTaP-IPV-Hib] [2017 NOV 16: Pneu-C-13] [2017 DEC 08: MMR] [2023 MAY 02: Tdap] [2024 JAN 18: Men-C-ACYW-135] [2024 MAY 01: Tdap-IPV] +WHISKER ELEMENTARY-1009876543,1009876543,Squeak,McCheese,2013-06-15,14 Burrow Lane,,Cheddarville,Ontario,M1C3E5,Varicella; HPV; Hepatitis B;,Var; HPV-9; Men-C-ACYW-135;,"Aug 20, 2013 - DTaP-IPV-Hib; Aug 20, 2013 - Pneu-C-13; Aug 20, 2013 - rota-unspecified; Nov 18, 2013 - DTaP-IPV-Hib; Nov 18, 2013 - Pneu-C-13; Jan 25, 2014 - DTaP-IPV-Hib; May 12, 2014 - MMR; May 12, 2014 - Men-C-C; Oct 3, 2014 - Var; Apr 14, 2024 - Tdap-IPV;",Varicella (Var),"[2013 AUG 20: DTaP-IPV-Hib, Pneu-C-13, rota-unspecified] [2013 NOV 18: DTaP-IPV-Hib, Pneu-C-13] [2014 JAN 25: DTaP-IPV-Hib] [2014 MAY 12: MMR, Men-C-C] [2014 OCT 03: Var] [2024 APR 14: Tdap-IPV, MMR-Var]" +CHEESE WHEEL ACADEMY-1009876544,1009876544,Nibble,Sharpcheddar,2014-04-22,22 Gouda St,,Fromage City,Ontario,C3H3Z9,Measles;,MMR;,"Jul 10, 2014 - DTaP-IPV-Hib; Jul 10, 2014 - Pneu-C-13; Sep 15, 2014 - DTaP-IPV-Hib; Nov 20, 2014 - rota-unspecified; Mar 2, 2015 - MMR; Mar 2, 2015 - Men-C-C; Aug 7, 2015 - Var; Oct 1, 2015 - DTaP-IPV-Hib; May 19, 2024 - Tdap-IPV;",Measles (MMR),"[2014 JUL 10: DTaP-IPV-Hib, Pneu-C-13] [2014 SEP 15: DTaP-IPV-Hib] [2014 NOV 20: rota-unspecified] [2015 MAR 02: MMR, Men-C-C] [2015 AUG 07: Var] [2015 OCT 01: DTaP-IPV-Hib] [2024 MAY 19: Tdap-IPV]" +BURROW PUBLIC SCHOOL-1009876545,1009876545,Scurry,Nutcracker,2012-11-30,7 Tunnel Road,Unit 2,Gnawtown,Ontario,G9N8R2,Hepatitis B;,HB;,"Jan 5, 2013 - DTaP-IPV-Hib; Jan 5, 2013 - rota-unspecified; Mar 7, 2013 - Pneu-C-13; May 9, 2013 - DTaP-IPV-Hib; Jun 11, 2013 - MMR; Oct 23, 2013 - Men-C-C; Feb 2, 2014 - Var; May 6, 2014 - Pneu-C-13; Sep 12, 2014 - DTaP-IPV-Hib; May 1, 2024 - Tdap-IPV;",Hepatitis B (HB),"[2013 JAN 05: DTaP-IPV-Hib, rota-unspecified] [2013 MAR 07: Pneu-C-13] [2013 MAY 09: DTaP-IPV-Hib] [2013 JUN 11: MMR] [2013 OCT 23: Men-C-C] [2014 FEB 02: Var] [2014 MAY 06: Pneu-C-13] [2014 SEP 12: DTaP-IPV-Hib] [2024 MAY 01: Tdap-IPV]" +TUNNEL ACADEMY-1009876546,1009876546,Whiskers,Greyfur,2013-09-10,88 Haystack Drive,,Burrowville,Ontario,H8Y6T5,Mumps;,MMR;,"Oct 15, 2013 - DTaP-IPV-Hib; Dec 12, 2013 - rota-unspecified; Jan 17, 2014 - Pneu-C-13; Apr 8, 2014 - DTaP-IPV-Hib; Jun 19, 2014 - MMR; Oct 22, 2014 - Men-C-C; Feb 4, 2015 - Var; Sep 9, 2015 - DTaP-IPV-Hib; Apr 10, 2024 - Tdap-IPV;",Mumps (MMR),[2013 OCT 15: DTaP-IPV-Hib] [2013 DEC 12: rota-unspecified] [2014 JAN 17: Pneu-C-13] [2014 APR 08: DTaP-IPV-Hib] [2014 JUN 19: MMR] [2014 OCT 22: Men-C-C] [2015 FEB 04: Var] [2015 SEP 09: DTaP-IPV-Hib] [2024 APR 10: Tdap-IPV] +NUTCRACKER ACADEMY-1009876547,1009876547,Chisel,Teetherson,2014-02-28,3 Acorn Ave,Suite 1,Hazelton,Ontario,N4U2L1,HPV;,HPV-9;,"Mar 12, 2014 - DTaP-IPV-Hib; Mar 12, 2014 - rota-unspecified; May 14, 2014 - Pneu-C-13; Jul 19, 2014 - DTaP-IPV-Hib; Sep 21, 2014 - MMR; Nov 25, 2014 - Men-C-C; Apr 17, 2015 - Var; Sep 13, 2015 - DTaP-IPV-Hib; May 5, 2024 - Tdap-IPV;",HPV (HPV-9),"[2014 MAR 12: DTaP-IPV-Hib, rota-unspecified] [2014 MAY 14: Pneu-C-13] [2014 JUL 19: DTaP-IPV-Hib] [2014 SEP 21: MMR] [2014 NOV 25: Men-C-C] [2015 APR 17: Var] [2015 SEP 13: DTaP-IPV-Hib] [2024 MAY 05: Tdap-IPV]" +NUTCRACKER ACADEMY-1009876547,1009876548,Ratty,Teetherson,2009-02-28,,,Hazelton,Ontario,N4U2L1,HPV;,HPV-9;,"Mar 12, 2014 - DTaP-IPV-Hib; Mar 12, 2014 - rota-unspecified; May 14, 2014 - Pneu-C-13; Jul 19, 2014 - DTaP-IPV-Hib; Sep 21, 2014 - MMR; Nov 25, 2014 - Men-C-C; Apr 17, 2015 - Var; Sep 13, 2015 - DTaP-IPV-Hib; May 5, 2024 - Tdap-IPV;",HPV (HPV-9),"[2014 MAR 12: DTaP-IPV-Hib, rota-unspecified] [2014 MAY 14: Pneu-C-13] [2014 JUL 19: DTaP-IPV-Hib] [2014 SEP 21: MMR] [2014 NOV 25: Men-C-C] [2015 APR 17: Var] [2015 SEP 13: DTaP-IPV-Hib] [2024 MAY 05: Tdap-IPV]" +NUTCRACKER ACADEMY-1009876547,1009876549,Rattata,Teetherson,2009-02-28,52 Claw Dr,,Hazelton,Ontario,N4U2L1,,,"Mar 12, 2014 - DTaP-IPV-Hib; Mar 12, 2014 - rota-unspecified; May 14, 2014 - Pneu-C-13; Jul 19, 2014 - DTaP-IPV-Hib; Sep 21, 2014 - MMR; Nov 25, 2014 - Men-C-C; Apr 17, 2015 - Var; Sep 13, 2015 - DTaP-IPV-Hib; May 5, 2024 - Tdap-IPV;",HPV (HPV-9),"[2014 MAR 12: DTaP-IPV-Hib, rota-unspecified] [2014 MAY 14: Pneu-C-13] [2014 JUL 19: DTaP-IPV-Hib] [2014 SEP 21: MMR] [2014 NOV 25: Men-C-C] [2015 APR 17: Var] [2015 SEP 13: DTaP-IPV-Hib] [2024 MAY 05: Tdap-IPV]" +TUNNEL ACADEMY-1009876550,1009876550,Cheddarina,Swiftpaws,2014-09-14,44 Hayloft Road,,Burrowville,Ontario,H8Y6T6,MMR;,MMR;,"Jan 10, 2015 - DTaP-IPV-Hib; Jan 29, 2015 - Pneu-C-13; Feb 18, 2015 - rota-unspecified; Mar 07, 2015 - DTaP-IPV-Hib; Mar 28, 2015 - MMR; Apr 15, 2015 - Men-C-C; May 02, 2015 - Var; May 27, 2015 - DTaP-IPV-Hib; Jun 16, 2015 - Pneu-C-13; Jul 09, 2015 - Influenza (IIV4); Aug 01, 2015 - Influenza (IIV4); Aug 29, 2015 - MMR; Sep 22, 2015 - Var; Oct 11, 2015 - DTaP-IPV-Hib; Nov 05, 2015 - Pneu-C-13; Dec 03, 2015 - Men-C-C; Jan 14, 2016 - MMR; Feb 06, 2016 - Influenza (IIV4); Mar 12, 2016 - Hep A; Apr 04, 2016 - Hep A booster; May 18, 2016 - Yellow Fever; Jun 07, 2016 - Rabies (pre-exposure); Jun 30, 2016 - Rabies (pre-exposure) dose 2; Jul 23, 2016 - Rabies (pre-exposure) dose 3; Aug 15, 2016 - Var; Sep 08, 2016 - DTaP-IPV-Hib; Oct 01, 2016 - Pneu-C-13; Oct 27, 2016 - Influenza (IIV4); Nov 19, 2016 - MMR; Dec 14, 2016 - Men-C-C; Jan 09, 2017 - Var; Feb 03, 2017 - DTaP-IPV-Hib; Mar 01, 2017 - Pneu-C-13; Mar 29, 2017 - MMR; Apr 18, 2017 - Influenza (IIV4); May 10, 2017 - COVID-19 (Pfizer Pediatric); Jun 02, 2017 - COVID-19 (Pfizer Pediatric) dose 2; Jun 28, 2017 - COVID-19 Booster; Jul 20, 2017 - Var; Aug 12, 2017 - Men-C-C; Sep 03, 2017 - Influenza (IIV4); Oct 25, 2017 - DTaP-IPV-Hib; Nov 16, 2017 - Pneu-C-13; Dec 08, 2017 - MMR; May 02, 2023 - Tdap; Jan 18, 2024 - Men-C-ACYW-135; May 01, 2024 - Tdap-IPV",Measles (MMR),[2015 JAN 10: DTaP-IPV-Hib] [2015 JAN 29: Pneu-C-13] [2015 FEB 18: rota-unspecified] [2015 MAR 07: DTaP-IPV-Hib] [2015 MAR 28: MMR] [2015 APR 15: Men-C-C] [2015 MAY 02: Var] [2015 MAY 27: DTaP-IPV-Hib] [2015 JUN 16: Pneu-C-13] [2015 JUL 09: Influenza (IIV4)] [2015 AUG 01: Influenza (IIV4)] [2015 AUG 29: MMR] [2015 SEP 22: Var] [2015 OCT 11: DTaP-IPV-Hib] [2015 NOV 05: Pneu-C-13] [2015 DEC 03: Men-C-C] [2016 JAN 14: MMR] [2016 FEB 06: Influenza (IIV4)] [2016 MAR 12: Hep A] [2016 APR 04: Hep A booster] [2016 MAY 18: Yellow Fever] [2016 JUN 07: Rabies (pre-exposure)] [2016 JUN 30: Rabies (pre-exposure) dose 2] [2016 JUL 23: Rabies (pre-exposure) dose 3] [2016 AUG 15: Var] [2016 SEP 08: DTaP-IPV-Hib] [2016 OCT 01: Pneu-C-13] [2016 OCT 27: Influenza (IIV4)] [2016 NOV 19: MMR] [2016 DEC 14: Men-C-C] [2017 JAN 09: Var] [2017 FEB 03: DTaP-IPV-Hib] [2017 MAR 01: Pneu-C-13] [2017 MAR 29: MMR] [2017 APR 18: Influenza (IIV4)] [2017 MAY 10: COVID-19 (Pfizer Pediatric)] [2017 JUN 02: COVID-19 (Pfizer Pediatric) dose 2] [2017 JUN 28: COVID-19 Booster] [2017 JUL 20: Var] [2017 AUG 12: Men-C-C] [2017 SEP 03: Influenza (IIV4)] [2017 OCT 25: DTaP-IPV-Hib] [2017 NOV 16: Pneu-C-13] [2017 DEC 08: MMR] [2023 MAY 02: Tdap] [2024 JAN 18: Men-C-ACYW-135] [2024 MAY 01: Tdap-IPV] diff --git a/pipeline/assignment_manifest.py b/pipeline/assignment_manifest.py new file mode 100644 index 0000000..80ac18b --- /dev/null +++ b/pipeline/assignment_manifest.py @@ -0,0 +1,244 @@ +"""Assignment manifest loading, reconciliation, and preflight reporting. + +A manifest is a JSON array that maps client IDs to notice versions, languages, +and optional experiment metadata. This module loads manifests, reconciles them +against the preprocessed cohort, and produces a ReconciliationResult that the +caller uses to decide whether to halt or continue. + +None of the functions here raise on policy violations - they populate the result +and the caller (build_preprocess_result / orchestrator) decides what to do. +""" + +from __future__ import annotations + +import dataclasses +import json +from pathlib import Path +from typing import TYPE_CHECKING, Dict, List, Optional + +from .notice_versioning import NoticeVersionCatalog, ResolvedNotice, validate_eligibility + +if TYPE_CHECKING: + from .data_models import ClientRecord + + +@dataclasses.dataclass(frozen=True) +class ManifestRow: + client_id: str + notice_version: str + language: Optional[str] + experiment_id: Optional[str] + experiment_arm: Optional[str] + + +@dataclasses.dataclass(frozen=True) +class ReconciliationResult: + # counts_by_version keys are "version_id (lang)" composite strings β€” matches + # the print_preflight_summary display format exactly. + counts_by_version: Dict[str, int] + counts_by_language: Dict[str, int] + missing_clients: List[str] # in cohort, not in manifest + extra_rows: List[str] # in manifest, not in cohort + duplicate_manifest_ids: List[str] # always empty; duplicates caught by load_manifest + unknown_versions: List[str] # version IDs not in catalog (deduplicated) + missing_language_clients: List[str] # client_ids whose manifest row has no language + eligibility_conflicts: List[str] # client_ids failing eligibility + default_language: str # catalog.default_language, used by print summary + + +def load_manifest(path: Path) -> Dict[str, ManifestRow]: + """Read a JSON assignment manifest and return a dict keyed by client_id. + + Raises ValueError for: + + - content that is not a JSON array + - rows missing client_id or notice_version + - duplicate client_id entries + """ + try: + raw = json.loads(path.read_text(encoding="utf-8")) + except json.JSONDecodeError as exc: + raise ValueError(f"Assignment manifest is not valid JSON: {path}") from exc + + if not isinstance(raw, list): + raise ValueError( + f"Assignment manifest must be a JSON array, got {type(raw).__name__}: {path}" + ) + + result: Dict[str, ManifestRow] = {} + seen: Dict[str, int] = {} # client_id -> first line index (1-based) + + for idx, item in enumerate(raw, start=1): + if not isinstance(item, dict): + raise ValueError( + f"Assignment manifest row {idx} must be an object, " + f"got {type(item).__name__}: {path}" + ) + + client_id = item.get("client_id") + if not client_id or not isinstance(client_id, str): + raise ValueError( + f"Assignment manifest row {idx} is missing required field 'client_id': {path}" + ) + + notice_version = item.get("notice_version") + if not notice_version or not isinstance(notice_version, str): + raise ValueError( + f"Assignment manifest row {idx} (client_id={client_id!r}) is missing " + f"required field 'notice_version': {path}" + ) + + if client_id in seen: + raise ValueError( + f"Assignment manifest has duplicate client_id {client_id!r} " + f"(first at row {seen[client_id]}, again at row {idx}): {path}" + ) + seen[client_id] = idx + + language = item.get("language") or None + experiment_id = item.get("experiment_id") or None + experiment_arm = item.get("experiment_arm") or None + + result[client_id] = ManifestRow( + client_id=client_id, + notice_version=notice_version, + language=language, + experiment_id=experiment_id, + experiment_arm=experiment_arm, + ) + + return result + + +def reconcile( + clients: "List[ClientRecord]", + manifest: Dict[str, ManifestRow], + catalog: NoticeVersionCatalog, + allow_unassigned: bool, + extra_manifest_rows: str, # "error" | "warn" +) -> ReconciliationResult: + """Reconcile a cohort against a manifest and return a populated ReconciliationResult. + + Does NOT raise - all policy decisions are left to the caller. + """ + cohort_ids = {c.client_id for c in clients} + manifest_ids = set(manifest.keys()) + + counts_by_version: Dict[str, int] = {} + counts_by_language: Dict[str, int] = {} + missing_clients: List[str] = [] + extra_rows: List[str] = sorted(manifest_ids - cohort_ids) + unknown_versions_set: set[str] = set() + missing_language_clients: List[str] = [] + eligibility_conflicts: List[str] = [] + + for client in clients: + cid = client.client_id + row = manifest.get(cid) + + if row is not None: + version = row.notice_version + + if version not in catalog.versions: + unknown_versions_set.add(version) + continue + + lang = row.language + if not lang: + lang = catalog.default_language + missing_language_clients.append(cid) + + catalog_version = catalog.versions[version] + resolved = ResolvedNotice( + notice_version=version, + notice_kind=catalog_version.kind.value, + language=lang, + experiment_id=row.experiment_id, + experiment_arm=row.experiment_arm, + assignment_source="manifest", + ) + try: + validate_eligibility(client, resolved, catalog) + except ValueError: + eligibility_conflicts.append(cid) + continue + + composite_key = f"{version} ({lang})" + counts_by_version[composite_key] = counts_by_version.get(composite_key, 0) + 1 + counts_by_language[lang] = counts_by_language.get(lang, 0) + 1 + + else: + if allow_unassigned: + version = catalog.default_version + lang = catalog.default_language + catalog_version = catalog.versions[version] + resolved = ResolvedNotice( + notice_version=version, + notice_kind=catalog_version.kind.value, + language=lang, + experiment_id=None, + experiment_arm=None, + assignment_source="default", + ) + try: + validate_eligibility(client, resolved, catalog) + except ValueError: + eligibility_conflicts.append(cid) + continue + + composite_key = f"{version} ({lang})" + counts_by_version[composite_key] = ( + counts_by_version.get(composite_key, 0) + 1 + ) + counts_by_language[lang] = counts_by_language.get(lang, 0) + 1 + else: + missing_clients.append(cid) + + return ReconciliationResult( + counts_by_version=counts_by_version, + counts_by_language=counts_by_language, + missing_clients=missing_clients, + extra_rows=extra_rows, + duplicate_manifest_ids=[], + unknown_versions=sorted(unknown_versions_set), + missing_language_clients=missing_language_clients, + eligibility_conflicts=eligibility_conflicts, + default_language=catalog.default_language, + ) + + +def has_errors(result: ReconciliationResult, extra_manifest_rows: str) -> bool: + """Return True if result contains anything that should halt the pipeline. + + The extra_manifest_rows policy ("error" | "warn") governs whether extra rows + count as an error. All other error categories are always fatal. + """ + if result.missing_clients: + return True + if result.unknown_versions: + return True + if result.eligibility_conflicts: + return True + if result.extra_rows and extra_manifest_rows == "error": + return True + return False + + +def print_preflight_summary(result: ReconciliationResult) -> None: + """Print the assignment preflight summary to stdout. + + No PII is emitted - only client counts, version IDs, and language codes. + """ + total = sum(result.counts_by_version.values()) + len(result.missing_clients) + print("Assignment mode: manifest") + print(f"Clients: {total}") + for composite_key, count in sorted(result.counts_by_version.items()): + print(f" {composite_key}: {count}") + print(f"Missing clients (no manifest row): {len(result.missing_clients)}") + print(f"Extra manifest rows (not in cohort): {len(result.extra_rows)}") + print(f"Unknown versions: {len(result.unknown_versions)}") + print( + f"Clients missing language (falling back to default " + f"'{result.default_language}'): {len(result.missing_language_clients)}" + ) + print(f"Eligibility conflicts: {len(result.eligibility_conflicts)}") diff --git a/pipeline/config_loader.py b/pipeline/config_loader.py index e4bb243..4d23689 100644 --- a/pipeline/config_loader.py +++ b/pipeline/config_loader.py @@ -241,3 +241,23 @@ def validate_config(config: Dict[str, Any]) -> None: f"cleanup.delete_unencrypted_pdfs must be a boolean, " f"got {type(delete_unencrypted).__name__}" ) + + # Validate optional notice_versioning config + notice_versioning_config = config.get("notice_versioning", {}) + if notice_versioning_config: + allow_unassigned = notice_versioning_config.get("allow_unassigned") + if allow_unassigned is not None and not isinstance(allow_unassigned, bool): + raise ValueError( + f"notice_versioning.allow_unassigned must be a boolean, " + f"got {type(allow_unassigned).__name__}" + ) + + extra_manifest_rows = notice_versioning_config.get("extra_manifest_rows") + if extra_manifest_rows is not None and extra_manifest_rows not in ( + "error", + "warn", + ): + raise ValueError( + f"notice_versioning.extra_manifest_rows must be 'error' or 'warn', " + f"got {extra_manifest_rows!r}" + ) diff --git a/pipeline/data_models.py b/pipeline/data_models.py index 033d8b7..2081650 100644 --- a/pipeline/data_models.py +++ b/pipeline/data_models.py @@ -17,6 +17,7 @@ class ClientRecord: This dataclass represents a single client (student) record passed through the entire pipeline. It contains all necessary information for: + - Generating personalized notices - Creating QR codes - Encrypting PDFs @@ -61,7 +62,10 @@ class ClientRecord: Comma-separated string of vaccines due (display format). vaccines_due_list : Optional[List[str]] - List of vaccine names/codes due. + List of canonical disease names due. + + vaccines_due_agent_list : Optional[List[str]] + List of vaccine agent names due. received : Optional[Sequence[Dict[str, object]]] List of vaccine records already received (structured data). @@ -86,6 +90,7 @@ class ClientRecord: contact: Dict[str, Any] vaccines_due: Optional[str] vaccines_due_list: Optional[List[str]] + vaccines_due_agent_list: Optional[List[str]] received: Optional[Sequence[Dict[str, object]]] metadata: Dict[str, object] qr: Optional[Dict[str, Any]] = None @@ -146,6 +151,8 @@ class ArtifactPayload: created_at: str input_file: Optional[str] = None total_clients: int = 0 + assignment_mode: str = "fixed" # "fixed" | "manifest" + default_version: Optional[str] = None @dataclass(frozen=True) diff --git a/pipeline/generate_notices.py b/pipeline/generate_notices.py index e0fb892..d943503 100644 --- a/pipeline/generate_notices.py +++ b/pipeline/generate_notices.py @@ -51,7 +51,7 @@ import re import sys from pathlib import Path -from typing import Dict, List, Mapping, Sequence +from typing import Callable, Dict, List, Mapping, Sequence, Set, Tuple from .config_loader import load_config from .data_models import ( @@ -143,6 +143,54 @@ def load_template_module(template_dir: Path, language_code: str): return module +def build_template_registry( + template_dir: Path, + needed: Set[Tuple[str, str]], +) -> Dict[Tuple[str, str], Callable]: + """Build a (version_id, language_code) β†’ renderer mapping for manifest mode. + + Fails at preflight (before any rendering) if any required template path is + missing. No fallback to templates/ when a PHU template_dir is set. + + Parameters + ---------- + template_dir : Path + Root template directory (templates/ or phu_templates//). + needed : Set[Tuple[str, str]] + Set of (version_id, language_code) pairs required for this run. + + Returns + ------- + Dict[Tuple[str, str], Callable] + Complete registry mapping each pair to its render_notice function. + + Raises + ------ + FileNotFoundError + If any required template path is missing, listing all missing paths. + """ + missing: List[str] = [] + registry: Dict[Tuple[str, str], Callable] = {} + + for version_id, lang_code in sorted(needed): + version_dir = template_dir / version_id + module_path = version_dir / f"{lang_code}_template.py" + if not module_path.exists(): + missing.append(str(module_path)) + else: + module = load_template_module(version_dir, lang_code) + registry[(version_id, lang_code)] = module.render_notice + + if missing: + raise FileNotFoundError( + "Missing template files for manifest mode. " + "The following paths are required but absent:\n" + + "\n".join(f" {p}" for p in missing) + ) + + return registry + + def build_language_renderers(template_dir: Path) -> dict: """Build renderer dictionary from templates in specified directory. @@ -290,6 +338,8 @@ def read_artifact(path: Path) -> ArtifactPayload: warnings=payload_dict.get("warnings", []), created_at=payload_dict.get("created_at", ""), total_clients=payload_dict.get("total_clients", len(clients)), + assignment_mode=payload_dict.get("assignment_mode", "fixed"), + default_version=payload_dict.get("default_version"), ) @@ -528,6 +578,10 @@ def build_template_context( else "" ) + # Agent list requires no translation β€” pass through as-is + vaccines_due_agents_array = client.vaccines_due_agent_list or [] + vaccines_due_agents_str = ", ".join(vaccines_due_agents_array) + # Translate received records' column keys received_translated: List[Dict[str, object]] = [] if client.received: @@ -549,6 +603,8 @@ def build_template_context( "client_data": to_typ_value(client_data), "vaccines_due_str": to_typ_value(vaccines_due_str_translated), "vaccines_due_array": to_typ_value(vaccines_due_array_translated), + "vaccines_due_agents_str": to_typ_value(vaccines_due_agents_str), + "vaccines_due_agents_array": to_typ_value(vaccines_due_agents_array), "received": to_typ_value(received_translated), "num_rows": str(len(received_translated)), "chart_diseases_translated": to_typ_value(chart_diseases_translated), @@ -663,34 +719,69 @@ def generate_typst_files( List[Path] List of generated .typ file paths """ - # Build renderers from specified template directory - renderers = build_language_renderers(template_dir) - output_dir.mkdir(parents=True, exist_ok=True) qr_output_dir = output_dir / "qr_codes" typst_output_dir = output_dir / "typst" typst_output_dir.mkdir(parents=True, exist_ok=True) files: List[Path] = [] - language = payload.language - for client in payload.clients: - if client.language != language: - raise ValueError( - f"Client {client.client_id} language {client.language!r} does not match artifact language {language!r}." + + # Detect manifest mode: any client with a resolved_notice in metadata. + manifest_mode = any( + isinstance(c.metadata, dict) and "resolved_notice" in c.metadata + for c in payload.clients + ) + + if manifest_mode: + # Collect all (version_id, language) pairs needed for this run and + # build the complete registry before touching any client (preflight). + needed: Set[Tuple[str, str]] = set() + for client in payload.clients: + resolved = client.metadata["resolved_notice"] # type: ignore[index] + needed.add((resolved["notice_version"], resolved["language"])) # type: ignore[index] + + registry = build_template_registry(template_dir, needed) + + for client in payload.clients: + resolved = client.metadata["resolved_notice"] # type: ignore[index] + version_id = resolved["notice_version"] # type: ignore[index] + lang = resolved["language"] # type: ignore[index] + renderer = registry[(version_id, lang)] + context = build_template_context(client, qr_output_dir, config_path) + typst_content = renderer( + context, + logo_path=to_root_relative(logo_path), + signature_path=to_root_relative(signature_path), ) - typst_content = render_notice( - client, - output_dir=output_dir, - logo=logo_path, - signature=signature_path, - renderers=renderers, - qr_output_dir=qr_output_dir, - config_path=config_path, - ) - filename = f"{language}_notice_{client.sequence}_{client.client_id}.typ" - file_path = typst_output_dir / filename - file_path.write_text(typst_content, encoding="utf-8") - files.append(file_path) - LOG.info("Wrote %s", file_path) + filename = f"{lang}_notice_{client.sequence}_{client.client_id}.typ" + file_path = typst_output_dir / filename + file_path.write_text(typst_content, encoding="utf-8") + files.append(file_path) + LOG.info("Wrote %s", file_path) + else: + # Fixed mode: single language, flat {lang}_template.py layout. + renderers = build_language_renderers(template_dir) + language = payload.language + for client in payload.clients: + if client.language != language: + raise ValueError( + f"Client {client.client_id} language {client.language!r} " + f"does not match artifact language {language!r}." + ) + typst_content = render_notice( + client, + output_dir=output_dir, + logo=logo_path, + signature=signature_path, + renderers=renderers, + qr_output_dir=qr_output_dir, + config_path=config_path, + ) + filename = f"{language}_notice_{client.sequence}_{client.client_id}.typ" + file_path = typst_output_dir / filename + file_path.write_text(typst_content, encoding="utf-8") + files.append(file_path) + LOG.info("Wrote %s", file_path) + return files diff --git a/pipeline/notice_versioning.py b/pipeline/notice_versioning.py new file mode 100644 index 0000000..3b4a831 --- /dev/null +++ b/pipeline/notice_versioning.py @@ -0,0 +1,172 @@ +"""Notice versioning models and catalog loader for the immunization pipeline. + +Supports the optional assignment manifest feature. When notice_versions.yaml is +absent from the config directory the feature is off and all callers receive None +from load_catalog(), leaving fixed-mode behaviour unchanged. +""" + +from __future__ import annotations + +import dataclasses +from enum import Enum +from pathlib import Path +from typing import TYPE_CHECKING, Callable, Dict, Optional + +import yaml + +if TYPE_CHECKING: + from .data_models import ClientRecord + + +class NoticeKind(str, Enum): + OVERDUE = "overdue" + AFFIRMATIVE = "affirmative" + INFORMATIONAL = "informational" + + +EligibilityRule = Callable[["ClientRecord"], bool] + +ELIGIBILITY_RULES: Dict[str, EligibilityRule] = { + "has_overdue": lambda c: bool(c.vaccines_due_list), + "no_overdue": lambda c: not c.vaccines_due_list, + "any": lambda _: True, +} + +# Fallback rule used when a version entry omits the `requires` field. +_KIND_DEFAULT_RULE: Dict[NoticeKind, str] = { + NoticeKind.OVERDUE: "has_overdue", + NoticeKind.AFFIRMATIVE: "no_overdue", + NoticeKind.INFORMATIONAL: "any", +} + + +@dataclasses.dataclass(frozen=True) +class NoticeVersion: + version_id: str + kind: NoticeKind + requires: str # key into ELIGIBILITY_RULES + + +@dataclasses.dataclass(frozen=True) +class NoticeVersionCatalog: + schema_version: int + default_version: str + default_language: str + versions: Dict[str, NoticeVersion] + + +@dataclasses.dataclass(frozen=True) +class ResolvedNotice: + notice_version: str + notice_kind: str # NoticeKind.value + language: str + experiment_id: Optional[str] + experiment_arm: Optional[str] + assignment_source: str # "manifest" | "default" + + +def load_catalog(config_dir: Path) -> Optional[NoticeVersionCatalog]: + """Load notice version catalog from config_dir/notice_versions.yaml. + + Returns None when the file is absent (feature stays off). + Raises ValueError with an actionable message if the file exists but is invalid. + """ + catalog_path = config_dir / "notice_versions.yaml" + if not catalog_path.exists(): + return None + + try: + raw = yaml.safe_load(catalog_path.read_text(encoding="utf-8")) or {} + except yaml.YAMLError as exc: + raise ValueError(f"notice_versions.yaml is invalid YAML: {exc}") from exc + + if "schema_version" not in raw: + raise ValueError( + "notice_versions.yaml is missing required field: schema_version" + ) + + if "default_version" not in raw: + raise ValueError( + "notice_versions.yaml is missing required field: default_version" + ) + + default_language = raw.get("default_language") + if not default_language or not isinstance(default_language, str): + raise ValueError( + "notice_versions.yaml: default_language must be a non-empty string" + ) + + raw_versions = raw.get("versions", {}) + if not isinstance(raw_versions, dict): + raise ValueError("notice_versions.yaml: versions must be a mapping") + + versions: Dict[str, NoticeVersion] = {} + for version_id, version_data in raw_versions.items(): + if not isinstance(version_id, str) or not version_id.strip(): + raise ValueError( + f"notice_versions.yaml: version ID must be a non-empty string, " + f"got {version_id!r}" + ) + kind_raw = ( + version_data.get("kind") if isinstance(version_data, dict) else None + ) + try: + kind = NoticeKind(kind_raw) + except (ValueError, KeyError): + valid = ", ".join(k.value for k in NoticeKind) + raise ValueError( + f"notice_versions.yaml: version {version_id!r} has invalid kind " + f"{kind_raw!r}. Valid kinds: {valid}" + ) + + requires_raw = ( + version_data.get("requires") if isinstance(version_data, dict) else None + ) + if requires_raw is None: + requires = _KIND_DEFAULT_RULE[kind] + elif requires_raw not in ELIGIBILITY_RULES: + valid_rules = ", ".join(sorted(ELIGIBILITY_RULES)) + raise ValueError( + f"notice_versions.yaml: version {version_id!r} has unknown requires " + f"{requires_raw!r}. Valid rules: {valid_rules}" + ) + else: + requires = requires_raw + + versions[version_id] = NoticeVersion( + version_id=version_id, kind=kind, requires=requires + ) + + default_version = raw["default_version"] + if default_version not in versions: + raise ValueError( + f"notice_versions.yaml: default_version {default_version!r} is not in " + f"versions. Available versions: {', '.join(sorted(versions.keys()))}" + ) + + return NoticeVersionCatalog( + schema_version=raw["schema_version"], + default_version=default_version, + default_language=default_language, + versions=versions, + ) + + +def validate_eligibility( + client_record: "ClientRecord", + resolved: ResolvedNotice, + catalog: NoticeVersionCatalog, +) -> None: + """Validate that the client satisfies the eligibility rule for the assigned version. + + Raises ValueError containing client_id (no name or DOB) if the rule is not met. + """ + version = catalog.versions[resolved.notice_version] + rule = ELIGIBILITY_RULES[version.requires] + if not rule(client_record): + raise ValueError( + f"Eligibility conflict for client {client_record.client_id}: " + f"assigned notice version '{resolved.notice_version}' " + f"(rule: '{version.requires}') but client does not satisfy it. " + "Check the manifest assignment or the client's overdue disease data." + ) diff --git a/pipeline/orchestrator.py b/pipeline/orchestrator.py index 4a18d2f..31cf95c 100755 --- a/pipeline/orchestrator.py +++ b/pipeline/orchestrator.py @@ -40,6 +40,7 @@ import traceback from datetime import datetime, timezone from pathlib import Path +from typing import Optional # Import pipeline steps from . import bundle_pdfs, cleanup, compile_notices, validate_pdfs @@ -50,8 +51,15 @@ prepare_output, preprocess, ) +from .assignment_manifest import ( + ReconciliationResult, + has_errors, + load_manifest, + print_preflight_summary, +) from .config_loader import load_config from .enums import Language +from .notice_versioning import NoticeVersionCatalog, load_catalog SCRIPT_DIR = Path(__file__).resolve().parent ROOT_DIR = SCRIPT_DIR.parent @@ -71,6 +79,7 @@ def parse_args() -> argparse.Namespace: Examples: %(prog)s students.xlsx en %(prog)s students.xlsx fr + %(prog)s students.xlsx --notice-assignments assignments.json """, ) @@ -81,8 +90,11 @@ def parse_args() -> argparse.Namespace: ) parser.add_argument( "language", + nargs="?", choices=sorted(Language.all_codes()), - help=f"Language for output ({', '.join(sorted(Language.all_codes()))})", + default=None, + help=f"Language for output ({', '.join(sorted(Language.all_codes()))}). " + "Required unless --notice-assignments is provided.", ) parser.add_argument( "--input", @@ -113,24 +125,54 @@ def parse_args() -> argparse.Namespace: help="PHU template name within phu_templates/ (e.g., 'wdgph'). " "If not specified, pipeline is run in testing mode, defaulting to the templates/ directory.", ) + parser.add_argument( + "--notice-assignments", + type=Path, + default=None, + dest="notice_assignments", + help="Path to JSON assignment manifest for notice versioning.", + ) return parser.parse_args() def validate_args(args: argparse.Namespace) -> None: """Validate command-line arguments and raise errors if invalid.""" + # --- Language / manifest mutual validation --- + if args.notice_assignments is None and args.language is None: + raise ValueError( + "language is required when not using --notice-assignments" + ) + + if args.notice_assignments is not None and args.language is not None: + print( + f"Warning: CLI language argument '{args.language}' is ignored in manifest " + "mode. Language is governed by the manifest and catalog default_language." + ) + args.language = None + + if args.notice_assignments is not None: + if not args.notice_assignments.exists(): + raise FileNotFoundError( + f"Assignment manifest not found: {args.notice_assignments}" + ) + catalog_path = args.config_dir / "notice_versions.yaml" + if not catalog_path.exists(): + raise ValueError( + "--notice-assignments requires notice_versions.yaml in the config " + f"directory ({args.config_dir})" + ) + + # --- Input file --- if args.input_file and not (args.input_dir / args.input_file).exists(): raise FileNotFoundError( f"Input file not found: {args.input_dir / args.input_file}" ) - # Resolve template directory + # --- Resolve template directory --- if args.template_dir is None: - # No custom template specified; use default args.template_dir = DEFAULT_TEMPLATES_DIR else: - # Custom PHU template specified; resolve within phu_templates/ - # Validate no path separators (prevent nested directories) if "/" in args.template_dir or "\\" in args.template_dir: raise ValueError( f"Template name cannot contain path separators: {args.template_dir}\n" @@ -148,10 +190,8 @@ def validate_args(args: argparse.Namespace) -> None: raise NotADirectoryError( f"PHU template path is not a directory: {phu_template_path}" ) - # Update args.template_dir to resolved Path args.template_dir = phu_template_path - # Validate template directory contents if not args.template_dir.is_dir(): raise NotADirectoryError( f"Template path is not a directory: {args.template_dir}" @@ -198,7 +238,6 @@ def run_step_1_prepare_output( ) if not success: - # User cancelled - exit with code 2 to match shell script return False return True @@ -208,63 +247,84 @@ def run_step_2_preprocess( input_dir: Path, input_file: str, output_dir: Path, - language: str, + language: Optional[str], run_id: str, config_dir: Path, -) -> int: + catalog: Optional[NoticeVersionCatalog] = None, + manifest: Optional[dict] = None, +) -> tuple[int, Optional[ReconciliationResult]]: """Step 2: Preprocessing. Returns: - Total number of clients processed. + Tuple of (total_clients, reconciliation_result). + reconciliation_result is None in fixed mode. """ print_step(2, "Preprocessing") - # Configure logging log_path = preprocess.configure_logging(output_dir, run_id) - # Load and process input data input_path = input_dir / input_file df_raw = preprocess.read_input(input_path) preprocess.validate_input(input_path) df = preprocess.normalize_dataframe(df_raw) + assignment_mode = "manifest" if catalog is not None else "fixed" + default_version = catalog.default_version if catalog is not None else None + # Check that addresses are complete, return only complete rows df = preprocess.check_addresses_complete(df, drop_incomplete=True) - df = preprocess.check_client_info_complete(df, drop_incomplete=True) + df = preprocess.check_client_info_complete(df, assignment_mode, drop_incomplete=True) - # Validate schools against PHIX mapping df, phix_warnings = preprocess.run_phix_validation(df, output_dir) - # Load configuration vaccine_reference_path = preprocess.VACCINE_REFERENCE_PATH vaccine_reference = json.loads(vaccine_reference_path.read_text(encoding="utf-8")) - # Build preprocessing result - result = preprocess.build_preprocess_result( + preprocess_result, reconciliation_result = preprocess.build_preprocess_result( df, language, vaccine_reference, preprocess.REPLACE_UNSPECIFIED, config_path=config_dir / "parameters.yaml", + catalog=catalog, + manifest=manifest, + ) + + # Determine effective language for the artifact header + effective_language = language or ( + catalog.default_language if catalog is not None else "en" ) - # Write artifact artifact_path = preprocess.write_artifact( - output_dir / "artifacts", language, run_id, result + output_dir / "artifacts", + effective_language, + run_id, + preprocess_result, + assignment_mode=assignment_mode, + default_version=default_version, ) + # Write per-client assignment metadata file in manifest mode + if catalog is not None and reconciliation_result is not None: + preprocess.write_assignment_metadata( + output_dir / "metadata", + run_id, + catalog, + reconciliation_result, + preprocess_result.clients, + ) + print(f"πŸ“„ Preprocessed artifact: {artifact_path}") print(f"Preprocess log written to {log_path}") - all_warnings = phix_warnings + result.warnings + all_warnings = phix_warnings + preprocess_result.warnings if all_warnings: print("Warnings detected during preprocessing:") for warning in all_warnings: print(f" - {warning}") - # Summarize the preprocessed clients - total_clients = len(result.clients) + total_clients = len(preprocess_result.clients) print(f"πŸ‘₯ Clients normalized: {total_clients}") - return total_clients + return total_clients, reconciliation_result def run_step_3_generate_qr_codes( @@ -292,7 +352,6 @@ def run_step_3_generate_qr_codes( artifacts_dir = output_dir / "artifacts" parameters_path = config_dir / "parameters.yaml" - # Generate QR codes generated = generate_qr_codes.generate_qr_codes( artifact_path, artifacts_dir, @@ -335,15 +394,9 @@ def run_step_4_generate_notices( artifact_path = output_dir / "artifacts" / f"preprocessed_clients_{run_id}.json" artifacts_dir = output_dir / "artifacts" - # Assets now come from template directory (optional) logo_path = template_dir / "assets" / "logo.png" signature_path = template_dir / "assets" / "signature.png" - # Note: Assets are NOT validated here. If a template references an asset - # that doesn't exist, the template rendering will fail with a clear error. - # This allows templates without assets to work without requiring dummy files. - - # Generate Typst files using main function generated = generate_notices.main( artifact_path, artifacts_dir, @@ -373,14 +426,12 @@ def run_step_5_compile_notices( """ print_step(5, "Compiling Typst templates") - # Load and validate configuration (fail-fast if invalid) load_config(config_dir / "parameters.yaml") artifacts_dir = output_dir / "artifacts" pdf_dir = output_dir / "pdf_individual" parameters_path = config_dir / "parameters.yaml" - # Compile Typst files using config-driven function compiled = compile_notices.compile_with_config( artifacts_dir, pdf_dir, @@ -406,24 +457,19 @@ def run_step_6_validate_pdfs( artifacts_dir = output_dir / "artifacts" preprocessed_json = artifacts_dir / f"preprocessed_clients_{run_id}.json" - # Load preprocessed clients to build client ID mapping client_id_map = {} import json with open(preprocessed_json, "r", encoding="utf-8") as f: preprocessed = json.load(f) clients = preprocessed.get("clients", []) - # Build map: filename -> client_id - # Filename format: {language}_notice_{sequence:05d}_{client_id}.pdf for idx, client in enumerate(clients, start=1): client_id = str(client.get("client_id", "")) - # Try to match any expected filename format for ext in [".pdf"]: for lang_prefix in ["en", "fr"]: filename = f"{lang_prefix}_notice_{idx:05d}_{client_id}{ext}" client_id_map[filename] = client_id - # Validate PDFs (module loads validation rules from config_dir) validate_pdfs.main( pdf_dir, language=language, @@ -445,7 +491,6 @@ def run_step_7_encrypt_pdfs( artifacts_dir = output_dir / "artifacts" json_file = artifacts_dir / f"preprocessed_clients_{run_id}.json" - # Encrypt PDFs using the combined preprocessed clients JSON encrypt_notice.encrypt_pdfs_in_directory( pdf_directory=pdf_dir, json_file=json_file, @@ -466,12 +511,10 @@ def run_step_8_bundle_pdfs( """ print_step(8, "Bundling PDFs") - # Load and validate configuration (fail-fast if invalid) config = load_config(config_dir / "parameters.yaml") parameters_path = config_dir / "parameters.yaml" - # Bundle PDFs using config-driven function results = bundle_pdfs.bundle_pdfs_with_config( output_dir, language, @@ -481,7 +524,6 @@ def run_step_8_bundle_pdfs( if results: print(f"Created {len(results)} bundles in {output_dir / 'pdf_combined'}") - # Display bundle information bundling_config = config.get("bundling", {}) bundle_size = bundling_config.get("bundle_size", 0) group_by = bundling_config.get("group_by") @@ -494,7 +536,6 @@ def run_step_8_bundle_pdfs( else: print("🏷️ Bundle scope: Sequential") - # Display manifest paths if results: print("πŸ“‹ Bundle manifests:") for result in results: @@ -546,21 +587,33 @@ def main() -> int: return 1 raise - # Setup paths and load configuration output_dir = args.output_dir.resolve() config_dir = args.config_dir.resolve() log_dir = output_dir / "logs" run_id = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S") - # Load configuration try: config = load_config(config_dir / "parameters.yaml") except FileNotFoundError as exc: print(f"Error: {exc}", file=sys.stderr) return 1 - # Extract config settings encryption_enabled = config.get("encryption", {}).get("enabled", False) + notice_versioning_cfg = config.get("notice_versioning", {}) + extra_manifest_rows_policy = notice_versioning_cfg.get("extra_manifest_rows", "error") + + # Load catalog and manifest before Step 1 so infrastructure errors fail fast. + catalog: Optional[NoticeVersionCatalog] = None + manifest: Optional[dict] = None + if args.notice_assignments: + catalog = load_catalog(config_dir) + manifest = load_manifest(args.notice_assignments) + + # Effective language for steps 3/6/7/8 that need a single language tag. + # In manifest mode args.language is None; fall back to catalog default. + effective_language: str = args.language or ( + catalog.default_language if catalog is not None else "en" + ) print_header(args.input_file) @@ -572,25 +625,37 @@ def main() -> int: # Step 1: Prepare output directory step_start = time.time() if not run_step_1_prepare_output(output_dir, log_dir, config_dir): - return 2 # User cancelled + return 2 step_duration = time.time() - step_start step_times.append(("Output Preparation", step_duration)) print_step_complete(1, "Output directory prepared", step_duration) # Step 2: Preprocessing step_start = time.time() - total_clients = run_step_2_preprocess( + total_clients, reconciliation_result = run_step_2_preprocess( args.input_dir, args.input_file, output_dir, args.language, run_id, config_dir, + catalog=catalog, + manifest=manifest, ) step_duration = time.time() - step_start step_times.append(("Preprocessing", step_duration)) print_step_complete(2, "Preprocessing", step_duration) + # Preflight gate (manifest mode only) + if reconciliation_result is not None: + print_preflight_summary(reconciliation_result) + if has_errors(reconciliation_result, extra_manifest_rows_policy): + print( + "\n❌ Manifest preflight failed. Correct the errors above and retry.", + file=sys.stderr, + ) + return 1 + # Step 3: Generating QR Codes (optional) step_start = time.time() qr_count = run_step_3_generate_qr_codes( @@ -630,7 +695,9 @@ def main() -> int: # Step 6: Validating PDFs step_start = time.time() - run_step_6_validate_pdfs(output_dir, args.language, run_id, config_dir) + run_step_6_validate_pdfs( + output_dir, effective_language, run_id, config_dir + ) step_duration = time.time() - step_start step_times.append(("PDF Validation", step_duration)) print_step_complete(6, "PDF validation", step_duration) @@ -638,7 +705,7 @@ def main() -> int: # Step 7: Encrypting PDFs (optional) if encryption_enabled: step_start = time.time() - run_step_7_encrypt_pdfs(output_dir, args.language, run_id) + run_step_7_encrypt_pdfs(output_dir, effective_language, run_id) step_duration = time.time() - step_start step_times.append(("PDF Encryption", step_duration)) print_step_complete(7, "Encryption", step_duration) @@ -651,7 +718,7 @@ def main() -> int: step_start = time.time() run_step_8_bundle_pdfs( output_dir, - args.language, + effective_language, run_id, config_dir, ) @@ -665,7 +732,6 @@ def main() -> int: # Step 9: Cleanup run_step_9_cleanup(output_dir, config_dir) - # Print summary total_duration = time.time() - total_start print_summary( diff --git a/pipeline/preprocess.py b/pipeline/preprocess.py index ff0e00c..13475c1 100644 --- a/pipeline/preprocess.py +++ b/pipeline/preprocess.py @@ -45,24 +45,27 @@ from __future__ import annotations +import dataclasses import json import logging import re from datetime import datetime, timezone from hashlib import sha1 from pathlib import Path -from typing import Any, Dict, List, Literal, Optional +from typing import Any, Dict, List, Literal, Optional, Tuple import pandas as pd import yaml from babel.dates import format_date from frictionless import Detector, Schema, validate as fl_validate +from .assignment_manifest import ManifestRow, ReconciliationResult, has_errors, reconcile from .data_models import ( ArtifactPayload, ClientRecord, PreprocessResult, ) from .enums import Language +from .notice_versioning import NoticeVersionCatalog, ResolvedNotice, validate_eligibility from .translation_helpers import normalize_disease SCRIPT_DIR = Path(__file__).resolve().parent @@ -209,7 +212,7 @@ def check_addresses_complete(df: pd.DataFrame, drop_incomplete=True) -> pd.DataF incomplete_records = df.loc[~df["address_complete"]] - incomplete_path = Path("output/incomplete_addresses.csv") + incomplete_path = SCRIPT_DIR.parent / "output" / "incomplete_addresses.csv" incomplete_records.to_csv(incomplete_path, index=False) LOG.info("Incomplete address records written to %s", incomplete_path) @@ -220,7 +223,7 @@ def check_addresses_complete(df: pd.DataFrame, drop_incomplete=True) -> pd.DataF return df.drop(columns=["address_complete"]) -def check_client_info_complete(df: pd.DataFrame, drop_incomplete=True) -> pd.DataFrame: +def check_client_info_complete(df: pd.DataFrame, assignment_mode, drop_incomplete=True) -> pd.DataFrame: """ Check if client fields are complete in the DataFrame. @@ -237,22 +240,21 @@ def check_client_info_complete(df: pd.DataFrame, drop_incomplete=True) -> pd.Dat "first_name", "last_name", "date_of_birth", - "overdue_disease", "imms_given", ] + # Default fixed mode should require non-empty overdue list + if assignment_mode == "fixed": + client_info_cols.extend([ + "overdue_disease", + "overdue_agent", + ]) + for col in client_info_cols: df[col] = df[col].astype(str).str.strip().replace({"": pd.NA, "nan": pd.NA}) # Check completeness - df["client_info_complete"] = ( - df["first_name"].notna() - & df["last_name"].notna() - & df["client_id"].notna() - & df["date_of_birth"].notna() - & df["overdue_disease"].notna() - & df["imms_given"].notna() - ) + df["client_info_complete"] = df[client_info_cols].notna().all(axis=1) if not df["client_info_complete"].all(): incomplete_count = (~df["client_info_complete"]).sum() @@ -266,7 +268,7 @@ def check_client_info_complete(df: pd.DataFrame, drop_incomplete=True) -> pd.Dat incomplete_records = df.loc[~df["client_info_complete"]] - incomplete_path = Path("output/incomplete_clients.csv") + incomplete_path = SCRIPT_DIR.parent / "output" / "incomplete_clients.csv" incomplete_records.to_csv(incomplete_path, index=False) LOG.info("Incomplete client records written to %s", incomplete_path) print(f"Incomplete client records written to {incomplete_path}") @@ -568,7 +570,7 @@ def synthesize_identifier(existing: str, source: str, prefix: str) -> str: return f"{prefix}_{digest}" -def process_vaccines_due(vaccines_due: Any, language: str) -> str: +def process_vaccines_due(vaccines_due: Any, mode: str) -> str: """Map overdue diseases to canonical disease names. Normalizes raw input disease strings to canonical disease names using @@ -579,8 +581,8 @@ def process_vaccines_due(vaccines_due: Any, language: str) -> str: ---------- vaccines_due : Any Raw string of comma-separated disease names from input. - language : str - Language code (e.g., "en", "fr"). Used for logging. + mode : str + Code for whether vaccine due list contains overdue "agent" or "disease". Returns ------- @@ -593,9 +595,17 @@ def process_vaccines_due(vaccines_due: Any, language: str) -> str: items: List[str] = [] for token in vaccines_due.split(";"): - # Normalize: raw input -> canonical disease name - normalized = normalize_disease(token.strip()) - items.append(normalized) + token = token.strip() + + # If list ends with ';', do not create enmpty entry in items + if not token: + continue + + if mode == "disease": + # Normalize: raw input -> canonical disease name + token: str = normalize_disease(token) + + items.append(token) # Filter empty items and clean quotes return ", ".join( @@ -1018,11 +1028,13 @@ def build_received_rows( def build_preprocess_result( df: pd.DataFrame, - language: str, + language: str | None, vaccine_reference: Dict[str, Any], replace_unspecified: List[str], config_path: Path | None = None, -) -> PreprocessResult: + catalog: Optional[NoticeVersionCatalog] = None, + manifest: Optional[Dict[str, ManifestRow]] = None, +) -> Tuple[PreprocessResult, Optional[ReconciliationResult]]: """Normalize client data and produce the structured preprocessing artifact. Orchestrates all per-dataset and per-client normalization: column @@ -1093,6 +1105,11 @@ def build_preprocess_result( include_dose: bool = preprocess_cfg.get("include_dose", False) show_validity_markers: bool = preprocess_cfg.get("show_validity_markers", False) + # Manifest-mode versioning settings + notice_versioning_cfg: Dict[str, Any] = params.get("notice_versioning", {}) + allow_unassigned: bool = notice_versioning_cfg.get("allow_unassigned", False) + extra_manifest_rows: str = notice_versioning_cfg.get("extra_manifest_rows", "error") + working["school_id"] = working.apply( lambda row: synthesize_identifier( row.get("school_id", ""), row["school_name"], "sch" @@ -1140,6 +1157,12 @@ def build_preprocess_result( "Default indicators will be used." ) + # Determine first-pass language: in manifest mode language arg is None, + # so fall back to catalog default for DOB formatting on the first pass. + first_pass_language: str = language or ( + catalog.default_language if catalog is not None else "en" + ) + clients: List[ClientRecord] = [] for row in sorted_df.itertuples(index=False): client_id = str(row.client_id) # type: ignore[attr-defined] @@ -1152,16 +1175,23 @@ def build_preprocess_result( if dob_iso is None: warnings.add(f"Missing date of birth for client {client_id}") - language_enum = Language.from_string(language) + language_enum = Language.from_string(first_pass_language) formatted_dob = ( convert_date_string(dob_iso, locale="fr") if language_enum == Language.FRENCH and dob_iso else (convert_date_string(dob_iso, locale="en") if dob_iso else None) ) - vaccines_due = process_vaccines_due(row.overdue_disease, language) # type: ignore[attr-defined] + vaccines_due = process_vaccines_due(row.overdue_disease, "disease") # type: ignore[attr-defined] + vaccines_due_agent = process_vaccines_due(row.overdue_agent, "agent") # type: ignore[attr-defined] + vaccines_due_list = [ item.strip() for item in vaccines_due.split(",") if item.strip() ] + + vaccines_due_agent_list = [ + item.strip() for item in vaccines_due_agent.split(",") if item.strip() + ] + for item in vaccines_due_list: disease, dose = split_vaccine_due_entry(item) if dose == "": @@ -1169,10 +1199,13 @@ def build_preprocess_result( f"Blank overdue dose number for client {client_id}: {disease}. " "Displaying disease without a dose number." ) + if include_dose: vaccines_due_list = format_vaccine_due_list(vaccines_due_list) else: vaccines_due_list = hide_vaccine_due_doses(vaccines_due_list) + + received = build_received_rows( row.imms_given, # type: ignore[attr-defined] replace_unspecified, @@ -1180,6 +1213,7 @@ def build_preprocess_result( chart_diseases_header, show_validity_markers, ) + postal_code = row.postal_code if row.postal_code else "Not provided" # type: ignore[attr-defined] address_line = " ".join( filter(None, [row.street_address_line_1, row.street_address_line_2]) # type: ignore[attr-defined] @@ -1222,13 +1256,14 @@ def build_preprocess_result( client = ClientRecord( sequence=sequence, client_id=client_id, - language=language, + language=first_pass_language, person=person, school=school, board=board, contact=contact, vaccines_due=vaccines_due if vaccines_due else None, vaccines_due_list=vaccines_due_list if vaccines_due_list else None, + vaccines_due_agent_list=vaccines_due_agent_list if vaccines_due_agent_list else None, received=received if received else None, metadata={ "version_id": row.version_id or None, # type: ignore[attr-defined] @@ -1252,9 +1287,73 @@ def build_preprocess_result( "Later records will overwrite earlier ones in generated notices." ) - return PreprocessResult( - clients=clients, - warnings=list(warnings), + # --- Fixed mode (no manifest) --- + if catalog is None or manifest is None: + return ( + PreprocessResult(clients=clients, warnings=list(warnings)), + None, + ) + + # --- Manifest mode --- + reconciliation_result = reconcile( + clients, manifest, catalog, allow_unassigned, extra_manifest_rows + ) + + if has_errors(reconciliation_result, extra_manifest_rows): + # Build a summary string for the error message (no PII). + lines = [ + "Manifest preflight failed:", + f" Missing clients (no manifest row): {len(reconciliation_result.missing_clients)}", + f" Unknown versions: {len(reconciliation_result.unknown_versions)}", + f" Eligibility conflicts: {len(reconciliation_result.eligibility_conflicts)}", + f" Extra manifest rows: {len(reconciliation_result.extra_rows)} " + f"(policy: {extra_manifest_rows})", + ] + raise ValueError("\n".join(lines)) + + # Rebuild each ClientRecord with resolved notice values. + rebuilt: List[ClientRecord] = [] + for first_pass_client in clients: + cid = first_pass_client.client_id + row_m = manifest.get(cid) + + if row_m is not None: + resolved_lang = row_m.language or catalog.default_language + resolved_version = row_m.notice_version + exp_id = row_m.experiment_id + exp_arm = row_m.experiment_arm + source = "manifest" + else: + # allow_unassigned=True (has_errors would have caught False case) + resolved_lang = catalog.default_language + resolved_version = catalog.default_version + exp_id = None + exp_arm = None + source = "default" + + catalog_version = catalog.versions[resolved_version] + resolved = ResolvedNotice( + notice_version=resolved_version, + notice_kind=catalog_version.kind.value, + language=resolved_lang, + experiment_id=exp_id, + experiment_arm=exp_arm, + assignment_source=source, + ) + + # Safety net β€” reconcile already caught conflicts before we reach here. + validate_eligibility(first_pass_client, resolved, catalog) + + rebuilt_client = dataclasses.replace( + first_pass_client, + language=resolved_lang, + metadata={"resolved_notice": dataclasses.asdict(resolved)}, + ) + rebuilt.append(rebuilt_client) + + return ( + PreprocessResult(clients=rebuilt, warnings=list(warnings)), + reconciliation_result, ) @@ -1313,12 +1412,16 @@ def run_phix_validation( def write_artifact( - output_dir: Path, language: str, run_id: str, result: PreprocessResult + output_dir: Path, + language: str, + run_id: str, + result: PreprocessResult, + assignment_mode: str = "fixed", + default_version: Optional[str] = None, ) -> Path: """Write preprocessed result to JSON artifact file.""" output_dir.mkdir(parents=True, exist_ok=True) - # Create ArtifactPayload with rich metadata artifact_payload = ArtifactPayload( run_id=run_id, language=language, @@ -1326,14 +1429,17 @@ def write_artifact( warnings=result.warnings, created_at=datetime.now(timezone.utc).isoformat(), total_clients=len(result.clients), + assignment_mode=assignment_mode, + default_version=default_version, ) - # Serialize to JSON (clients are dataclasses, so convert to dict) payload_dict = { "run_id": artifact_payload.run_id, "language": artifact_payload.language, "created_at": artifact_payload.created_at, "total_clients": artifact_payload.total_clients, + "assignment_mode": artifact_payload.assignment_mode, + "default_version": artifact_payload.default_version, "warnings": artifact_payload.warnings, "clients": [ { @@ -1365,6 +1471,7 @@ def write_artifact( }, "vaccines_due": client.vaccines_due, "vaccines_due_list": client.vaccines_due_list or [], + "vaccines_due_agent_list": client.vaccines_due_agent_list or [], "received": client.received or [], "metadata": client.metadata, } @@ -1378,6 +1485,60 @@ def write_artifact( return artifact_path +def write_assignment_metadata( + metadata_dir: Path, + run_id: str, + catalog: "NoticeVersionCatalog", + reconciliation_result: "ReconciliationResult", + clients: List[ClientRecord], +) -> Path: + """Write per-client assignment metadata to a JSON file (manifest mode only). + + Records contain only client_id, sequence, and resolved notice fields. + No name, date_of_birth, address, school name, or balancing attributes. + """ + metadata_dir.mkdir(parents=True, exist_ok=True) + + # Compute simple per-version and per-language totals from clients. + counts_by_version: Dict[str, int] = {} + counts_by_language: Dict[str, int] = {} + records = [] + for client in clients: + resolved = client.metadata.get("resolved_notice", {}) + version = resolved.get("notice_version", "") + lang = resolved.get("language", "") + counts_by_version[version] = counts_by_version.get(version, 0) + 1 + counts_by_language[lang] = counts_by_language.get(lang, 0) + 1 + records.append({ + "client_id": client.client_id, + "sequence": client.sequence, + "notice_version": version, + "notice_kind": resolved.get("notice_kind", ""), + "language": lang, + "experiment_id": resolved.get("experiment_id"), + "experiment_arm": resolved.get("experiment_arm"), + "assignment_source": resolved.get("assignment_source", ""), + }) + + payload = { + "schema_version": 1, + "run_id": run_id, + "created_at": datetime.now(timezone.utc).isoformat(), + "assignment_mode": "manifest", + "default_version": catalog.default_version, + "default_language": catalog.default_language, + "total_clients": len(clients), + "counts_by_version": counts_by_version, + "counts_by_language": counts_by_language, + "records": records, + } + + out_path = metadata_dir / f"notice_assignments_{run_id}.json" + out_path.write_text(json.dumps(payload, indent=2), encoding="utf-8") + LOG.info("Wrote assignment metadata to %s", out_path) + return out_path + + if __name__ == "__main__": import sys diff --git a/pipeline/utils.py b/pipeline/utils.py index 7528e27..13ddd2a 100644 --- a/pipeline/utils.py +++ b/pipeline/utils.py @@ -293,6 +293,7 @@ def deserialize_client_record(client_dict: dict) -> ClientRecord: contact=client_dict.get("contact", {}), vaccines_due=client_dict.get("vaccines_due"), vaccines_due_list=client_dict.get("vaccines_due_list"), + vaccines_due_agent_list=client_dict.get("vaccines_due_agent_list"), received=client_dict.get("received"), metadata=client_dict.get("metadata", {}), qr=client_dict.get("qr"), diff --git a/templates/affirmative_schedule_v1/en_template.py b/templates/affirmative_schedule_v1/en_template.py new file mode 100644 index 0000000..f7fce98 --- /dev/null +++ b/templates/affirmative_schedule_v1/en_template.py @@ -0,0 +1,203 @@ +"""English Typst template renderer. + +This module contains the English version of the immunization notice template. The +template generates a 2025 immunization notice in Typst format for dynamic PDF +rendering. + +The template defines the notice layout, including client information, immunization +requirements, vaccine records, QR codes, and contact instructions. All placeholder +values (client data, dates, vaccines) are dynamically substituted during rendering. + +Available placeholder variables include: +- client: Client data dict with person, school, board, contact info +- client_id: Unique client identifier +- immunizations_due: List of required vaccines +- qr_code: Optional QR code image path (if QR generation is enabled) +- date: Delivery/notice date +""" + +from __future__ import annotations + +from typing import Mapping + +TEMPLATE_PREFIX = """// --- CCEYA NOTICE TEMPLATE (TEST VERSION) --- // +// Description: A typst template that dynamically generates CCEYA templates. +// NOTE: All contact details are placeholders for testing purposes only. +// ----------------------------------------- // + +#import "/templates/conf.typ" + +// General document formatting +#set text(fill: black) +#set par(justify: false) +#set page("us-letter") + +// Formatting links - prevent URLs from splitting across lines +#show link: it => box(underline(it)) + +// Font formatting +#set text( + font: "FreeSans", + size: 10pt +) + +// Immunization Notice Section +#let immunization_notice(client, client_id, immunizations_due, date, font_size) = block[ + +#v(0.2cm) + +#conf.header_info_cim("__LOGO_PATH__", 6cm, black, 16pt, "Request for Immunization Record") + +#v(0.2cm) + +#conf.client_info_tbl_en(equal_split: false, vline: false, client, client_id, font_size, "Childcare Centre", 81pt, border: false) + +#v(0.3cm) + +// Notice for immunizations +As of *#date* our files show that your child is up-to-date for their immunization(s)! Their Immunization Record can be viewed on page 2. + + +For any future received vaccines, you can update your child's record by using one of the following options: + +1. By visiting #text(fill:conf.linkcolor)[#link("https://www.test-immunization.ca")] +2. By emailing #text(fill:conf.linkcolor)[#link("records@test-immunization.ca")] +3. By mailing a photocopy of your child's immunization record to Test Health, 123 Placeholder Street, Sample City, ON A1A 1A1 +4. By Phone: 555-555-5555 ext. 1234 + +Please update Public Health and your childcare centre every time your child receives a vaccine. + + +If you have any questions, please call 555-555-5555 ext. 1234. + + +Thank you for helping to keep our children safe. + + + Sincerely, + +#conf.signature("__SIGNATURE_PATH__", "Dr. Jane Smith, MPH", "Associate Medical Officer of Health") + +// Invisible marker for layout validation +#box(width: 0pt, height: 0pt)[ + #text(size: 0.1pt, fill: white)[MARK_END_SIGNATURE_BLOCK] +] + +] + +#let vaccine_table_page(client_id) = block[ + + #v(0.5cm) + + #grid( + + columns: (50%,50%), + gutter: 5%, + [#image("__LOGO_PATH__", width: 6cm)], + [#set align(center + bottom) + #text(size: 20.5pt, fill: black)[*Immunization Record*]] + +) + + #v(0.5cm) + + For your reference, the immunization(s) on file with Public Health are as follows: + +] + +#let end_of_immunization_notice() = [ + #set align(center) + End of immunization record ] +""" + +DYNAMIC_BLOCK = """ +#let client_row = __CLIENT_ROW__ +#let data = __CLIENT_DATA__ +#let vaccines_due = __VACCINES_DUE_STR__ +#let vaccines_due_array = __VACCINES_DUE_ARRAY__ +#let received = __RECEIVED__ +#let num_rows = __NUM_ROWS__ +#let diseases = __CHART_DISEASES_TRANSLATED__ +#let show_validity_markers = __SHOW_VALIDITY_MARKERS__ +#let date = data.date_data_cutoff + +#set page( + margin: (top: 1cm, bottom: 2cm, left: 1.75cm, right: 2cm), + footer: align(center, context numbering("1 / " + str(counter(page).final().first()), counter(page).get().first())) +) + +#immunization_notice(data, client_row, vaccines_due_array, date, 11pt) +#pagebreak() +#vaccine_table_page(client_row.at(0)) +#conf.immunization-table(5, num_rows, received, diseases, 11pt, "en", show_validity_markers) +#end_of_immunization_notice() +""" + + +def render_notice( + context: Mapping[str, str], + *, + logo_path: str, + signature_path: str, +) -> str: + """Render the Typst document for a single English notice. + + Parameters + ---------- + context : Mapping[str, str] + Dictionary containing template placeholder values. Must include: + - client_row: Row identifier + - client_data: Client information dict + - vaccines_due_str: Formatted string of vaccines due + - vaccines_due_array: Array of vaccines due + - received: Received vaccine data + - num_rows: Number of table rows + - chart_diseases_translated: Translated disease names for chart columns + + logo_path : str + Absolute path to logo image file + signature_path : str + Absolute path to signature image file + + Returns + ------- + str + Rendered Typst template with all placeholders replaced + + Raises + ------ + KeyError + If any required context keys are missing + """ + required_keys = ( + "client_row", + "client_data", + "vaccines_due_str", + "vaccines_due_array", + "vaccines_due_agents_str", + "vaccines_due_agents_array", + "received", + "num_rows", + "chart_diseases_translated", + ) + + missing = [key for key in required_keys if key not in context] + if missing: + missing_keys = ", ".join(missing) + raise KeyError(f"Missing context keys: {missing_keys}") + + prefix = TEMPLATE_PREFIX.replace("__LOGO_PATH__", logo_path).replace( + "__SIGNATURE_PATH__", signature_path + ) + + dynamic = ( + DYNAMIC_BLOCK.replace("__CLIENT_ROW__", context["client_row"]) + .replace("__CLIENT_DATA__", context["client_data"]) + .replace("__VACCINES_DUE_STR__", context["vaccines_due_agents_str"]) + .replace("__VACCINES_DUE_ARRAY__", context["vaccines_due_agents_array"]) + .replace("__RECEIVED__", context["received"]) + .replace("__NUM_ROWS__", context["num_rows"]) + .replace("__CHART_DISEASES_TRANSLATED__", context["chart_diseases_translated"]) + .replace("__SHOW_VALIDITY_MARKERS__", context.get("show_validity_markers", "false")) + ) + return prefix + dynamic diff --git a/templates/overdue_standard_v1/en_template.py b/templates/overdue_standard_v1/en_template.py new file mode 100644 index 0000000..399772e --- /dev/null +++ b/templates/overdue_standard_v1/en_template.py @@ -0,0 +1,214 @@ +"""English Typst template renderer. + +This module contains the English version of the immunization notice template. The +template generates a 2025 immunization notice in Typst format for dynamic PDF +rendering. + +The template defines the notice layout, including client information, immunization +requirements, vaccine records, QR codes, and contact instructions. All placeholder +values (client data, dates, vaccines) are dynamically substituted during rendering. + +Available placeholder variables include: +- client: Client data dict with person, school, board, contact info +- client_id: Unique client identifier +- immunizations_due: List of required vaccines +- qr_code: Optional QR code image path (if QR generation is enabled) +- date: Delivery/notice date +""" + +from __future__ import annotations + +from typing import Mapping + +TEMPLATE_PREFIX = """// --- CCEYA NOTICE TEMPLATE (TEST VERSION) --- // +// Description: A typst template that dynamically generates CCEYA templates. +// NOTE: All contact details are placeholders for testing purposes only. +// ----------------------------------------- // + +#import "/templates/conf.typ" + +// General document formatting +#set text(fill: black) +#set par(justify: false) +#set page("us-letter") + +// Formatting links - prevent URLs from splitting across lines +#show link: it => box(underline(it)) + +// Font formatting +#set text( + font: "FreeSans", + size: 10pt +) + +// Immunization Notice Section +#let immunization_notice(client, client_id, immunizations_due, date, font_size) = block[ + +#v(0.2cm) + +#conf.header_info_cim("__LOGO_PATH__", 6cm, black, 16pt, "Request for Immunization Record") + +#v(0.2cm) + +#conf.client_info_tbl_en(equal_split: false, vline: false, client, client_id, font_size, "Childcare Centre", 81pt, border: false) + +#v(0.3cm) + +// Notice for immunizations +As of *#date* our files show that your child has not received the following immunization(s): + +#conf.client_immunization_list(immunizations_due) + +Please review the Immunization Record on page 2 and update your child's record by using one of the following options: + +1. By visiting #text(fill:conf.linkcolor)[#link("https://www.test-immunization.ca")] +2. By emailing #text(fill:conf.linkcolor)[#link("records@test-immunization.ca")] +3. By mailing a photocopy of your child's immunization record to Test Health, 123 Placeholder Street, Sample City, ON A1A 1A1 +4. By Phone: 555-555-5555 ext. 1234 + +Please update Public Health and your childcare centre every time your child receives a vaccine. + +#grid( + columns: (1fr, auto), + gutter: 10pt, + [*If you are choosing not to immunize your child*, a valid medical exemption or statement of conscience or religious belief must be submitted. Links to these forms can be located at #text(fill:conf.wdgteal)[#link("https://www.test-immunization.ca/exemptions")]. Please note this exemption is for childcare only and a new exemption will be required upon enrollment in elementary school.], + [#if "qr_img" in client [ + #if "qr_url" in client [ + #link(client.qr_url)[#image(client.qr_img, width: 3cm)] + ] else [ + #image(client.qr_img, width: 3cm) + ] + ]] +) + +If there is an outbreak, children who are not adequately immunized may be excluded. + +If you have any questions, please call 555-555-5555 ext. 1234. + + Sincerely, + +#conf.signature("__SIGNATURE_PATH__", "Dr. Jane Smith, MPH", "Associate Medical Officer of Health") + +// Invisible marker for layout validation +#box(width: 0pt, height: 0pt)[ + #text(size: 0.1pt, fill: white)[MARK_END_SIGNATURE_BLOCK] +] + +] + +#let vaccine_table_page(client_id) = block[ + + #v(0.5cm) + + #grid( + + columns: (50%,50%), + gutter: 5%, + [#image("__LOGO_PATH__", width: 6cm)], + [#set align(center + bottom) + #text(size: 20.5pt, fill: black)[*Immunization Record*]] + +) + + #v(0.5cm) + + For your reference, the immunization(s) on file with Public Health are as follows: + +] + +#let end_of_immunization_notice() = [ + #set align(center) + End of immunization record ] +""" + +DYNAMIC_BLOCK = """ +#let client_row = __CLIENT_ROW__ +#let data = __CLIENT_DATA__ +#let vaccines_due = __VACCINES_DUE_STR__ +#let vaccines_due_array = __VACCINES_DUE_ARRAY__ +#let received = __RECEIVED__ +#let num_rows = __NUM_ROWS__ +#let diseases = __CHART_DISEASES_TRANSLATED__ +#let show_validity_markers = __SHOW_VALIDITY_MARKERS__ +#let date = data.date_data_cutoff + +#set page( + margin: (top: 1cm, bottom: 2cm, left: 1.75cm, right: 2cm), + footer: align(center, context numbering("1 / " + str(counter(page).final().first()), counter(page).get().first())) +) + +#immunization_notice(data, client_row, vaccines_due_array, date, 11pt) +#pagebreak() +#vaccine_table_page(client_row.at(0)) +#conf.immunization-table(5, num_rows, received, diseases, 11pt, "en", show_validity_markers) +#end_of_immunization_notice() +""" + + +def render_notice( + context: Mapping[str, str], + *, + logo_path: str, + signature_path: str, +) -> str: + """Render the Typst document for a single English notice. + + Parameters + ---------- + context : Mapping[str, str] + Dictionary containing template placeholder values. Must include: + - client_row: Row identifier + - client_data: Client information dict + - vaccines_due_str: Formatted string of vaccines due + - vaccines_due_array: Array of vaccines due + - received: Received vaccine data + - num_rows: Number of table rows + - chart_diseases_translated: Translated disease names for chart columns + + logo_path : str + Absolute path to logo image file + signature_path : str + Absolute path to signature image file + + Returns + ------- + str + Rendered Typst template with all placeholders replaced + + Raises + ------ + KeyError + If any required context keys are missing + """ + required_keys = ( + "client_row", + "client_data", + "vaccines_due_str", + "vaccines_due_array", + "vaccines_due_agents_str", + "vaccines_due_agents_array", + "received", + "num_rows", + "chart_diseases_translated", + ) + + missing = [key for key in required_keys if key not in context] + if missing: + missing_keys = ", ".join(missing) + raise KeyError(f"Missing context keys: {missing_keys}") + + prefix = TEMPLATE_PREFIX.replace("__LOGO_PATH__", logo_path).replace( + "__SIGNATURE_PATH__", signature_path + ) + + dynamic = ( + DYNAMIC_BLOCK.replace("__CLIENT_ROW__", context["client_row"]) + .replace("__CLIENT_DATA__", context["client_data"]) + .replace("__VACCINES_DUE_STR__", context["vaccines_due_agents_str"]) + .replace("__VACCINES_DUE_ARRAY__", context["vaccines_due_agents_array"]) + .replace("__RECEIVED__", context["received"]) + .replace("__NUM_ROWS__", context["num_rows"]) + .replace("__CHART_DISEASES_TRANSLATED__", context["chart_diseases_translated"]) + .replace("__SHOW_VALIDITY_MARKERS__", context.get("show_validity_markers", "false")) + ) + return prefix + dynamic diff --git a/templates/overdue_standard_v1/fr_template.py b/templates/overdue_standard_v1/fr_template.py new file mode 100644 index 0000000..60fc209 --- /dev/null +++ b/templates/overdue_standard_v1/fr_template.py @@ -0,0 +1,212 @@ +"""French Typst template renderer. + +This module contains the French version of the immunization notice template. The +template generates a 2025 immunization notice in Typst format for dynamic PDF +rendering. + +The template defines the notice layout in French, including client information, +immunization requirements, vaccine records, QR codes, and contact instructions. +All placeholder values (client data, dates, vaccines) are dynamically substituted +during rendering. + +Available placeholder variables include: +- client: Client data dict with person, school, board, contact info +- client_id: Unique client identifier +- immunizations_due: List of required vaccines +- qr_code: Optional QR code image path (if QR generation is enabled) +- date: Delivery/notice date +""" + +from __future__ import annotations + +from typing import Mapping + +TEMPLATE_PREFIX = """// --- CCEYA NOTICE TEMPLATE (TEST VERSION) --- // +// Description: A typst template that dynamically generates CCEYA templates. +// NOTE: All contact details are placeholders for testing purposes only. +// ----------------------------------------- // + +#import "/templates/conf.typ" + +// General document formatting +#set text(fill: black) +#set par(justify: false) +#set page("us-letter") + +// Formatting links - prevent URLs from splitting across lines +#show link: it => box(underline(it)) + +// Font formatting +#set text( + font: "FreeSans", + size: 10pt +) + +// Immunization Notice Section +#let immunization_notice(client, client_id, immunizations_due, date, font_size) = block[ + +#v(0.2cm) + +#conf.header_info_cim("__LOGO_PATH__", 6cm, black, 16pt, "Demande de dossier d'immunisation") + +#v(0.2cm) + +#conf.client_info_tbl_fr(equal_split: false, vline: false, client, client_id, font_size, "Centre de garde d'enfants", 81pt, border: false) + +#v(0.3cm) + +// Notice for immunizations +En date du *#date*, nos dossiers indiquent que votre enfant n'a pas reΓ§u les immunisations suivantes : + +#conf.client_immunization_list(immunizations_due) + +Veuillez examiner le dossier d'immunisation Γ  la page 2 et mettre Γ  jour le dossier de votre enfant en utilisant l'une des options suivantes : + +1. En visitant #text(fill:conf.linkcolor)[#link("https://www.test-immunization.ca")] +2. En envoyant un courriel Γ  #text(fill:conf.linkcolor)[#link("records@test-immunization.ca")] +3. En envoyant par la poste une photocopie du dossier d'immunisation de votre enfant Γ  Test Health, 123 Placeholder Street, Sample City, ON A1A 1A1 +4. Par tΓ©lΓ©phone : 555-555-5555 poste 1234 + +Veuillez informer la SantΓ© publique et votre centre de garde d'enfants chaque fois que votre enfant reΓ§oit un vaccin. + +#grid( + columns: (1fr, auto), + gutter: 10pt, + [*Si vous choisissez de ne pas immuniser votre enfant*, une exemption mΓ©dicale valide ou une dΓ©claration de conscience ou de croyance religieuse doit Γͺtre remplie et soumise Γ  la SantΓ© publique. Les liens vers ces formulaires se trouvent Γ  #text(fill:conf.wdgteal)[#link("https://www.test-immunization.ca/exemptions")]. Veuillez noter que cette exemption est uniquement pour la garde d'enfants et qu'une nouvelle exemption sera requise lors de l'inscription Γ  l'Γ©cole primaire.], + [#if "qr_img" in client [ + #if "qr_url" in client [ + #link(client.qr_url)[#image(client.qr_img, width: 3cm)] + ] else [ + #image(client.qr_img, width: 3cm) + ] + ]] +) + +En cas d'Γ©closion, les enfants qui ne sont pas adΓ©quatement immunisΓ©s peuvent Γͺtre exclus du centre de garde d'enfants. + +Si vous avez des questions sur les vaccins de votre enfant, veuillez appeler le 555-555-5555 poste 1234 pour parler Γ  une infirmiΓ¨re de la SantΓ© publique. + + SincΓ¨res salutations, + +#conf.signature("__SIGNATURE_PATH__", "Dr. Jane Smith, MPH", "MΓ©decin hygiΓ©niste adjoint") + +// Invisible marker for layout validation +#box(width: 0pt, height: 0pt)[ + #text(size: 0.1pt, fill: white)[MARK_END_SIGNATURE_BLOCK] +] + +] + +#let vaccine_table_page(client_id) = block[ + + #v(0.5cm) + + #grid( + + columns: (50%,50%), + gutter: 5%, + [#image("__LOGO_PATH__", width: 6cm)], + [#set align(center + bottom) + #text(size: 20.5pt, fill: black)[*Dossier d'immunisation*]] + +) + + #v(0.5cm) + + Pour votre rΓ©fΓ©rence, les immunisations enregistrΓ©es auprΓ¨s de la SantΓ© publique sont les suivantes : + +] + +#let end_of_immunization_notice() = [ + #set align(center) + Fin du dossier d'immunisation ] +""" + +DYNAMIC_BLOCK = """ +#let client_row = __CLIENT_ROW__ +#let data = __CLIENT_DATA__ +#let vaccines_due = __VACCINES_DUE_STR__ +#let vaccines_due_array = __VACCINES_DUE_ARRAY__ +#let received = __RECEIVED__ +#let num_rows = __NUM_ROWS__ +#let diseases = __CHART_DISEASES_TRANSLATED__ +#let show_validity_markers = __SHOW_VALIDITY_MARKERS__ +#let date = data.date_data_cutoff + +#set page( + margin: (top: 1cm, bottom: 2cm, left: 1.75cm, right: 2cm), + footer: align(center, context numbering("1 / " + str(counter(page).final().first()), counter(page).get().first())) +) + +#immunization_notice(data, client_row, vaccines_due_array, date, 11pt) +#pagebreak() +#vaccine_table_page(client_row.at(0)) +#conf.immunization-table(5, num_rows, received, diseases, 10.6pt, "fr", show_validity_markers) +#end_of_immunization_notice() +""" + + +def render_notice( + context: Mapping[str, str], + *, + logo_path: str, + signature_path: str, +) -> str: + """Render the Typst document for a single French notice. + + Parameters + ---------- + context : Mapping[str, str] + Dictionary containing template placeholder values. Must include: + - client_row: Row identifier + - client_data: Client information dict + - vaccines_due_str: Formatted string of vaccines due + - vaccines_due_array: Array of vaccines due + - received: Received vaccine data + - num_rows: Number of table rows + - chart_diseases_translated: Translated disease names for chart columns + + logo_path : str + Absolute path to logo image file + signature_path : str + Absolute path to signature image file + + Returns + ------- + str + Rendered Typst template with all placeholders replaced + + Raises + ------ + KeyError + If any required context keys are missing + """ + required_keys = ( + "client_row", + "client_data", + "vaccines_due_str", + "vaccines_due_array", + "received", + "num_rows", + "chart_diseases_translated", + ) + missing = [key for key in required_keys if key not in context] + if missing: + missing_keys = ", ".join(missing) + raise KeyError(f"Missing context keys: {missing_keys}") + + prefix = TEMPLATE_PREFIX.replace("__LOGO_PATH__", logo_path).replace( + "__SIGNATURE_PATH__", signature_path + ) + + dynamic = ( + DYNAMIC_BLOCK.replace("__CLIENT_ROW__", context["client_row"]) + .replace("__CLIENT_DATA__", context["client_data"]) + .replace("__VACCINES_DUE_STR__", context["vaccines_due_str"]) + .replace("__VACCINES_DUE_ARRAY__", context["vaccines_due_array"]) + .replace("__RECEIVED__", context["received"]) + .replace("__NUM_ROWS__", context["num_rows"]) + .replace("__CHART_DISEASES_TRANSLATED__", context["chart_diseases_translated"]) + .replace("__SHOW_VALIDITY_MARKERS__", context.get("show_validity_markers", "false")) + ) + return prefix + dynamic diff --git a/tests/fixtures/sample_input.py b/tests/fixtures/sample_input.py index 84c88e8..9168e65 100644 --- a/tests/fixtures/sample_input.py +++ b/tests/fixtures/sample_input.py @@ -221,6 +221,7 @@ def create_test_client_record( contact=contact_dict, vaccines_due=vaccines_due, vaccines_due_list=vaccines_due_list, + vaccines_due_agent_list=None, received=received, metadata={}, qr=None, @@ -357,6 +358,7 @@ def write_test_artifact( "contact": client.contact, "vaccines_due": client.vaccines_due, "vaccines_due_list": client.vaccines_due_list, + "vaccines_due_agent_list": client.vaccines_due_agent_list, "received": list(client.received) if client.received else [], "metadata": client.metadata, "qr": client.qr, diff --git a/tests/integration/test_error_propagation.py b/tests/integration/test_error_propagation.py index 6422c86..924ea42 100644 --- a/tests/integration/test_error_propagation.py +++ b/tests/integration/test_error_propagation.py @@ -54,6 +54,7 @@ def test_notice_generation_raises_on_language_mismatch(self, tmp_path): }, vaccines_due="", vaccines_due_list=[], + vaccines_due_agent_list=None, received=[], metadata={}, qr=None, @@ -109,6 +110,7 @@ def test_notice_generation_returns_all_or_nothing(self, tmp_path): }, vaccines_due="Polio", vaccines_due_list=["Polio"], + vaccines_due_agent_list=None, received=[], metadata={}, qr=None, @@ -132,6 +134,7 @@ def test_notice_generation_returns_all_or_nothing(self, tmp_path): }, vaccines_due="MMR", vaccines_due_list=["MMR"], + vaccines_due_agent_list=None, received=[], metadata={}, qr=None, diff --git a/tests/integration/test_pipeline_contracts.py b/tests/integration/test_pipeline_contracts.py index 2b39aa6..18788f3 100644 --- a/tests/integration/test_pipeline_contracts.py +++ b/tests/integration/test_pipeline_contracts.py @@ -255,7 +255,7 @@ def test_disease_alias_normalized_to_canonical_name( df = sample_input.create_test_input_dataframe(num_clients=1) df["overdue_disease"] = ["Poliomyelitis"] - result = preprocess.build_preprocess_result( + result, _ = preprocess.build_preprocess_result( df, language="en", vaccine_reference=default_vaccine_reference, @@ -303,7 +303,7 @@ def test_unknown_validity_warns_when_markers_enabled( df = sample_input.create_test_input_dataframe(num_clients=1) df["imms_given"] = ["May 1, 2020 - DTaP"] - result = preprocess.build_preprocess_result( + result, _ = preprocess.build_preprocess_result( df, language="en", vaccine_reference=default_vaccine_reference, @@ -403,7 +403,7 @@ def test_mixed_validity_with_markers_disabled_warns_and_succeeds( "Jun 15, 2021 - MMR", ] - result = preprocess.build_preprocess_result( + result, _ = preprocess.build_preprocess_result( df, language="en", vaccine_reference=default_vaccine_reference, @@ -451,7 +451,7 @@ def test_include_dose_formats_vaccines_due_list( df = sample_input.create_test_input_dataframe(num_clients=1) df["overdue_disease"] = ["DTaP - 2"] - result = preprocess.build_preprocess_result( + result, _ = preprocess.build_preprocess_result( df, language="en", vaccine_reference=default_vaccine_reference, @@ -510,7 +510,7 @@ def test_blank_dose_warns_and_displays_only_disease( df = sample_input.create_test_input_dataframe(num_clients=1) df["overdue_disease"] = ["Polio - "] - result = preprocess.build_preprocess_result( + result, _ = preprocess.build_preprocess_result( df, language="en", vaccine_reference=default_vaccine_reference, @@ -595,7 +595,7 @@ def test_phix_enabled_does_not_break_artifact_schema( output_dir=tmp_path, ) - result = preprocess.build_preprocess_result( + result, _ = preprocess.build_preprocess_result( enriched_df, language="en", vaccine_reference=default_vaccine_reference, @@ -636,7 +636,7 @@ def test_phix_disabled_produces_same_artifact_schema( Assertion: artifact from a non-enriched DataFrame has the same required keys and client count as the PHIX-enabled path. """ - result = preprocess.build_preprocess_result( + result, _ = preprocess.build_preprocess_result( normalized_test_df, language="en", vaccine_reference=default_vaccine_reference, diff --git a/tests/integration/test_translation_integration.py b/tests/integration/test_translation_integration.py index f4e9dcc..6926514 100644 --- a/tests/integration/test_translation_integration.py +++ b/tests/integration/test_translation_integration.py @@ -66,6 +66,7 @@ def test_build_template_context_translates_vaccines_due( }, vaccines_due="Polio, Measles", vaccines_due_list=["Polio", "Measles"], + vaccines_due_agent_list=None, received=None, metadata={}, ) @@ -113,6 +114,7 @@ def test_build_template_context_preserves_english( }, vaccines_due="Polio, Measles", vaccines_due_list=["Polio", "Measles"], + vaccines_due_agent_list=None, received=None, metadata={}, ) @@ -160,6 +162,7 @@ def test_build_template_context_translates_received_vaccines( }, vaccines_due=None, vaccines_due_list=None, + vaccines_due_agent_list=None, received=[ {"date_given": "2010-06-01", "vaccine": ["Polio", "Measles"]}, {"date_given": "2011-01-15", "vaccine": ["Tetanus"]}, @@ -217,6 +220,7 @@ def test_build_template_context_includes_formatted_date( }, vaccines_due=None, vaccines_due_list=None, + vaccines_due_agent_list=None, received=None, metadata={}, ) @@ -260,6 +264,7 @@ def test_build_template_context_includes_formatted_date( }, vaccines_due=None, vaccines_due_list=None, + vaccines_due_agent_list=None, received=None, metadata={}, ) diff --git a/tests/unit/test_assignment_manifest.py b/tests/unit/test_assignment_manifest.py new file mode 100644 index 0000000..0768345 --- /dev/null +++ b/tests/unit/test_assignment_manifest.py @@ -0,0 +1,323 @@ +"""Unit tests for pipeline/assignment_manifest.py.""" + +from __future__ import annotations + +import json +import io +from pathlib import Path +from unittest.mock import MagicMock, patch + +import pytest + +from pipeline.assignment_manifest import ( + ManifestRow, + ReconciliationResult, + has_errors, + load_manifest, + print_preflight_summary, + reconcile, +) +from pipeline.notice_versioning import NoticeKind, NoticeVersion, NoticeVersionCatalog + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +def _write_manifest(tmp_path: Path, rows: list) -> Path: + p = tmp_path / "assignments.json" + p.write_text(json.dumps(rows), encoding="utf-8") + return p + + +def _catalog(default_version: str = "overdue_standard_v1") -> NoticeVersionCatalog: + return NoticeVersionCatalog( + schema_version=1, + default_version=default_version, + default_language="en", + versions={ + "overdue_standard_v1": NoticeVersion( + version_id="overdue_standard_v1", kind=NoticeKind.OVERDUE, requires="has_overdue" + ), + "affirmative_schedule_v1": NoticeVersion( + version_id="affirmative_schedule_v1", kind=NoticeKind.AFFIRMATIVE, requires="no_overdue" + ), + }, + ) + + +def _client(client_id: str, vaccines_due=None) -> MagicMock: + m = MagicMock() + m.client_id = client_id + m.vaccines_due_list = vaccines_due + return m + + +def _row(client_id: str, version: str = "overdue_standard_v1", language: str | None = "en") -> dict: + return { + "client_id": client_id, + "notice_version": version, + "language": language, + } + + +def _empty_result(**overrides) -> ReconciliationResult: + defaults = dict( + counts_by_version={}, + counts_by_language={}, + missing_clients=[], + extra_rows=[], + duplicate_manifest_ids=[], + unknown_versions=[], + missing_language_clients=[], + eligibility_conflicts=[], + default_language="en", + ) + defaults.update(overrides) + return ReconciliationResult(**defaults) + + +# --------------------------------------------------------------------------- +# load_manifest +# --------------------------------------------------------------------------- + +@pytest.mark.unit +class TestLoadManifest: + def test_loads_valid_manifest(self, tmp_path: Path) -> None: + p = _write_manifest(tmp_path, [_row("C001"), _row("C002")]) + result = load_manifest(p) + assert "C001" in result + assert "C002" in result + assert result["C001"].notice_version == "overdue_standard_v1" + + def test_raises_on_non_list_json(self, tmp_path: Path) -> None: + p = tmp_path / "assignments.json" + p.write_text('{"client_id": "C001", "notice_version": "v1"}', encoding="utf-8") + with pytest.raises(ValueError, match="must be a JSON array"): + load_manifest(p) + + def test_raises_missing_client_id(self, tmp_path: Path) -> None: + p = _write_manifest(tmp_path, [{"notice_version": "overdue_standard_v1"}]) + with pytest.raises(ValueError, match="client_id"): + load_manifest(p) + + def test_raises_missing_notice_version(self, tmp_path: Path) -> None: + p = _write_manifest(tmp_path, [{"client_id": "C001"}]) + with pytest.raises(ValueError, match="notice_version"): + load_manifest(p) + + def test_raises_on_duplicate_client_ids(self, tmp_path: Path) -> None: + p = _write_manifest(tmp_path, [_row("C001"), _row("C001")]) + with pytest.raises(ValueError, match="duplicate client_id"): + load_manifest(p) + + def test_optional_fields_default_to_none(self, tmp_path: Path) -> None: + p = _write_manifest(tmp_path, [{"client_id": "C001", "notice_version": "v1"}]) + result = load_manifest(p) + assert result["C001"].language is None + assert result["C001"].experiment_id is None + assert result["C001"].experiment_arm is None + + def test_preserves_experiment_fields(self, tmp_path: Path) -> None: + rows = [{ + "client_id": "C001", + "notice_version": "v1", + "experiment_id": "exp_a", + "experiment_arm": "treatment", + }] + p = _write_manifest(tmp_path, rows) + result = load_manifest(p) + assert result["C001"].experiment_id == "exp_a" + assert result["C001"].experiment_arm == "treatment" + + def test_invalid_json_raises(self, tmp_path: Path) -> None: + p = tmp_path / "bad.json" + p.write_text("{not valid json", encoding="utf-8") + with pytest.raises(ValueError, match="not valid JSON"): + load_manifest(p) + + +# --------------------------------------------------------------------------- +# reconcile +# --------------------------------------------------------------------------- + +@pytest.mark.unit +class TestReconcile: + def test_happy_path_all_matched(self, tmp_path: Path) -> None: + clients = [_client("C001", ["Measles"]), _client("C002", ["Polio"])] + manifest = { + "C001": ManifestRow("C001", "overdue_standard_v1", "en", None, None), + "C002": ManifestRow("C002", "overdue_standard_v1", "fr", None, None), + } + result = reconcile(clients, manifest, _catalog(), False, "error") + assert result.missing_clients == [] + assert result.extra_rows == [] + assert result.unknown_versions == [] + assert result.eligibility_conflicts == [] + assert result.counts_by_language.get("en", 0) == 1 + assert result.counts_by_language.get("fr", 0) == 1 + + def test_detects_missing_clients(self) -> None: + clients = [_client("C001", ["Measles"]), _client("C002", ["Polio"])] + manifest = { + "C001": ManifestRow("C001", "overdue_standard_v1", "en", None, None), + } + result = reconcile(clients, manifest, _catalog(), allow_unassigned=False, extra_manifest_rows="error") + assert "C002" in result.missing_clients + + def test_allow_unassigned_uses_defaults(self) -> None: + clients = [_client("C001", ["Measles"]), _client("C002", ["Polio"])] + manifest = { + "C001": ManifestRow("C001", "overdue_standard_v1", "en", None, None), + } + result = reconcile(clients, manifest, _catalog(), allow_unassigned=True, extra_manifest_rows="error") + assert result.missing_clients == [] + # C002 resolved with catalog defaults (overdue_standard_v1, en) + assert result.counts_by_version.get("overdue_standard_v1 (en)", 0) >= 1 + + def test_detects_extra_rows(self) -> None: + clients = [_client("C001", ["Measles"])] + manifest = { + "C001": ManifestRow("C001", "overdue_standard_v1", "en", None, None), + "EXTRA": ManifestRow("EXTRA", "overdue_standard_v1", "en", None, None), + } + result = reconcile(clients, manifest, _catalog(), False, "warn") + assert "EXTRA" in result.extra_rows + + def test_detects_unknown_versions(self) -> None: + clients = [_client("C001", ["Measles"])] + manifest = { + "C001": ManifestRow("C001", "no_such_version", "en", None, None), + } + result = reconcile(clients, manifest, _catalog(), False, "error") + assert "no_such_version" in result.unknown_versions + # Client should not appear in counts + assert "C001" not in result.missing_clients + + def test_detects_missing_language_clients(self) -> None: + clients = [_client("C001", ["Measles"])] + manifest = { + "C001": ManifestRow("C001", "overdue_standard_v1", None, None, None), + } + result = reconcile(clients, manifest, _catalog(), False, "error") + assert "C001" in result.missing_language_clients + # Should still be counted with default language + assert result.counts_by_language.get("en", 0) == 1 + + def test_detects_eligibility_conflicts(self) -> None: + # Affirmative assigned but client has vaccines due + clients = [_client("C001", ["Measles"])] + manifest = { + "C001": ManifestRow("C001", "affirmative_schedule_v1", "en", None, None), + } + result = reconcile(clients, manifest, _catalog(), False, "error") + assert "C001" in result.eligibility_conflicts + + def test_duplicate_manifest_ids_always_empty(self) -> None: + # load_manifest raises on duplicates; reconcile always produces empty list + clients = [_client("C001", ["Measles"])] + manifest = { + "C001": ManifestRow("C001", "overdue_standard_v1", "en", None, None), + } + result = reconcile(clients, manifest, _catalog(), False, "error") + assert result.duplicate_manifest_ids == [] + + def test_counts_by_version_uses_composite_keys(self) -> None: + clients = [_client("C001", ["Measles"]), _client("C002", ["Polio"])] + manifest = { + "C001": ManifestRow("C001", "overdue_standard_v1", "en", None, None), + "C002": ManifestRow("C002", "overdue_standard_v1", "fr", None, None), + } + result = reconcile(clients, manifest, _catalog(), False, "error") + assert "overdue_standard_v1 (en)" in result.counts_by_version + assert "overdue_standard_v1 (fr)" in result.counts_by_version + + +# --------------------------------------------------------------------------- +# has_errors +# --------------------------------------------------------------------------- + +@pytest.mark.unit +class TestHasErrors: + def test_no_errors_returns_false(self) -> None: + assert not has_errors(_empty_result(), "error") + + def test_missing_clients_is_always_error(self) -> None: + result = _empty_result(missing_clients=["C001"]) + assert has_errors(result, "error") + assert has_errors(result, "warn") + + def test_unknown_versions_is_always_error(self) -> None: + result = _empty_result(unknown_versions=["bad_version"]) + assert has_errors(result, "error") + assert has_errors(result, "warn") + + def test_eligibility_conflicts_is_always_error(self) -> None: + result = _empty_result(eligibility_conflicts=["C001"]) + assert has_errors(result, "error") + assert has_errors(result, "warn") + + def test_extra_rows_respects_error_policy(self) -> None: + result = _empty_result(extra_rows=["EXTRA"]) + assert has_errors(result, "error") + assert not has_errors(result, "warn") + + def test_missing_language_clients_not_an_error(self) -> None: + result = _empty_result(missing_language_clients=["C001"]) + assert not has_errors(result, "error") + + +# --------------------------------------------------------------------------- +# print_preflight_summary +# --------------------------------------------------------------------------- + +@pytest.mark.unit +class TestPrintPreflightSummary: + def _capture(self, result: ReconciliationResult) -> str: + import io + buf = io.StringIO() + with patch("builtins.print", side_effect=lambda *args, **kw: buf.write(" ".join(str(a) for a in args) + "\n")): + print_preflight_summary(result) + return buf.getvalue() + + def test_output_contains_no_pii(self) -> None: + result = _empty_result( + counts_by_version={"overdue_standard_v1 (en)": 100}, + counts_by_language={"en": 100}, + ) + output = self._capture(result) + pii_candidates = ["John", "Jane", "Smith", "1990-01-01", "123 Main St"] + for pii in pii_candidates: + assert pii not in output + + def test_shows_assignment_mode(self) -> None: + output = self._capture(_empty_result()) + assert "manifest" in output + + def test_shows_version_language_counts(self) -> None: + result = _empty_result( + counts_by_version={ + "overdue_standard_v1 (en)": 500, + "overdue_standard_v1 (fr)": 125, + }, + counts_by_language={"en": 500, "fr": 125}, + ) + output = self._capture(result) + assert "overdue_standard_v1 (en)" in output + assert "500" in output + assert "overdue_standard_v1 (fr)" in output + assert "125" in output + + def test_shows_missing_language_with_default(self) -> None: + result = _empty_result( + missing_language_clients=["C001", "C002"], + default_language="fr", + ) + output = self._capture(result) + assert "2" in output + assert "fr" in output + + def test_shows_zero_counts_for_clean_run(self) -> None: + output = self._capture(_empty_result()) + assert "Missing clients" in output + assert "0" in output diff --git a/tests/unit/test_data_models.py b/tests/unit/test_data_models.py index cf562cb..0ec0c8a 100644 --- a/tests/unit/test_data_models.py +++ b/tests/unit/test_data_models.py @@ -38,6 +38,7 @@ def test_client_record_language_must_be_valid_enum_value(self) -> None: contact={}, vaccines_due=None, vaccines_due_list=None, + vaccines_due_agent_list=None, received=None, metadata={}, ) @@ -55,6 +56,7 @@ def test_client_record_language_must_be_valid_enum_value(self) -> None: contact={}, vaccines_due=None, vaccines_due_list=None, + vaccines_due_agent_list=None, received=None, metadata={}, ) @@ -90,6 +92,7 @@ def test_client_record_invalid_language_rejected_by_enum_validation( contact={}, vaccines_due=None, vaccines_due_list=None, + vaccines_due_agent_list=None, received=None, metadata={}, ) diff --git a/tests/unit/test_fr_template.py b/tests/unit/test_fr_template.py index 0959f62..3c53081 100644 --- a/tests/unit/test_fr_template.py +++ b/tests/unit/test_fr_template.py @@ -126,15 +126,7 @@ def test_render_notice_substitutes_logo_path(self) -> None: - Logo path must match actual file location - Output Typst must reference correct logo path """ - context = { - "client_row": "()", - "client_data": "{}", - "vaccines_due_str": '""', - "vaccines_due_array": "()", - "received": "()", - "num_rows": "0", - "chart_diseases_translated": '("DiphtΓ©rie", "TΓ©tanos", "Coqueluche")', - } + context = _valid_context() logo_path = "/custom/logo/path.png" result = render_notice( @@ -152,15 +144,7 @@ def test_render_notice_substitutes_signature_path(self) -> None: - Signature path must match actual file location - Output Typst must reference correct signature path """ - context = { - "client_row": "()", - "client_data": "{}", - "vaccines_due_str": '""', - "vaccines_due_array": "()", - "received": "()", - "num_rows": "0", - "chart_diseases_translated": '("DiphtΓ©rie", "TΓ©tanos", "Coqueluche")', - } + context = _valid_context() signature_path = "/custom/signature.png" result = render_notice( @@ -178,15 +162,7 @@ def test_render_notice_includes_template_prefix(self) -> None: - Typst setup code must be included - Import statement for conf.typ is required """ - context = { - "client_row": "()", - "client_data": "{}", - "vaccines_due_str": '""', - "vaccines_due_array": "()", - "received": "()", - "num_rows": "0", - "chart_diseases_translated": '("DiphtΓ©rie", "TΓ©tanos", "Coqueluche")', - } + context = _valid_context() result = render_notice( context, @@ -260,15 +236,7 @@ def test_render_notice_empty_vaccines_handled(self) -> None: - Child might have all required vaccines - Template must handle empty vaccines_due_array """ - context = { - "client_row": "()", - "client_data": "{}", - "vaccines_due_str": '""', - "vaccines_due_array": "()", - "received": "()", - "num_rows": "0", - "chart_diseases_translated": '("DiphtΓ©rie", "TΓ©tanos", "Coqueluche")', - } + context = _valid_context() result = render_notice( context, @@ -287,15 +255,7 @@ def test_render_notice_french_content(self) -> None: - Output must be in French for French-language processing - Key terms like "Dossier d'immunisation" must appear """ - context = { - "client_row": "()", - "client_data": "{}", - "vaccines_due_str": '""', - "vaccines_due_array": "()", - "received": "()", - "num_rows": "0", - "chart_diseases_translated": '("DiphtΓ©rie", "TΓ©tanos", "Coqueluche")', - } + context = _valid_context() result = render_notice( context, diff --git a/tests/unit/test_generate_notices.py b/tests/unit/test_generate_notices.py index 55683b5..b3011c9 100644 --- a/tests/unit/test_generate_notices.py +++ b/tests/unit/test_generate_notices.py @@ -815,3 +815,214 @@ def test_render_notice_french_client(self, tmp_test_dir: Path) -> None: generate_notices.Language.FRENCH, renderers ) assert french_renderer is not None + + +# --------------------------------------------------------------------------- +# build_template_registry (manifest mode) +# --------------------------------------------------------------------------- + +@pytest.mark.unit +class TestBuildTemplateRegistry: + """Unit tests for build_template_registry().""" + + def _make_template(self, directory: Path, lang: str) -> None: + directory.mkdir(parents=True, exist_ok=True) + src = Path(__file__).parent.parent.parent / "templates" / f"{lang}_template.py" + import shutil + shutil.copy2(src, directory / f"{lang}_template.py") + + def test_builds_registry_for_needed_pairs(self, tmp_path: Path) -> None: + version_dir = tmp_path / "overdue_standard_v1" + self._make_template(version_dir, "en") + registry = generate_notices.build_template_registry( + tmp_path, {("overdue_standard_v1", "en")} + ) + assert ("overdue_standard_v1", "en") in registry + assert callable(registry[("overdue_standard_v1", "en")]) + + def test_raises_for_missing_template_path(self, tmp_path: Path) -> None: + """Fails at preflight listing all missing paths β€” not a per-client error.""" + # Create one but not the other + version_dir = tmp_path / "overdue_standard_v1" + self._make_template(version_dir, "en") + needed = { + ("overdue_standard_v1", "en"), + ("affirmative_schedule_v1", "en"), # missing + } + with pytest.raises(FileNotFoundError, match="affirmative_schedule_v1"): + generate_notices.build_template_registry(tmp_path, needed) + + def test_raises_listing_all_missing_not_just_first(self, tmp_path: Path) -> None: + needed = { + ("overdue_standard_v1", "en"), # missing + ("affirmative_schedule_v1", "fr"), # also missing + } + with pytest.raises(FileNotFoundError) as exc_info: + generate_notices.build_template_registry(tmp_path, needed) + msg = str(exc_info.value) + assert "overdue_standard_v1" in msg + assert "affirmative_schedule_v1" in msg + + def test_no_fallback_when_template_dir_set(self, tmp_path: Path) -> None: + """PHU dir specified: no fallback to templates/ for missing version subdir.""" + phu_dir = tmp_path / "phu" + phu_dir.mkdir() + # Only en available in phu dir; fr is NOT available + phu_version_dir = phu_dir / "overdue_standard_v1" + self._make_template(phu_version_dir, "en") + + # Request fr β€” should fail even though templates/ might have it + with pytest.raises(FileNotFoundError): + generate_notices.build_template_registry( + phu_dir, {("overdue_standard_v1", "fr")} + ) + + def test_fixed_mode_uses_flat_layout_no_regression(self, tmp_path: Path) -> None: + """Fixed mode (build_language_renderers) still works flat β€” no regression.""" + templates_dir = Path(__file__).parent.parent.parent / "templates" + renderers = generate_notices.build_language_renderers(templates_dir) + assert "en" in renderers + assert "fr" in renderers + assert callable(renderers["en"]) + + +# --------------------------------------------------------------------------- +# generate_typst_files β€” manifest mode branch +# --------------------------------------------------------------------------- + + +@pytest.mark.unit +class TestGenerateTypstFilesManifestMode: + """Unit tests for the manifest-mode branch of generate_typst_files(). + + Manifest mode is triggered when at least one client has + ``metadata["resolved_notice"]`` set. It uses build_template_registry() + for a preflight check and then dispatches each client to the renderer + identified by its (notice_version, language) pair. + + Fixed mode is already covered by integration tests; these tests focus + exclusively on the manifest branch. + """ + + _ASSETS = Path(__file__).parent.parent.parent / "templates" / "assets" + _LOGO = _ASSETS / "logo.png" + _SIG = _ASSETS / "signature.png" + + def _make_template(self, directory: Path, lang: str) -> None: + """Copy a real template into a versioned subdirectory.""" + import shutil + + directory.mkdir(parents=True, exist_ok=True) + src = Path(__file__).parent.parent.parent / "templates" / f"{lang}_template.py" + shutil.copy2(src, directory / f"{lang}_template.py") + + def _make_manifest_payload( + self, + clients: list, + ) -> "generate_notices.ArtifactPayload": + """Wrap a client list in a minimal ArtifactPayload.""" + from pipeline import data_models + + return data_models.ArtifactPayload( + run_id="test_manifest_run", + language="en", + clients=clients, + warnings=[], + created_at="2025-01-01T00:00:00Z", + total_clients=len(clients), + assignment_mode="manifest", + ) + + def _make_client( + self, + sequence: str, + client_id: str, + notice_version: str, + lang: str = "en", + ) -> "generate_notices.ClientRecord": + """Create a ClientRecord with resolved_notice metadata.""" + from dataclasses import replace + + base = sample_input.create_test_client_record( + sequence=sequence, client_id=client_id, language=lang + ) + return replace( + base, + metadata={ + "resolved_notice": { + "notice_version": notice_version, + "language": lang, + } + }, + ) + + def test_manifest_mode_writes_typ_files_for_each_client( + self, tmp_path: Path + ) -> None: + """Happy path: manifest clients produce correctly named .typ files. + + Real-world significance: + - generate_typst_files must enter manifest mode when resolved_notice is + present and dispatch each client through the registry renderer + - Filename convention (``{lang}_notice_{seq}_{id}.typ``) is consumed by + compile_notices; a mismatch silently drops PDFs + + Assertion: one .typ file per client, named with the manifest convention + """ + if not self._LOGO.exists() or not self._SIG.exists(): + pytest.skip("Template assets not available") + + version_dir = tmp_path / "overdue_standard_v1" + self._make_template(version_dir, "en") + + clients = [ + self._make_client("00001", "C001", "overdue_standard_v1"), + self._make_client("00002", "C002", "overdue_standard_v1"), + ] + payload = self._make_manifest_payload(clients) + + files = generate_notices.generate_typst_files( + payload, tmp_path, self._LOGO, self._SIG, tmp_path + ) + + assert len(files) == 2 + names = {f.name for f in files} + assert "en_notice_00001_C001.typ" in names + assert "en_notice_00002_C002.typ" in names + for f in files: + assert f.exists() + + def test_manifest_mode_preflight_raises_before_writing_any_file( + self, tmp_path: Path + ) -> None: + """Missing template version raises FileNotFoundError before any output is written. + + Real-world significance: + - Preflight failure must be all-or-nothing: if one version is missing the + run should abort cleanly rather than producing a partial batch of notices + - This protects against silently generating the wrong template for some + clients while skipping others + + Assertion: FileNotFoundError raised; typst output directory is empty + """ + if not self._LOGO.exists() or not self._SIG.exists(): + pytest.skip("Template assets not available") + + # Only provide one version; client references a second that doesn't exist + good_dir = tmp_path / "overdue_standard_v1" + self._make_template(good_dir, "en") + + clients = [ + self._make_client("00001", "C001", "overdue_standard_v1"), + self._make_client("00002", "C002", "missing_version_v1"), + ] + payload = self._make_manifest_payload(clients) + + with pytest.raises(FileNotFoundError, match="missing_version_v1"): + generate_notices.generate_typst_files( + payload, tmp_path, self._LOGO, self._SIG, tmp_path + ) + + typst_dir = tmp_path / "typst" + written = list(typst_dir.glob("*.typ")) if typst_dir.exists() else [] + assert written == [], "No .typ files should be written when preflight fails" diff --git a/tests/unit/test_notice_versioning.py b/tests/unit/test_notice_versioning.py new file mode 100644 index 0000000..fb8a689 --- /dev/null +++ b/tests/unit/test_notice_versioning.py @@ -0,0 +1,322 @@ +"""Unit tests for pipeline/notice_versioning.py.""" + +from __future__ import annotations + +from pathlib import Path +from unittest.mock import MagicMock + +import pytest +import yaml + +from pipeline.notice_versioning import ( + ELIGIBILITY_RULES, + NoticeKind, + NoticeVersion, + NoticeVersionCatalog, + ResolvedNotice, + load_catalog, + validate_eligibility, +) + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +def _write_catalog(tmp_path: Path, content: dict) -> Path: + p = tmp_path / "notice_versions.yaml" + p.write_text(yaml.dump(content), encoding="utf-8") + return tmp_path + + +def _make_catalog() -> NoticeVersionCatalog: + return NoticeVersionCatalog( + schema_version=1, + default_version="overdue_standard_v1", + default_language="en", + versions={ + "overdue_standard_v1": NoticeVersion( + version_id="overdue_standard_v1", kind=NoticeKind.OVERDUE, requires="has_overdue" + ), + "affirmative_schedule_v1": NoticeVersion( + version_id="affirmative_schedule_v1", kind=NoticeKind.AFFIRMATIVE, requires="no_overdue" + ), + "info_v1": NoticeVersion( + version_id="info_v1", kind=NoticeKind.INFORMATIONAL, requires="any" + ), + }, + ) + + +def _resolved(kind: str, version: str = "overdue_standard_v1") -> ResolvedNotice: + return ResolvedNotice( + notice_version=version, + notice_kind=kind, + language="en", + experiment_id=None, + experiment_arm=None, + assignment_source="manifest", + ) + + +def _client(vaccines_due_list): + m = MagicMock() + m.client_id = "C001" + m.vaccines_due_list = vaccines_due_list + return m + + +# --------------------------------------------------------------------------- +# load_catalog +# --------------------------------------------------------------------------- + +@pytest.mark.unit +class TestLoadCatalog: + def test_returns_none_when_file_absent(self, tmp_path: Path) -> None: + result = load_catalog(tmp_path) + assert result is None + + def test_loads_valid_catalog(self, tmp_path: Path) -> None: + _write_catalog(tmp_path, { + "schema_version": 1, + "default_version": "overdue_standard_v1", + "default_language": "en", + "versions": { + "overdue_standard_v1": {"kind": "overdue"}, + "affirmative_schedule_v1": {"kind": "affirmative"}, + }, + }) + catalog = load_catalog(tmp_path) + assert catalog is not None + assert catalog.schema_version == 1 + assert catalog.default_version == "overdue_standard_v1" + assert catalog.default_language == "en" + assert "overdue_standard_v1" in catalog.versions + assert catalog.versions["overdue_standard_v1"].kind == NoticeKind.OVERDUE + + def test_raises_on_invalid_yaml(self, tmp_path: Path) -> None: + (tmp_path / "notice_versions.yaml").write_text( + "key: [unclosed", encoding="utf-8" + ) + with pytest.raises(ValueError, match="invalid YAML"): + load_catalog(tmp_path) + + def test_raises_missing_schema_version(self, tmp_path: Path) -> None: + _write_catalog(tmp_path, { + "default_version": "v1", + "default_language": "en", + "versions": {"v1": {"kind": "overdue"}}, + }) + with pytest.raises(ValueError, match="schema_version"): + load_catalog(tmp_path) + + def test_raises_missing_default_version(self, tmp_path: Path) -> None: + _write_catalog(tmp_path, { + "schema_version": 1, + "default_language": "en", + "versions": {"v1": {"kind": "overdue"}}, + }) + with pytest.raises(ValueError, match="default_version"): + load_catalog(tmp_path) + + def test_raises_empty_default_language(self, tmp_path: Path) -> None: + _write_catalog(tmp_path, { + "schema_version": 1, + "default_version": "v1", + "default_language": "", + "versions": {"v1": {"kind": "overdue"}}, + }) + with pytest.raises(ValueError, match="default_language"): + load_catalog(tmp_path) + + def test_raises_unknown_kind(self, tmp_path: Path) -> None: + _write_catalog(tmp_path, { + "schema_version": 1, + "default_version": "v1", + "default_language": "en", + "versions": {"v1": {"kind": "unknown_kind"}}, + }) + with pytest.raises(ValueError, match="invalid kind"): + load_catalog(tmp_path) + + def test_raises_default_version_not_in_versions(self, tmp_path: Path) -> None: + _write_catalog(tmp_path, { + "schema_version": 1, + "default_version": "missing_version", + "default_language": "en", + "versions": {"v1": {"kind": "overdue"}}, + }) + with pytest.raises(ValueError, match="default_version.*not in versions"): + load_catalog(tmp_path) + + def test_raises_empty_version_id(self, tmp_path: Path) -> None: + p = tmp_path / "notice_versions.yaml" + # Write raw YAML with an empty-string key manually + p.write_text( + "schema_version: 1\ndefault_version: ''\ndefault_language: en\n" + "versions:\n '': {kind: overdue}\n", + encoding="utf-8", + ) + with pytest.raises(ValueError): + load_catalog(tmp_path) + + def test_all_notice_kinds_accepted(self, tmp_path: Path) -> None: + _write_catalog(tmp_path, { + "schema_version": 1, + "default_version": "overdue_v1", + "default_language": "fr", + "versions": { + "overdue_v1": {"kind": "overdue"}, + "affirmative_v1": {"kind": "affirmative"}, + "informational_v1": {"kind": "informational"}, + }, + }) + catalog = load_catalog(tmp_path) + assert catalog is not None + assert catalog.versions["overdue_v1"].kind == NoticeKind.OVERDUE + assert catalog.versions["affirmative_v1"].kind == NoticeKind.AFFIRMATIVE + assert catalog.versions["informational_v1"].kind == NoticeKind.INFORMATIONAL + + def test_explicit_requires_field_is_loaded(self, tmp_path: Path) -> None: + _write_catalog(tmp_path, { + "schema_version": 1, + "default_version": "v1", + "default_language": "en", + "versions": {"v1": {"kind": "overdue", "requires": "any"}}, + }) + catalog = load_catalog(tmp_path) + assert catalog is not None + assert catalog.versions["v1"].requires == "any" + + def test_omitted_requires_falls_back_to_kind_default(self, tmp_path: Path) -> None: + _write_catalog(tmp_path, { + "schema_version": 1, + "default_version": "overdue_v1", + "default_language": "en", + "versions": { + "overdue_v1": {"kind": "overdue"}, + "affirmative_v1": {"kind": "affirmative"}, + "info_v1": {"kind": "informational"}, + }, + }) + catalog = load_catalog(tmp_path) + assert catalog is not None + assert catalog.versions["overdue_v1"].requires == "has_overdue" + assert catalog.versions["affirmative_v1"].requires == "no_overdue" + assert catalog.versions["info_v1"].requires == "any" + + def test_raises_on_unknown_requires_value(self, tmp_path: Path) -> None: + _write_catalog(tmp_path, { + "schema_version": 1, + "default_version": "v1", + "default_language": "en", + "versions": {"v1": {"kind": "overdue", "requires": "no_such_rule"}}, + }) + with pytest.raises(ValueError, match="unknown requires"): + load_catalog(tmp_path) + + +# --------------------------------------------------------------------------- +# validate_eligibility +# --------------------------------------------------------------------------- + +@pytest.mark.unit +class TestValidateEligibility: + def test_overdue_passes_with_vaccines_due(self) -> None: + client = _client(["Measles", "Polio"]) + resolved = _resolved(NoticeKind.OVERDUE) + validate_eligibility(client, resolved, _make_catalog()) + + def test_overdue_raises_with_none_vaccines_due(self) -> None: + client = _client(None) + resolved = _resolved(NoticeKind.OVERDUE) + with pytest.raises(ValueError, match="C001"): + validate_eligibility(client, resolved, _make_catalog()) + + def test_overdue_raises_with_empty_vaccines_due(self) -> None: + client = _client([]) + resolved = _resolved(NoticeKind.OVERDUE) + with pytest.raises(ValueError, match="C001"): + validate_eligibility(client, resolved, _make_catalog()) + + def test_affirmative_passes_with_no_vaccines_due(self) -> None: + client = _client(None) + resolved = _resolved(NoticeKind.AFFIRMATIVE, "affirmative_schedule_v1") + validate_eligibility(client, resolved, _make_catalog()) + + def test_affirmative_passes_with_empty_vaccines_due(self) -> None: + client = _client([]) + resolved = _resolved(NoticeKind.AFFIRMATIVE, "affirmative_schedule_v1") + validate_eligibility(client, resolved, _make_catalog()) + + def test_affirmative_raises_with_non_empty_vaccines_due(self) -> None: + client = _client(["Measles"]) + resolved = _resolved(NoticeKind.AFFIRMATIVE, "affirmative_schedule_v1") + with pytest.raises(ValueError, match="C001"): + validate_eligibility(client, resolved, _make_catalog()) + + def test_informational_passes_regardless_of_vaccines_due(self) -> None: + client_with = _client(["Measles"]) + client_without = _client(None) + resolved = _resolved(NoticeKind.INFORMATIONAL, "info_v1") + validate_eligibility(client_with, resolved, _make_catalog()) + validate_eligibility(client_without, resolved, _make_catalog()) + + def test_error_message_contains_client_id_not_name(self) -> None: + client = _client([]) + client.client_id = "SENSITIVE_CLIENT_001" + client.full_name = "John Smith" + resolved = _resolved(NoticeKind.OVERDUE) + with pytest.raises(ValueError) as exc_info: + validate_eligibility(client, resolved, _make_catalog()) + assert "SENSITIVE_CLIENT_001" in str(exc_info.value) + assert "John Smith" not in str(exc_info.value) + + def test_error_message_includes_rule_name(self) -> None: + client = _client([]) + resolved = _resolved(NoticeKind.OVERDUE) + with pytest.raises(ValueError, match="has_overdue"): + validate_eligibility(client, resolved, _make_catalog()) + + def test_explicit_any_rule_overrides_kind_default(self) -> None: + # An overdue-kind version with requires=any accepts clients with no vaccines due. + catalog = NoticeVersionCatalog( + schema_version=1, + default_version="overdue_open_v1", + default_language="en", + versions={ + "overdue_open_v1": NoticeVersion( + version_id="overdue_open_v1", kind=NoticeKind.OVERDUE, requires="any" + ), + }, + ) + client = _client(None) + resolved = _resolved(NoticeKind.OVERDUE, "overdue_open_v1") + validate_eligibility(client, resolved, catalog) # should not raise + + +# --------------------------------------------------------------------------- +# ELIGIBILITY_RULES registry +# --------------------------------------------------------------------------- + +@pytest.mark.unit +class TestEligibilityRules: + def test_has_overdue_true_when_list_non_empty(self) -> None: + client = _client(["Measles"]) + assert ELIGIBILITY_RULES["has_overdue"](client) is True + + def test_has_overdue_false_when_list_empty(self) -> None: + assert ELIGIBILITY_RULES["has_overdue"](_client([])) is False + assert ELIGIBILITY_RULES["has_overdue"](_client(None)) is False + + def test_no_overdue_true_when_list_empty(self) -> None: + assert ELIGIBILITY_RULES["no_overdue"](_client([])) is True + assert ELIGIBILITY_RULES["no_overdue"](_client(None)) is True + + def test_no_overdue_false_when_list_non_empty(self) -> None: + assert ELIGIBILITY_RULES["no_overdue"](_client(["Polio"])) is False + + def test_any_always_true(self) -> None: + assert ELIGIBILITY_RULES["any"](_client(["Measles"])) is True + assert ELIGIBILITY_RULES["any"](_client(None)) is True + assert ELIGIBILITY_RULES["any"](_client([])) is True diff --git a/tests/unit/test_orchestrator.py b/tests/unit/test_orchestrator.py index 171bf82..da7e17d 100644 --- a/tests/unit/test_orchestrator.py +++ b/tests/unit/test_orchestrator.py @@ -127,11 +127,88 @@ def test_validate_args_existing_input_file(self, tmp_test_dir: Path) -> None: args = MagicMock() args.input_file = "students.xlsx" args.input_dir = tmp_test_dir + args.notice_assignments = None + args.language = "en" args.template_dir = None # Use default templates # Should not raise orchestrator.validate_args(args) + def test_language_required_in_fixed_mode(self, tmp_test_dir: Path) -> None: + """language is required when --notice-assignments is not provided.""" + test_file = tmp_test_dir / "students.xlsx" + test_file.write_text("test") + + args = MagicMock() + args.input_file = "students.xlsx" + args.input_dir = tmp_test_dir + args.notice_assignments = None + args.language = None + + with pytest.raises(ValueError, match="language is required"): + orchestrator.validate_args(args) + + def test_language_warned_and_cleared_in_manifest_mode(self, tmp_path: Path) -> None: + """When both language and --notice-assignments are provided, language is cleared.""" + xlsx = tmp_path / "students.xlsx" + xlsx.write_text("test") + manifest = tmp_path / "assignments.json" + manifest.write_text("[]") + catalog = tmp_path / "notice_versions.yaml" + catalog.write_text("schema_version: 1\n") + + args = MagicMock() + args.input_file = "students.xlsx" + args.input_dir = tmp_path + args.notice_assignments = manifest + args.language = "en" + args.config_dir = tmp_path + args.template_dir = None + + import io + with patch("builtins.print") as mock_print: + orchestrator.validate_args(args) + + assert args.language is None + # Warning must have been printed + printed = " ".join(str(c) for call in mock_print.call_args_list for c in call.args) + assert "Warning" in printed or "ignored" in printed.lower() + + def test_notice_assignments_missing_manifest_file(self, tmp_path: Path) -> None: + """Missing manifest file raises FileNotFoundError before Step 1.""" + xlsx = tmp_path / "students.xlsx" + xlsx.write_text("test") + (tmp_path / "notice_versions.yaml").write_text("schema_version: 1\n") + + args = MagicMock() + args.input_file = "students.xlsx" + args.input_dir = tmp_path + args.notice_assignments = tmp_path / "missing_manifest.json" + args.language = None + args.config_dir = tmp_path + args.template_dir = None + + with pytest.raises(FileNotFoundError, match="manifest"): + orchestrator.validate_args(args) + + def test_notice_assignments_missing_catalog_raises(self, tmp_path: Path) -> None: + """Missing notice_versions.yaml raises ValueError before Step 1.""" + xlsx = tmp_path / "students.xlsx" + xlsx.write_text("test") + manifest = tmp_path / "assignments.json" + manifest.write_text("[]") + + args = MagicMock() + args.input_file = "students.xlsx" + args.input_dir = tmp_path + args.notice_assignments = manifest + args.language = None + args.config_dir = tmp_path # no notice_versions.yaml here + args.template_dir = None + + with pytest.raises(ValueError, match="notice_versions.yaml"): + orchestrator.validate_args(args) + @pytest.mark.unit class TestPrintFunctions: @@ -216,7 +293,7 @@ def test_run_step_2_passes_selected_config_path(self, tmp_path: Path) -> None: - The --config option must control include_dose and validity handling - Step 2 remains independently rerunnable from its disk inputs """ - result = MagicMock(clients=[], warnings=[]) + preprocess_result = MagicMock(clients=[], warnings=[]) config_dir = tmp_path / "selected-config" with ( @@ -243,7 +320,8 @@ def test_run_step_2_passes_selected_config_path(self, tmp_path: Path) -> None: ) as mock_check_client_info, patch( "pipeline.orchestrator.preprocess.build_preprocess_result", - return_value=result, + # build_preprocess_result now returns (PreprocessResult, Optional[ReconciliationResult]) + return_value=(preprocess_result, None), ) as mock_build_result, patch( "pipeline.orchestrator.preprocess.write_artifact", @@ -251,7 +329,7 @@ def test_run_step_2_passes_selected_config_path(self, tmp_path: Path) -> None: ), patch("builtins.print"), ): - total_clients = orchestrator.run_step_2_preprocess( + total_clients, reconciliation_result = orchestrator.run_step_2_preprocess( input_dir=tmp_path, input_file="students.xlsx", output_dir=tmp_path / "output", @@ -261,6 +339,7 @@ def test_run_step_2_passes_selected_config_path(self, tmp_path: Path) -> None: ) assert total_clients == 0 + assert reconciliation_result is None assert mock_build_result.call_args.args[0] is mock_check_client_info.return_value assert mock_build_result.call_args.kwargs["config_path"] == ( @@ -428,6 +507,7 @@ def test_user_cancel_returns_exit_code_2(self, tmp_path: Path) -> None: output_dir=tmp_path / "output", config_dir=tmp_path / "config", template_dir=None, + notice_assignments=None, ) exit_code = orchestrator.main() diff --git a/tests/unit/test_preprocess.py b/tests/unit/test_preprocess.py index aa00597..6cdb45d 100644 --- a/tests/unit/test_preprocess.py +++ b/tests/unit/test_preprocess.py @@ -29,28 +29,6 @@ from tests.fixtures import sample_input -def _make_conforming_df(**overrides) -> pd.DataFrame: - """Build a minimal DataFrame with all required input columns.""" - row = { - "school_name": ["Test School"], - "client_id": ["C001"], - "first_name": ["Alice"], - "last_name": ["Zephyr"], - "date_of_birth": ["2015-01-01"], - "street_address_line_1": ["123 Main St"], - "street_address_line_2": [""], - "city": ["Guelph"], - "province": ["ON"], - "postal_code": ["N1H 2T2"], - "overdue_disease": ["Measles"], - "overdue_agent": ["MMR"], - "imms_given": [""], - } - row.update(overrides) - return pd.DataFrame(row) - - - @pytest.mark.unit class TestFormatVaccineDueList: """Unit tests for overdue-vaccine dose formatting.""" @@ -349,7 +327,7 @@ def test_build_result_generates_clients_with_sequences( """ df = sample_input.create_test_input_dataframe(num_clients=3) - result = preprocess.build_preprocess_result( + result, _ = preprocess.build_preprocess_result( df, language="en", vaccine_reference=default_vaccine_reference, @@ -373,14 +351,14 @@ def test_build_result_sorts_clients_deterministically( """ df = sample_input.create_test_input_dataframe(num_clients=3) - result1 = preprocess.build_preprocess_result( + result1, _ = preprocess.build_preprocess_result( df, language="en", vaccine_reference=default_vaccine_reference, replace_unspecified=[], ) - result2 = preprocess.build_preprocess_result( + result2, _ = preprocess.build_preprocess_result( df, language="en", vaccine_reference=default_vaccine_reference, @@ -418,7 +396,7 @@ def test_build_result_sorts_by_school_then_name( "imms_given": ["", "", "", ""], } ) - result = preprocess.build_preprocess_result( + result, _ = preprocess.build_preprocess_result( df, language="en", vaccine_reference=default_vaccine_reference, @@ -443,7 +421,7 @@ def test_build_result_maps_vaccines_correctly( df = sample_input.create_test_input_dataframe(num_clients=1) df["imms_given"] = ["May 1, 2020 - DTaP"] - result = preprocess.build_preprocess_result( + result, _ = preprocess.build_preprocess_result( df, language="en", vaccine_reference=default_vaccine_reference, @@ -484,7 +462,7 @@ def test_build_result_uses_explicit_config_path( df["overdue_disease"] = ["DTaP - 2"] df["imms_given"] = ["May 1, 2020 - DTaP"] - result = preprocess.build_preprocess_result( + result, _ = preprocess.build_preprocess_result( df, language="en", vaccine_reference=default_vaccine_reference, @@ -525,7 +503,7 @@ def test_build_result_handles_missing_board_name_with_warning( "imms_given": [""], } ) - result = preprocess.build_preprocess_result( + result, _ = preprocess.build_preprocess_result( df, language="en", vaccine_reference=default_vaccine_reference, @@ -548,7 +526,7 @@ def test_build_result_french_language_support( """ df = sample_input.create_test_input_dataframe(num_clients=1, language="fr") - result = preprocess.build_preprocess_result( + result, _ = preprocess.build_preprocess_result( df, language="fr", vaccine_reference=default_vaccine_reference, @@ -569,7 +547,7 @@ def test_build_result_handles_replace_unspecified( """ df = sample_input.create_test_input_dataframe(num_clients=1) - result = preprocess.build_preprocess_result( + result, _ = preprocess.build_preprocess_result( df, language="en", vaccine_reference=default_vaccine_reference, @@ -593,7 +571,7 @@ def test_build_result_detects_duplicate_client_ids( df.loc[0, "client_id"] = "C123456789" df.loc[1, "client_id"] = "C123456789" - result = preprocess.build_preprocess_result( + result, _ = preprocess.build_preprocess_result( df, language="en", vaccine_reference=default_vaccine_reference, @@ -627,7 +605,7 @@ def test_build_result_detects_multiple_duplicate_client_ids( df.loc[3, "client_id"] = "C222222222" df.loc[4, "client_id"] = "C222222222" - result = preprocess.build_preprocess_result( + result, _ = preprocess.build_preprocess_result( df, language="en", vaccine_reference=default_vaccine_reference, @@ -658,7 +636,7 @@ def test_build_result_no_warning_for_unique_client_ids( """ df = sample_input.create_test_input_dataframe(num_clients=3) - result = preprocess.build_preprocess_result( + result, _ = preprocess.build_preprocess_result( df, language="en", vaccine_reference=default_vaccine_reference, @@ -1242,7 +1220,7 @@ def test_build_result_maps_vaccines_correctly(self, default_vaccine_reference) - df = sample_input.create_test_input_dataframe(num_clients=1) df["imms_given"] = ["May 1, 2020 - DTaP"] - result = preprocess.build_preprocess_result( + result, _ = preprocess.build_preprocess_result( df, language="en", vaccine_reference=default_vaccine_reference, @@ -1256,17 +1234,735 @@ def test_build_result_maps_vaccines_correctly(self, default_vaccine_reference) - assert isinstance(columns, dict) and "Diphtheria" in columns +@pytest.mark.unit +class TestCheckAddressesComplete: + """Unit tests for check_addresses_complete(). + + Covers: + - All addresses complete β†’ all rows returned, no warning + - Some addresses incomplete β†’ warning logged, incomplete rows dropped by default + - drop_incomplete=False β†’ all rows returned regardless of completeness + - Incomplete rows written to CSV side-effect + - Blank strings and whitespace-only values treated as missing + + Real-world significance: + - check_addresses_complete gates which clients receive a mailed notice; + dropping a client with a missing postal code is correct; silently keeping + one with no street address would produce an undeliverable envelope. + """ + + @pytest.fixture + def output_dir(self, tmp_path, monkeypatch) -> Path: + """Redirect the hardcoded output path to tmp_path and return it. + + The function writes incomplete_addresses.csv to SCRIPT_DIR.parent/output. + Patching SCRIPT_DIR keeps test I/O isolated from the real output/ folder. + Returns the output directory path so CSV-checking tests can use it directly. + """ + out = tmp_path / "output" + out.mkdir(parents=True) + monkeypatch.setattr(preprocess, "SCRIPT_DIR", tmp_path / "pipeline") + return out + + @pytest.fixture + def complete_df(self) -> pd.DataFrame: + """Three rows with fully populated address fields.""" + return pd.DataFrame( + { + "street_address_line_1": ["123 Main St", "456 Side Rd", "789 Oak Ave"], + "street_address_line_2": ["", "Suite 5", ""], + "city": ["Guelph", "Guelph", "Wellington"], + "province": ["ON", "ON", "ON"], + "postal_code": ["N1H 2T2", "N1H 2T3", "N1K 1B2"], + } + ) + + @pytest.fixture + def mixed_df(self) -> pd.DataFrame: + """Two complete rows and one row with missing city and postal_code. + + Uses float("nan") so the normalisation step (.astype(str) β†’ "nan" β†’ + replaced with pd.NA) correctly detects the values as absent. + Plain Python None becomes the string "None" and would not be caught. + """ + return pd.DataFrame( + { + "street_address_line_1": ["123 Main St", "456 Side Rd", "789 Oak Ave"], + "street_address_line_2": ["", "", ""], + "city": ["Guelph", "Guelph", float("nan")], + "province": ["ON", "ON", "ON"], + "postal_code": ["N1H 2T2", "N1H 2T3", float("nan")], + } + ) + + def test_all_complete_returns_all_rows(self, complete_df) -> None: + """Verify all rows are returned when every address is fully populated. + + Real-world significance: + - Clean input must not accidentally drop any clients. + + Assertion: Output has the same row count as input + """ + result = preprocess.check_addresses_complete(complete_df) + + assert len(result) == len(complete_df) + + def test_all_complete_no_warning( + self, complete_df, caplog: pytest.LogCaptureFixture + ) -> None: + """Verify no warning is logged when all addresses are complete. + + Assertion: No warning message is emitted + """ + with caplog.at_level("WARNING"): + preprocess.check_addresses_complete(complete_df) + + assert "incomplete address" not in caplog.text.lower() + + def test_incomplete_rows_dropped_by_default(self, mixed_df, output_dir) -> None: + """Verify incomplete rows are excluded from the return value by default. + + Real-world significance: + - Clients without a deliverable address must be excluded so the mailer + does not attempt to print an undeliverable envelope. + + Assertion: Only the two complete rows are returned + """ + result = preprocess.check_addresses_complete(mixed_df) + + assert len(result) == 2 + + def test_incomplete_rows_logged_as_warning( + self, mixed_df, output_dir, caplog: pytest.LogCaptureFixture + ) -> None: + """Verify a warning is logged reporting how many records are incomplete. + + Real-world significance: + - Operators must be alerted when clients are silently excluded so they + can investigate the source data and resubmit corrected records. + + Assertion: Warning message contains the count of incomplete records + """ + with caplog.at_level("WARNING"): + preprocess.check_addresses_complete(mixed_df) + + assert "There are 1 records with incomplete address information" in caplog.text + + def test_incomplete_rows_written_to_csv(self, mixed_df, output_dir) -> None: + """Verify incomplete records are written to incomplete_addresses.csv. + + Real-world significance: + - The CSV gives operators a machine-readable list of excluded clients + so they can fix addresses and rerun without manually identifying gaps. + + Assertion: CSV exists and contains exactly the incomplete rows + """ + preprocess.check_addresses_complete(mixed_df) + + csv_path = output_dir / "incomplete_addresses.csv" + assert csv_path.exists() + written = pd.read_csv(csv_path) + assert len(written) == 1 + + def test_drop_incomplete_false_returns_all_rows(self, mixed_df, output_dir) -> None: + """Verify drop_incomplete=False keeps all rows regardless of completeness. + + Real-world significance: + - Some callers (e.g. inspection or dry-run modes) need to see the full + dataset including incomplete records to audit what would be excluded. + + Assertion: All rows are returned when drop_incomplete is False + """ + result = preprocess.check_addresses_complete(mixed_df, drop_incomplete=False) + + assert len(result) == len(mixed_df) + + def test_whitespace_only_fields_treated_as_missing(self, output_dir) -> None: + """Verify whitespace-only strings are normalised to NA and trigger incompleteness. + + Real-world significance: + - Source exports sometimes contain cells filled with spaces rather than + a true empty value; these must not pass the completeness check. + + Assertion: Row with whitespace-only city is dropped + """ + df = pd.DataFrame( + { + "street_address_line_1": ["123 Main St"], + "street_address_line_2": [""], + "city": [" "], + "province": ["ON"], + "postal_code": ["N1H 2T2"], + } + ) + + result = preprocess.check_addresses_complete(df) + + assert len(result) == 0 + + def test_address_complete_column_not_in_output(self, complete_df) -> None: + """Verify the temporary address_complete column is not present in output. + + Real-world significance: + - Downstream steps depend on a stable column schema; leaking an + internal boolean column would break downstream consumers. + + Assertion: 'address_complete' is absent from the returned DataFrame + """ + result = preprocess.check_addresses_complete(complete_df) + + assert "address_complete" not in result.columns + + +@pytest.mark.unit +class TestCheckClientInfoComplete: + """Unit tests for check_client_info_complete(). + + Covers: + - All client fields present β†’ all rows returned, no warning + - Missing required field β†’ warning logged, incomplete rows dropped by default + - drop_incomplete=False β†’ all rows returned regardless of completeness + - assignment_mode="fixed" requires overdue_disease and overdue_agent + - assignment_mode="manifest" does not require overdue columns + - Incomplete rows written to CSV side-effect + - Blank strings and whitespace-only values treated as missing + + Real-world significance: + - check_client_info_complete is the gate before client records enter the + pipeline proper; a record with a missing name or date of birth cannot + produce a correct notice and must be excluded and reported. + """ + + @pytest.fixture + def output_dir(self, tmp_path, monkeypatch) -> Path: + """Redirect the hardcoded output path to tmp_path and return it. + + The function writes incomplete_clients.csv to SCRIPT_DIR.parent/output. + Patching SCRIPT_DIR keeps test I/O isolated from the real output/ folder. + Returns the output directory path so CSV-checking tests can use it directly. + """ + out = tmp_path / "output" + out.mkdir(parents=True) + monkeypatch.setattr(preprocess, "SCRIPT_DIR", tmp_path / "pipeline") + return out + + @pytest.fixture + def complete_fixed_df(self) -> pd.DataFrame: + """Two rows with all fields required by fixed-mode assignment. + + imms_given must be non-empty; the normalisation step converts "" to pd.NA, + which would flag the row as incomplete. + """ + return pd.DataFrame( + { + "school_name": ["Tunnel Academy", "River School"], + "client_id": ["C001", "C002"], + "first_name": ["Alice", "Bob"], + "last_name": ["Zephyr", "Smith"], + "date_of_birth": ["2015-01-01", "2014-06-15"], + "imms_given": ["May 1, 2020 - DTaP", "Apr 10, 2019 - IPV"], + "overdue_disease": ["Measles", "Polio"], + "overdue_agent": ["MMR", "IPV"], + } + ) + + @pytest.fixture + def complete_manifest_df(self) -> pd.DataFrame: + """Two rows sufficient for manifest-mode (overdue columns may be empty). + + In manifest mode overdue_disease and overdue_agent are not required, + so they can be absent or empty without flagging a row as incomplete. + imms_given must still be non-empty. + """ + return pd.DataFrame( + { + "school_name": ["Tunnel Academy", "River School"], + "client_id": ["C001", "C002"], + "first_name": ["Alice", "Bob"], + "last_name": ["Zephyr", "Smith"], + "date_of_birth": ["2015-01-01", "2014-06-15"], + "imms_given": ["May 1, 2020 - DTaP", "Apr 10, 2019 - IPV"], + "overdue_disease": ["", ""], + "overdue_agent": ["", ""], + } + ) + + def test_all_complete_fixed_returns_all_rows(self, complete_fixed_df) -> None: + """Verify all rows are returned when every required field is present (fixed mode). + + Assertion: Output has the same row count as input + """ + result = preprocess.check_client_info_complete(complete_fixed_df, assignment_mode="fixed") + + assert len(result) == len(complete_fixed_df) + + def test_all_complete_fixed_no_warning( + self, complete_fixed_df, caplog: pytest.LogCaptureFixture + ) -> None: + """Verify no warning is logged when all client info is present. + + Assertion: No warning message is emitted + """ + with caplog.at_level("WARNING"): + preprocess.check_client_info_complete(complete_fixed_df, assignment_mode="fixed") + + assert "incomplete" not in caplog.text.lower() + + def test_missing_required_field_drops_row_fixed(self, output_dir) -> None: + """Verify rows with a missing required field are excluded (fixed mode). + + Real-world significance: + - A notice without a last name cannot be addressed and must not be + generated; dropping the row and reporting it is the correct response. + + Assertion: Only the one complete row is returned; the incomplete row's + client ID does not appear in the output + """ + df = pd.DataFrame( + { + "school_name": ["Tunnel Academy", "River School"], + "client_id": ["C001", "C002"], + "first_name": ["Alice", "Bob"], + "last_name": ["Zephyr", float("nan")], + "date_of_birth": ["2015-01-01", "2014-06-15"], + "imms_given": ["May 1, 2020 - DTaP", "Apr 10, 2019 - IPV"], + "overdue_disease": ["Measles", "Polio"], + "overdue_agent": ["MMR", "IPV"], + } + ) + + result = preprocess.check_client_info_complete(df, assignment_mode="fixed") + + assert len(result) == 1 + assert result.iloc[0]["client_id"] == "C001" + + def test_missing_required_field_logs_warning( + self, output_dir, caplog: pytest.LogCaptureFixture + ) -> None: + """Verify a warning is logged when incomplete client records are found. + + Real-world significance: + - Operators must be alerted so they can correct the source data; + silent exclusion would cause unnoticed gaps in delivered notices. + + Assertion: Warning message contains the count of incomplete records + """ + df = pd.DataFrame( + { + "school_name": ["Tunnel Academy", "River School"], + "client_id": ["C001", "C002"], + "first_name": ["Alice", float("nan")], + "last_name": ["Zephyr", "Smith"], + "date_of_birth": ["2015-01-01", "2014-06-15"], + "imms_given": ["May 1, 2020 - DTaP", "Apr 10, 2019 - IPV"], + "overdue_disease": ["Measles", "Polio"], + "overdue_agent": ["MMR", "IPV"], + } + ) + + with caplog.at_level("WARNING"): + preprocess.check_client_info_complete(df, assignment_mode="fixed") + + assert "There are 1 records with incomplete/invalid client information" in caplog.text + + def test_incomplete_rows_written_to_csv(self, output_dir) -> None: + """Verify incomplete client records are written to incomplete_clients.csv. + + Real-world significance: + - The CSV gives operators a targeted list of records that need to be + corrected, without requiring manual inspection of the full dataset. + + Assertion: CSV exists and contains exactly the incomplete rows + """ + df = pd.DataFrame( + { + "school_name": ["Tunnel Academy", "River School"], + "client_id": ["C001", "C002"], + "first_name": ["Alice", "Bob"], + "last_name": ["Zephyr", float("nan")], + "date_of_birth": ["2015-01-01", "2014-06-15"], + "imms_given": ["May 1, 2020 - DTaP", "Apr 10, 2019 - IPV"], + "overdue_disease": ["Measles", "Polio"], + "overdue_agent": ["MMR", "IPV"], + } + ) + + preprocess.check_client_info_complete(df, assignment_mode="fixed") + + csv_path = output_dir / "incomplete_clients.csv" + assert csv_path.exists() + written = pd.read_csv(csv_path) + assert len(written) == 1 + + def test_drop_incomplete_false_returns_all_rows(self, output_dir) -> None: + """Verify drop_incomplete=False retains all rows regardless of completeness. + + Real-world significance: + - Inspection or dry-run modes need to see what would be excluded before + committing to dropping records. + + Assertion: All rows returned when drop_incomplete is False + """ + df = pd.DataFrame( + { + "school_name": ["Tunnel Academy", "River School"], + "client_id": ["C001", "C002"], + "first_name": ["Alice", float("nan")], + "last_name": ["Zephyr", "Smith"], + "date_of_birth": ["2015-01-01", "2014-06-15"], + "imms_given": ["May 1, 2020 - DTaP", "Apr 10, 2019 - IPV"], + "overdue_disease": ["Measles", "Polio"], + "overdue_agent": ["MMR", "IPV"], + } + ) + + result = preprocess.check_client_info_complete( + df, assignment_mode="fixed", drop_incomplete=False + ) + + assert len(result) == 2 + + def test_fixed_mode_requires_overdue_columns(self, output_dir) -> None: + """Verify fixed mode treats empty overdue fields as incomplete. + + Real-world significance: + - In fixed mode every client must have an overdue disease and agent; + a record without them cannot produce a valid overdue notice. + + Assertion: Row with empty overdue_disease and overdue_agent is dropped + """ + df = pd.DataFrame( + { + "school_name": ["Tunnel Academy", "River School"], + "client_id": ["C001", "C002"], + "first_name": ["Alice", "Bob"], + "last_name": ["Zephyr", "Smith"], + "date_of_birth": ["2015-01-01", "2014-06-15"], + "imms_given": ["May 1, 2020 - DTaP", "Apr 10, 2019 - IPV"], + "overdue_disease": ["Measles", ""], + "overdue_agent": ["MMR", ""], + } + ) + + result = preprocess.check_client_info_complete(df, assignment_mode="fixed") + + assert len(result) == 1 + assert result.iloc[0]["client_id"] == "C001" + + def test_manifest_mode_does_not_require_overdue_columns( + self, complete_manifest_df + ) -> None: + """Verify manifest mode accepts records with empty overdue fields. + + Real-world significance: + - In manifest mode the notice version is assigned externally; overdue + columns may legitimately be empty for affirmative-schedule notices. + + Assertion: All rows are returned even when overdue columns are empty + """ + result = preprocess.check_client_info_complete( + complete_manifest_df, assignment_mode="manifest" + ) + + assert len(result) == len(complete_manifest_df) + + def test_whitespace_only_field_treated_as_missing(self, output_dir) -> None: + """Verify whitespace-only strings are normalised to NA and flag a record incomplete. + + Real-world significance: + - Source exports may contain cells filled with spaces; these must not + pass the completeness check as if they held real values. + + Assertion: Row with whitespace-only first_name is dropped; CSV is written + """ + df = pd.DataFrame( + { + "school_name": ["Tunnel Academy"], + "client_id": ["C001"], + "first_name": [" "], + "last_name": ["Zephyr"], + "date_of_birth": ["2015-01-01"], + "imms_given": ["May 1, 2020 - DTaP"], + "overdue_disease": ["Measles"], + "overdue_agent": ["MMR"], + } + ) + + result = preprocess.check_client_info_complete(df, assignment_mode="fixed") + + assert len(result) == 0 + + + def test_client_info_complete_column_not_in_output(self, complete_fixed_df) -> None: + """Verify the temporary client_info_complete column is not present in output. + + Real-world significance: + - Downstream steps depend on a stable column schema; leaking an + internal boolean column would break downstream consumers. + + Assertion: 'client_info_complete' is absent from the returned DataFrame + """ + result = preprocess.check_client_info_complete( + complete_fixed_df, assignment_mode="fixed" + ) + + assert "client_info_complete" not in result.columns + + @pytest.mark.unit class TestProcessVaccinesDue: """Unit tests for process_vaccines_due.""" def test_normalizes_disease_names(self) -> None: - result = preprocess.process_vaccines_due("Poliomyelitis;Measles", "en") + result = preprocess.process_vaccines_due("Poliomyelitis;Measles", "disease") assert "Polio" in result assert "Measles" in result def test_empty_input_returns_empty_string(self) -> None: - assert preprocess.process_vaccines_due("", "en") == "" + assert preprocess.process_vaccines_due("", "disease") == "" def test_non_string_input_returns_empty_string(self) -> None: - assert preprocess.process_vaccines_due(None, "en") == "" + assert preprocess.process_vaccines_due(None, "disease") == "" + + +# --------------------------------------------------------------------------- +# Manifest-mode tests for build_preprocess_result +# --------------------------------------------------------------------------- + +def _make_catalog(): + from pipeline.notice_versioning import NoticeKind, NoticeVersion, NoticeVersionCatalog + return NoticeVersionCatalog( + schema_version=1, + default_version="overdue_standard_v1", + default_language="en", + versions={ + "overdue_standard_v1": NoticeVersion( + version_id="overdue_standard_v1", kind=NoticeKind.OVERDUE, requires="has_overdue" + ), + "affirmative_schedule_v1": NoticeVersion( + version_id="affirmative_schedule_v1", kind=NoticeKind.AFFIRMATIVE, requires="no_overdue" + ), + }, + ) + + +def _make_manifest(*rows): + from pipeline.assignment_manifest import ManifestRow + return {r["client_id"]: ManifestRow(**r) for r in rows} + + +def _simple_df(num=2, with_overdue=True): + """Minimal DataFrame for build_preprocess_result tests.""" + data = { + "school_name": ["School A"] * num, + "client_id": [f"C{i:03d}" for i in range(1, num + 1)], + "first_name": ["Alice"] * num, + "last_name": ["Smith"] * num, + "date_of_birth": ["2015-01-01"] * num, + "city": ["Guelph"] * num, + "postal_code": ["N1H 2T2"] * num, + "province": ["ON"] * num, + "overdue_disease": (["Measles;Polio"] * num if with_overdue else [""] * num), + "overdue_agent": (["MMR;IPV"] * num if with_overdue else [""] * num), + "imms_given": [""] * num, + "street_address_line_1": ["123 Main St"] * num, + "street_address_line_2": [""] * num, + } + import pandas as pd + return pd.DataFrame(data) + + +@pytest.mark.unit +class TestBuildPreprocessResultFixedMode: + """Fixed-mode: metadata empty, returns None as second element.""" + + def test_fixed_mode_returns_tuple_with_none_result(self, tmp_path) -> None: + result, reconciliation_result = preprocess.build_preprocess_result( + _simple_df(2), "en", {}, preprocess.REPLACE_UNSPECIFIED + ) + assert reconciliation_result is None + + def test_fixed_mode_metadata_empty_for_all_clients(self, tmp_path) -> None: + result, _ = preprocess.build_preprocess_result( + _simple_df(2), "en", {}, preprocess.REPLACE_UNSPECIFIED + ) + for client in result.clients: + assert "resolved_notice" not in client.metadata + + +@pytest.mark.unit +class TestBuildPreprocessResultManifestMode: + """Manifest-mode: metadata resolved_notice present, language set from manifest.""" + + def test_manifest_mode_returns_reconciliation_result(self, tmp_path) -> None: + catalog = _make_catalog() + manifest = _make_manifest( + {"client_id": "C001", "notice_version": "overdue_standard_v1", "language": "en", "experiment_id": None, "experiment_arm": None}, + {"client_id": "C002", "notice_version": "overdue_standard_v1", "language": "fr", "experiment_id": None, "experiment_arm": None}, + ) + result, reconciliation_result = preprocess.build_preprocess_result( + _simple_df(2), None, {}, preprocess.REPLACE_UNSPECIFIED, catalog=catalog, manifest=manifest + ) + assert reconciliation_result is not None + + def test_manifest_mode_resolved_notice_in_metadata(self, tmp_path) -> None: + catalog = _make_catalog() + manifest = _make_manifest( + {"client_id": "C001", "notice_version": "overdue_standard_v1", "language": "en", "experiment_id": None, "experiment_arm": None}, + {"client_id": "C002", "notice_version": "overdue_standard_v1", "language": "en", "experiment_id": None, "experiment_arm": None}, + ) + result, _ = preprocess.build_preprocess_result( + _simple_df(2), None, {}, preprocess.REPLACE_UNSPECIFIED, catalog=catalog, manifest=manifest + ) + for client in result.clients: + assert "resolved_notice" in client.metadata + + def test_manifest_mode_language_from_manifest_not_cli(self, tmp_path) -> None: + catalog = _make_catalog() + manifest = _make_manifest( + {"client_id": "C001", "notice_version": "overdue_standard_v1", "language": "en", "experiment_id": None, "experiment_arm": None}, + {"client_id": "C002", "notice_version": "overdue_standard_v1", "language": "fr", "experiment_id": None, "experiment_arm": None}, + ) + result, _ = preprocess.build_preprocess_result( + _simple_df(2), None, {}, preprocess.REPLACE_UNSPECIFIED, catalog=catalog, manifest=manifest + ) + langs = {c.client_id: c.language for c in result.clients} + assert langs["C001"] == "en" + assert langs["C002"] == "fr" + + def test_manifest_mode_eligibility_conflict_halts(self, tmp_path) -> None: + """Affirmative assigned to client with vaccines_due β†’ raises before artifact write.""" + catalog = _make_catalog() + # Assign affirmative to a client that has vaccines due + manifest = _make_manifest( + {"client_id": "C001", "notice_version": "affirmative_schedule_v1", "language": "en", "experiment_id": None, "experiment_arm": None}, + {"client_id": "C002", "notice_version": "affirmative_schedule_v1", "language": "en", "experiment_id": None, "experiment_arm": None}, + ) + with pytest.raises(ValueError, match="[Pp]reflight"): + preprocess.build_preprocess_result( + _simple_df(2, with_overdue=True), None, {}, preprocess.REPLACE_UNSPECIFIED, catalog=catalog, manifest=manifest + ) + + def test_manifest_mode_allow_unassigned_true_uses_defaults(self, tmp_path) -> None: + """allow_unassigned=True: client missing from manifest uses catalog defaults.""" + catalog = _make_catalog() + # Only assign C001; C002 is missing from manifest + manifest = _make_manifest( + {"client_id": "C001", "notice_version": "overdue_standard_v1", "language": "en", "experiment_id": None, "experiment_arm": None}, + ) + # Write config with allow_unassigned=true + config_path = tmp_path / "parameters.yaml" + config_path.write_text( + "notice_versioning:\n allow_unassigned: true\n extra_manifest_rows: error\n", + encoding="utf-8", + ) + result, reconciliation_result = preprocess.build_preprocess_result( + _simple_df(2), None, {}, preprocess.REPLACE_UNSPECIFIED, + config_path=config_path, catalog=catalog, manifest=manifest + ) + # C002 should be resolved with catalog defaults, not missing + assert reconciliation_result is not None + assert "C002" not in reconciliation_result.missing_clients + c002 = next(c for c in result.clients if c.client_id == "C002") + assert c002.language == catalog.default_language + + def test_manifest_mode_allow_unassigned_false_raises(self, tmp_path) -> None: + """allow_unassigned=False: missing client causes preflight failure.""" + catalog = _make_catalog() + manifest = _make_manifest( + {"client_id": "C001", "notice_version": "overdue_standard_v1", "language": "en", "experiment_id": None, "experiment_arm": None}, + ) + # allow_unassigned defaults to False + with pytest.raises(ValueError, match="[Pp]reflight"): + preprocess.build_preprocess_result( + _simple_df(2), None, {}, preprocess.REPLACE_UNSPECIFIED, + catalog=catalog, manifest=manifest + ) + + def test_manifest_mode_extra_rows_error_raises(self, tmp_path) -> None: + catalog = _make_catalog() + # EXTRA_CLIENT is in manifest but not in cohort + manifest = _make_manifest( + {"client_id": "C001", "notice_version": "overdue_standard_v1", "language": "en", "experiment_id": None, "experiment_arm": None}, + {"client_id": "C002", "notice_version": "overdue_standard_v1", "language": "en", "experiment_id": None, "experiment_arm": None}, + {"client_id": "EXTRA_CLIENT", "notice_version": "overdue_standard_v1", "language": "en", "experiment_id": None, "experiment_arm": None}, + ) + config_path = tmp_path / "parameters.yaml" + config_path.write_text( + "notice_versioning:\n allow_unassigned: false\n extra_manifest_rows: error\n", + encoding="utf-8", + ) + with pytest.raises(ValueError, match="[Pp]reflight"): + preprocess.build_preprocess_result( + _simple_df(2), None, {}, preprocess.REPLACE_UNSPECIFIED, + config_path=config_path, catalog=catalog, manifest=manifest + ) + + def test_manifest_mode_extra_rows_warn_continues(self, tmp_path) -> None: + catalog = _make_catalog() + manifest = _make_manifest( + {"client_id": "C001", "notice_version": "overdue_standard_v1", "language": "en", "experiment_id": None, "experiment_arm": None}, + {"client_id": "C002", "notice_version": "overdue_standard_v1", "language": "en", "experiment_id": None, "experiment_arm": None}, + {"client_id": "EXTRA_CLIENT", "notice_version": "overdue_standard_v1", "language": "en", "experiment_id": None, "experiment_arm": None}, + ) + config_path = tmp_path / "parameters.yaml" + config_path.write_text( + "notice_versioning:\n allow_unassigned: false\n extra_manifest_rows: warn\n", + encoding="utf-8", + ) + # Should NOT raise because extra_manifest_rows=warn + result, reconciliation_result = preprocess.build_preprocess_result( + _simple_df(2), None, {}, preprocess.REPLACE_UNSPECIFIED, + config_path=config_path, catalog=catalog, manifest=manifest + ) + assert reconciliation_result is not None + assert "EXTRA_CLIENT" in reconciliation_result.extra_rows + + def test_manifest_mode_missing_language_falls_back_to_default(self, tmp_path) -> None: + catalog = _make_catalog() + # C001 has no language in manifest row + manifest = _make_manifest( + {"client_id": "C001", "notice_version": "overdue_standard_v1", "language": None, "experiment_id": None, "experiment_arm": None}, + {"client_id": "C002", "notice_version": "overdue_standard_v1", "language": "fr", "experiment_id": None, "experiment_arm": None}, + ) + result, reconciliation_result = preprocess.build_preprocess_result( + _simple_df(2), None, {}, preprocess.REPLACE_UNSPECIFIED, catalog=catalog, manifest=manifest + ) + assert reconciliation_result is not None + assert "C001" in reconciliation_result.missing_language_clients + c001 = next(c for c in result.clients if c.client_id == "C001") + assert c001.language == catalog.default_language + + +@pytest.mark.unit +class TestWriteAssignmentMetadata: + """write_assignment_metadata: file creation and count aggregation.""" + + def _clients_with_resolved(self, result): + return [c for c in result.clients if "resolved_notice" in c.metadata] + + def test_writes_file_with_correct_counts(self, tmp_path) -> None: + """Verify the metadata file is written and per-version/language counts are correct. + + Real-world significance: + - The file is the only machine-readable audit record of which notice version + each client received for a given run; incorrect counts would mislead auditors. + + Assertion: counts_by_version and counts_by_language match the manifest assignments + """ + catalog = _make_catalog() + manifest = _make_manifest( + {"client_id": "C001", "notice_version": "overdue_standard_v1", "language": "en", "experiment_id": None, "experiment_arm": None}, + {"client_id": "C002", "notice_version": "overdue_standard_v1", "language": "fr", "experiment_id": None, "experiment_arm": None}, + ) + result, reconciliation_result = preprocess.build_preprocess_result( + _simple_df(2), None, {}, preprocess.REPLACE_UNSPECIFIED, catalog=catalog, manifest=manifest + ) + assert reconciliation_result is not None + import json + out_path = preprocess.write_assignment_metadata(tmp_path, "run123", catalog, reconciliation_result, result.clients) + assert out_path.exists() + payload = json.loads(out_path.read_text()) + assert payload["counts_by_version"] == {"overdue_standard_v1": 2} + assert payload["counts_by_language"] == {"en": 1, "fr": 1} + assert payload["total_clients"] == 2