feat: Input Schema - #207
Conversation
There was a problem hiding this comment.
There was a problem hiding this comment.
Can we snake_case-ify like standards in other pipelines?
There was a problem hiding this comment.
Went with yaml because the internet suggested it was a little more human readable - so though it might be nicer for other PHUs looking to user our code; but can switch it to json to match our existing pipeline schemas!
There was a problem hiding this comment.
Ah, I thought frictionless tabular schema was officially only a JSON spec https://specs.frictionlessdata.io/table-schema/#language
I agree it's quite readable
There was a problem hiding this comment.
Yeahh you're right though, I'll change it - readability argument is weak since non-developers probably wouldn't be editing a schema file anyway, and using proper frictionless descriptor makes other tooling we decide to implement straightforward (thanks for the review btw!!)
| | `Client Id` | 10-digit numeric string | | ||
| | `First Name` | | | ||
| | `Last Name` | | | ||
| | `Age` | Integer | |
There was a problem hiding this comment.
Should work to also refine and minimize this set... Age and Birth Year are best as derived in pipeline as needed rather than relied upon from input data.
| | `Board Name` | | ||
| | `Board Id` | | ||
| | `School Id` | | ||
| | `Unique Id` | |
There was a problem hiding this comment.
This is different from client ID?
There was a problem hiding this comment.
In test sets contains 'SCHOOL NAME - PHIX ID' - perhaps a relic of earlier report formats. This is the structure we currently expect from SCHOOL_NAME column in new reports; will remove 'Unique Id' from expected list and test set to reflect new standards
|
|
||
| The following columns are **optional** and will be used when present: | ||
|
|
||
| | Column name | |
There was a problem hiding this comment.
Support optional "version" identifier (e.g. for multi-language support, A/B tests, etc)
|
|
||
| | Column name | Notes | | ||
| |---|---| | ||
| | `School Type` | | |
There was a problem hiding this comment.
I see in test fixtures this included "catholic". I would've expected this was secondary/elementary. Is it also possible however, that type is maintained in our look up table from #194 or perhaps isn't needed at all - and simply any dynamics around separate notices for elementary vs secondary is dealt with via the optional, more generic version code approach (see other comment on optional fields)
There was a problem hiding this comment.
In current code, school type column is never actually used - templates pass hard-coded values to conf.typ for info box; in the interest of min required cols, this could be easily handled as template/version specific
… to enforced frictionless schema
ef21801 to
4894d0e
Compare
| @@ -249,6 +249,7 @@ uv run pytest -m "not e2e" | |||
| - Use data extracts from [Panorama PEAR](https://accessonehealth.ca/) | |||
| - Place input files in the `input/` subfolder (not tracked by Git) | |||
| - Files must be `.xlsx` format with a **single worksheet** per file | |||
There was a problem hiding this comment.
Is there a reason that we want .xlsx over .csv? .xlsx can do some weird things with dates/formatting we may like to avoid dealing with at all, and incurs some xlsx handling dependencies. Perhaps this is a PEAR thing though - if more rows can be exported at a time in xlsx format vs csv?
I believe with PEAR you can export the report template definitions/schema which we could include in this repo once revised (in addition for PHU users to access the template from our collaboration folder in PEAR)
| | `School Id` | | ||
| | `Unique Id` | | ||
|
|
||
| The full schema is defined in `config/input_schema.yaml`. If the file is missing any required column, the pipeline will stop immediately with a clear error message listing the missing columns. |
There was a problem hiding this comment.
Another idea on this... I am imagining that it's going to come up a lot where we want to list a schema in a user friendly way in documentation, that is also coded in the frictionless JSON and that this will create an opportunity for drift between documentation and implementation. Should we consider (perhaps separate of this PR, or even this project), a tool that will enable us to have an approachable display method in mkdocs for our various schemas?
There was a problem hiding this comment.
Current approach: uv run python docs/generate_schema_docs.py run before mkdocs build in docs.yml. Could add a more standardized generate_schema_docs.py file to our template repo to keep consistent across our projects
…ower_snake_case; adding generated schema .md to docs; support for optional 'version_id column'
…e with lower_snake_case
… info check will drop or retain incomplete records
jangevaare
changed the title
2 participants
Switching from input dataframe normalization / fuzzy matching of columns to enforced frictionless schema.
Expects immunization database report column names in
input_schema.yaml- then maps to internal values.