Adding script for exporting study plan from Ladok

[cohm]

Summary

Addresses #11.

Adds a Python extractor that builds candidate src/data/<PROG>.json files from Ladok's program struktur, plus a small --include flag on the validator so candidates can be schema-checked outside programs.json. First target: CTFYS. Verified data is never touched — the extractor writes to <PROG>.ladok.json for human diff-and-merge review.
End-to-end smoke test against CTFYS HT23: 48 entries (46 courses, 2 option groups), structurally aligned with the hand-curated CTFYS.json for years 1–2 and the year-3 mandatories + bachelor thesis.

What's new

• scripts/extract-from-ladok.py — Python 3.11+ CLI. Talks to Ladok via dbosk/ladok3 for the programme struktur, period scheduling, credits, and Swedish/English titles. Enriches with KOPPS (api.kth.se/api/kopps/v2/course/<code>) and the public kursplan HTML page for prerequisites, grading scale, and course level. Output goes to a candidate file alongside the verified one; existing <PROG>.json is never overwritten.
• scripts/requirements.txt — pins ladok3, requests, beautifulsoup4.
• scripts/validate-data.mjs — new --include <programCode>=<path> flag so a candidate JSON can be validated against the corresponding programme entry from programs.json without registering it as the active data file.
• CLAUDE.md — extractor recipe added to the Commands block (venv setup, ladok login, run, validate).
• .gitignore — adds /.venv and /scripts/.venv.

CLI surface

python scripts/extract-from-ladok.py CTFYS                       # current läsår
python scripts/extract-from-ladok.py CTFYS --cohort HT25
python scripts/extract-from-ladok.py CTFYS --cohort-range 2016-2025
python scripts/extract-from-ladok.py CTFYS --use-production-server
python scripts/extract-from-ladok.py CTFYS --list-rounds         # enumerate, exit
python scripts/extract-from-ladok.py CTFYS --round-uid <uuid>    # use this round
python scripts/extract-from-ladok.py CTFYS --inspect-round       # raw round JSON
python scripts/extract-from-ladok.py CTFYS --inspect-programme   # raw struktur JSON
node scripts/validate-data.mjs --include CTFYS=src/data/CTFYS.ladok.json

How it works

1. Find the round. program_rounds_JSON(code) lists every round Ladok has for that programme. Numeric admission codes win over OP* year-2-entry rounds; DTF*/DTM*/DFA*/DFY* exchange/double-degree rounds are filtered out. Cohort is chosen by SammanslagenTillfallesperiod.Startdatum.
2. Fetch the struktur via the hypermedia rel http://relations.ladok.se/utbildningsinformation/struktur on the programme round. The response's Innehall is a tree of typed Boxtyp boxes: UTBILDNINGSTILLFALLESBOX is a course-round leaf OR a sub-programme reference (distinguished by whether Utbildningskod matches the KTH course-code regex ^[A-Z]{2,3}\d{3,4}[A-Z]?$); VALBOX is an elective container whose descendants become options; SEKVENSBOX and friends are pass-through containers. Sub-programme references are followed (one extra level) via their own struktur link. VALBOX context propagates across sub-programme boundaries so options inside a sub-programme stay grouped with their enclosing VALBOX.
3. Period credits come from course_round_utbildningsinformation_JSON(cr_uid).Tillfallesperioder. Two shapes are handled: the structured (HT25+) form Lasperiodsfordelning.Fordelningsposter[*] with Lasperiodskod + Omfattningsvarde, and the legacy / date-based (HT23 and earlier) form with one Tillfallesperioder entry per period carrying ForstaUndervisningsdatum + Omfattningsvarde — period derived from the start date.
4. Year-in-programme is derived from the leaf's Startdatum versus the cohort's admission year.
5. VALBOX → optionGroup. Each VALBOX produces one optionGroup entry (named from Benamning, or Valbart block N numbered chronologically when anonymous), plus a Course entry for each option (the validator requires options[] to reference real courses).
6. KOPPS enrichment adds prerequisites (scraped from the public kursplan page's Särskild behörighet section), the canonical Swedish/English titles, grading scale, and course level.
7. exams / reexams stay empty — those require manual verification against Riktlinje om läsårets förläggning and are left for the human reviewer.

Validation against CTFYS

HT23 cohort, 46 courses + 2 option groups:

• Years 1 & 2: all shared courses align on year, total credits, and period credits.
• Year 3 mandatory courses (5/5): EI1320, EL1000, SG1218, SH1015, SI1336 all captured.
• Bachelor thesis: Valbart block 2 (year 3, 15 hp, P3+P4) with options [EF112X, SA114X] — matches the verified Kandidatexamensarbete modulo the name (rename during merge).
• Year-3 electives: Valbart block 1 with 20 KTH-recommended tillvalskurser — the real Ladok representation of the verified file's XY123Z / XY456Z placeholders.
• Validator: 0 errors, 83 warnings — mostly KOPPS-scraped prerequisites referencing courses from other programmes (SF1675, SF1918, etc.) and the expected cosmetics warnings about year-3 placeholders.

Known limitations

• Some HT23 courses have only single-period Tillfallesperioder in Ladok even when the verified file splits them across periods (DD1331, SE1055, SF1683, SG1112, SI1121). Total credits and the start period are correct; the trailing period attribution is missing from Ladok's source. Newer cohorts (HT25) use the structured shape and don't have this issue.
• exams/reexams are always empty — by design.
• KOPPS prerequisite scrape pulls in cross-programme alternatives (e.g. SF1922 lists SF1625, which is the CFATE variant of SF1673). The validator warns; the reviewer filters during merge.
• Anonymous VALBOX naming falls back to Valbart block N. CTFYS's two year-3 VALBOXes both come out anonymous and should be renamed during merge (Rekommenderade kurser, Kandidatexamensarbete).

Test plan

• pip install -r scripts/requirements.txt succeeds in a fresh .venv.
• python scripts/extract-from-ladok.py --help works without ladok3 installed.
• npm run validate-data still passes (no regression).
• node scripts/validate-data.mjs --include CTFYS=src/data/CTFYS.ladok.json reports 0 errors.
• python scripts/extract-from-ladok.py CTFYS --list-rounds enumerates all CTFYS rounds with [✓] / [skip] markers for the regular admission vs special-purpose codes.
• python scripts/extract-from-ladok.py CTFYS --cohort HT25 produces a candidate covering years 1–2 (HT25 cohort hasn't reached year 3 yet, and Ladok struktur object is not complete).
• python scripts/extract-from-ladok.py CTFYS --cohort HT23 produces a candidate with the year-3 content including both VALBOXes.
• Generated candidate JSON shape matches what useCourseModel.ts expects (flat periodCredits + year, optionGroup entries with kind: "pickN", etc.) — verified by the --include check above.

Still to do

The ladok3 package should be extended to allow for reading out the prerequisite information so that everything can come from Ladok, and we don't need to resort to Kopps (discontinued) or course web pages (which anyway take their info from Ladok, I think).

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
program-visualization	Ready	Preview, Comment	May 19, 2026 4:20pm