Skip to content

Add people directory scraper for periodic JSON export - #9

Merged
lkiesow merged 4 commits into
mainfrom
scrape
Jul 15, 2026
Merged

lkiesow merged 4 commits into
mainfrom
scrape

Conversation

@lkiesow

@lkiesow lkiesow commented Jul 14, 2026

Copy link
Copy Markdown
Member

Walks the public A-Z Personensuche listing, fetches full contact details per person, and writes the result to a JSON file for later use by MCP tools. Scheduling is left to an external cron/systemd timer (mcp-uos-scrape-people).

lkiesow added 4 commits July 14, 2026 10:40
Walks the public A-Z Personensuche listing, fetches full contact
details per person, and writes the result to a JSON file for later
use by MCP tools. Scheduling is left to an external cron/systemd
timer (mcp-uos-scrape-people).
Print an updated count after every person's detail fetch within the
current letter (e.g. "B: 12/145 people details fetched"), so it's
possible to tell how far through a letter the scraper is instead of
just a global every-100 counter. Drop that old counter since it's now
redundant.
uos_people_search now answers entirely from the JSON file produced by
mcp-uos-scrape-people, returning full contact details inline instead of
just name/details_url. This removes the need for a live per-person
lookup, so uos_person_details is dropped along with the now-unused
UOSWebsiteClient.people_search().

The snapshot is loaded once at startup (a missing/unreadable file is a
fatal startup error) and can be refreshed without a restart via SIGHUP,
which reloads in place and falls back to the last-known-good data on
failure.
Importing mcpuos loads the people directory JSON at import time, so
pytest collection (and CI) crashed with FileNotFoundError whenever
./data/people.json didn't exist. Point UOS_MCP_PEOPLE_DATA_PATH at a
committed fixture in tests/conftest.py before any test module imports
the package, and reuse that fixture for the MCP server integration
tests instead of generating one in tmp_path.

The import-time load also means mcp-uos-scrape-people can't run on a
fresh install until a snapshot exists; document a one-line stub
workaround in the README.
@lkiesow
lkiesow merged commit f98ee26 into main Jul 15, 2026
2 checks passed
@lkiesow
lkiesow deleted the scrape branch July 15, 2026 23:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant