This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
Personal project (nitrocode/omaha-deals-map). Multi-source Python scraping pipeline that emits data/deals.json, consumed by a static Leaflet site deployed via GitHub Pages.
Live: https://nitrocode.github.io/omaha-deals-map/
Design: docs/superpowers/specs/2026-06-06-omaha-deals-map-design.md
Plan: docs/superpowers/plans/2026-06-06-omaha-deals-map.md
Five numbered stages in scripts/, each split into two files:
<NN>_<name>.py, CLI entry point. Keep this stub small. Just argparse +sys.exit(main(...))._<name>_main.py, the real logic. Importable, mockable, tested directly.
Order: 01_fetch → 02_parse → 03_extract_times → 04_geocode → 05_build. Run all via make all; individually via make scrape | parse | extract | geocode | build; with caches bypassed via make rebuild.
Sources are plug-in modules under sources/<name>/:
__init__.pyre-exportsfetchandparse.fetch.pyexportsfetch(client, cache_path) -> SomePayload(dataclass).parse.pyexportsparse(payload) -> list[SourceRecord].
Then add the name to sources/registry.yaml and a fixture-based test (use tests/fixtures/<name>/ for captured payloads).
SourceRecord and Window live in sources/_common.py. kind is one of happy_hour | special | voucher. Days are mon..sun (lowercase, 3-char). Times are HH:MM 24h.
| Cache | File | Survives across runs? |
|---|---|---|
| HTTP body + ETag/Last-Modified | data/http_cache.yaml |
yes |
| Per-source raw payload | data/raw/<source>/latest.pickle |
yes (gitignored, regenerable) |
| Geocode name to lat/lng | data/geocode_cache.yaml |
yes |
make all is offline-safe for unchanged content. make rebuild forces every cache miss.
Fast suite (default): pytest -q -m "not slow", no network, runs in <1s.
Live integration: pytest -m slow, hits real APIs (growomaha, visitomaha, bigdealsmedia, Photon, the SRI gate). Add the pytestmark = pytest.mark.slow line to any new test that touches the network.
This was a real bug shipped on this repo. Every <link>/<script> integrity attribute in site/*.html is verified by scripts/check_sri.py (used by pre-commit + CI). To get a correct hash:
curl -sL <url> | openssl dgst -sha384 -binary | openssl base64 -AUse sha384, not sha256. Cross-verify against cdnjs when reasonable.
Order: override -> source-provided lat/lng -> name cache -> Nominatim -> Photon.
_is_plausible_match in _geocode_main.py is a load-bearing strict-token-overlap guard against Photon's fuzzy false positives ("Nick's Quorum" matched "Fairfield Inn" before this check existed). Don't relax the 5-char threshold without understanding why it's there.
Storage everywhere = HH:MM 24h. Display in the UI = 12h via formatTime12() in site/app.js. Tests assert 24h. Never mix the two.
data/raw/(binary pickles, regenerable)data/overrides/personal.yaml(your tags/ratings/notes, private).venv/,__pycache__/,*.egg-info/
For sources that don't work in CI (Cloudflare bot protection, etc), one-off
scripts live in scripts/oneoff/. They use the [oneoff] extra in
pyproject.toml so the main + dev installs stay lean.
pip install -e ".[oneoff]" # ~150MB (Playwright + browser drivers)
playwright install chromium # one-time, downloads Chromium
make scrape-npdodge # appends to data/overrides/manual_venues.yamlOutputs go through manual_venues.yaml (no automated commit) so the maintainer reviews + geocodes before pushing.
Conventional Commits with Refs: omaha-deals-map trailer. Working directly on main is the convention (personal repo, no PR gate). Pre-commit (pre-commit install once) runs ruff + the fast pytest suite + SRI verification on every commit.
make install # installs from requirements.lock with --require-hashes
pre-commit installAfter that, hooks fire automatically on git commit. CI runs the same checks.
Every Python dep (direct + transitive) is pinned in requirements.lock with
sha256 hashes. CI installs with pip install --require-hashes -r requirements.lock
so a compromised upstream release fails fast.
After editing pyproject.toml, regenerate the lock:
make lock # uv pip compile --generate-hashesCommit requirements.lock alongside the pyproject.toml change. Don't
hand-edit the lockfile.