Fetch existing HACS data once before generation

[ludeeus]

Summary

The generate workflow fetched the existing published data from R2 inside every category-data leg: each leg called HacsDataClient.get_data(<category>) for the category's data.json, and get_repositories("removed") for the removed list. Since the legs run serially, the removed list was fetched once per category, and a republish of R2 mid-run could leave later legs working off a different baseline than earlier ones.

This PR fetches that data exactly once in a new lightweight job and shares it with every category leg via an artifact, so the whole run uses one consistent snapshot. It is a standalone change against main; the sharding work in #5419 can rebase on top and reuse the same mechanism.

Key Changes

• New helpers in generate_category_data.py (_read_snapshot, get_stored_data, get_removed_repositories): read each category's data.json and the removed list from $HACS_STORED_DATA_DIR when set, and fall back to the data client when the dir is unset or when a file is missing, is invalid JSON, or has the wrong type (a warning is logged in each fallback case). The two existing fetches now go through these helpers.

• GitHub Actions workflow (generate-hacs-data.yml):

  • New fetch-stored-data job that fetches every category's data.json plus the removed list once with curl (retry 5× then hard-fail the run), into outputdata/stored, and uploads it as the stored-data artifact. A comment documents the accepted trade-off: any single fetch failing fails the whole run (one consistent snapshot), rather than being isolated to that category's leg.
  • category-data depends on fetch-stored-data, downloads that artifact, and runs with HACS_STORED_DATA_DIR=outputdata/stored.
  • generate-matrix now emits the categories list as valid JSON (via jq) so both fromJSON and the fetch job's jq -r parse it robustly.
  • summarize now iterates only the expected category directories (from generate-matrix) that produced a summary.json, so the new artifact (and any non-category directory) can't break it.
  • notify_on_failure depends on fetch-stored-data.

• Tests (test_generate_category_data.py): _StubDataClient/_StubHacs return the real data-client shapes; tests cover the snapshot-read path, every fallback (unset dir, missing file, invalid JSON, wrong shape), and an end-to-end test proving the stored-snapshot path reproduces the fetch-path output.

Implementation Details

• HACS_STORED_DATA_DIR is optional. When unset (tests, local dev, single-repo validate.yml) the original data-client fetch behavior is preserved, so output is unchanged.
• Reading the snapshot is network-free file I/O; the fallbacks make a partial or corrupt snapshot degrade to fetching rather than failing generation.
• The existing per-category stored.json published to R2 is untouched — only the source of the baseline changed.
• Known limitation: because category-data runs serially over hours, a repo removed mid-run is not dropped until the next scheduled run (the removed list is part of the run-start snapshot). Honoring mid-run removals uniformly is a possible follow-up.

https://claude.ai/code/session_01L22jYdNGF4hB4fvTYPTzP4

[Copilot]

Pull request overview

This PR adds optional “snapshot” inputs to the category data generator so it can reuse already-published data from disk (when provided) instead of always fetching from the remote data client, and updates the GitHub Actions workflow to prefetch that snapshot once per run.

Changes:

• Add get_stored_data() and get_removed_repositories() helpers to read pre-fetched JSON from HACS_EXISTING_DATA_DIR (with fallback to the data client).
• Update the generator to use the new helpers for stored category data and removed repositories.
• Extend CI with a preflight job to download published data into an artifact, then pass it to category jobs via HACS_EXISTING_DATA_DIR; add tests for the new helper behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
scripts/data/generate_category_data.py	Adds snapshot-aware helper functions and wires them into category generation.
tests/scripts/data/test_generate_category_data.py	Adds tests/stubs verifying snapshot read behavior and fallback fetching.
.github/workflows/generate-hacs-data.yml	Adds a preflight fetch job + artifact handoff and improves matrix JSON handling.

Comments suppressed due to low confidence (2)

scripts/data/generate_category_data.py:80

• When HACS_EXISTING_DATA_DIR is set but removed.json is missing/invalid, this will raise and fail the whole run. To match the intended "use snapshot when available" behavior, fall back to fetching removed repositories on file/JSON errors.

    if EXISTING_DATA_DIR:
        with open(
            os.path.join(EXISTING_DATA_DIR, "removed.json"), encoding="utf-8"
        ) as file:
            return json.load(file)

tests/scripts/data/test_generate_category_data.py:372

• This assertion should match the (repo-id -> repo-data) shape returned by the real data client and the updated test stub, otherwise the test can pass while still diverging from production expectations.

    assert await get_stored_data(hacs, "plugin") == {"fetched": "plugin"}

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

scripts/data/generate_category_data.py:92

• Similarly, get_removed_repositories will return whatever JSON is in removed.json if it parses, even if it isn't a list of repository names. Validating that the snapshot is a list (and falling back when it's not) avoids hard-to-debug failures later in the generation flow.

async def get_removed_repositories(hacs: AdjustedHacs) -> list[str]:
    """Return the removed-repositories list from the snapshot dir when available, else fetch."""
    if (removed := _read_snapshot(hacs, "removed.json")) is not None:
        return removed
    return await hacs.data_client.get_repositories("removed")

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.