chore: skill pipeline updates#1679
Conversation
🤖 Skill pipeline status
This comment is updated each iteration. Failure mid-chain leaves the chain stopped — re-trigger manually with |
Replaces the legacy `reviewed_no_skill` status (which conflated
blocked / not_applicable / supplementary) with the granular reason
sourced from skill_candidates.json. Ran via:
python3 agent-ready-docs-hub/migrate_coverage_status.py \
polkadot-docs --mkdocs-root .
Migration result:
before: 23 reviewed_no_skill
after: 23 blocked
All 23 are pages whose skills were blocked in the previous PR cycle
(transfer-assets-into-polkadot, convert-assets, register-foreign-
asset, register-local-asset, run-a-collator, bootnode, etc.). None
mapped to not_applicable or supplementary — those classifications
were never assigned `reviewed_no_skill` by the legacy logic; they
sat as `uncovered`, and the updated generate_coverage.py will pick
them up correctly on the next workflow run.
Companion to agent-ready-docs-hub@a3ace06 which updates the writer
(generate_coverage.py) to emit the granular statuses natively. That
PR must merge before the next workflow run, otherwise the old writer
will re-emit `reviewed_no_skill` and undo this migration.
Ran the new stale-sweep tool against pages whose skills are missing the project_structure field — which 44 skills are, post PR #1679 (only the 6 new + 11 incidentally-touched skills got it; the other 44 weren't stale and so never picked up the field). python3 agent-ready-docs-hub/mark_skills_stale.py polkadot-docs \\ --mkdocs-root . --missing-field project_structure Selected skills: 44 Pages flipped to stale (last_scanned -> null): 44 Pages already stale (no-op): 0 The next workflow run will compute status="stale" from the timestamp mismatch and drain these via the generate-skills job. Per the batch-tuning change (workflows-private#15, batch-size 5→7, stale counts against batch limit), the agent will process at most 7 skills per chained iteration: 44 stale ÷ 7 per batch ≈ 7 chained iterations Wall time ≈ 7 × (~5-10 min/batch + 60s rate-limit pause) Total cost ≈ 7 × ~$1-2/iteration After completion, `report_schema_drift.py` should show project_structure at 61/61 (100%) and the next stale-sweep will report nothing to do.
aljosamakevic
left a comment
aljosamakevic
left a comment
There was a problem hiding this comment.
Skill-gen-workflow improvements — recent batch
Race conditions: workflow-level concurrency: block (per-client group, cancel-in-progress: false) + git pull --rebase -X theirs before push in detect-changes. Self-races serialized, external races recover.
Status taxonomy (hub): split reviewed_no_skill into blocked / not_applicable / supplementary. Updated generate_coverage.py, app.py, RUNBOOK. New migrate_coverage_status.py migration tool. Live skill_coverage.json migrated — 23 pages flipped to blocked, zero residue.
Batch tuning: batch-size default 5 → 7, stale now counts against batch limit (alerts still exempt). Prevents context-window blowup on mass stale-sweeps.
Stale-management tools (hub): mark_skills_stale.py (queue skills for regen by --missing-field / --skill-ids / --skill-id-pattern), report_schema_drift.py (passive per-field coverage report), new RUNBOOK section.
First mass stale-sweep (077e0a5 + 3a30908): 23 statuses migrated + 44 skills flipped stale for project_structure backfill. Surfaced two bugs:
Chain + canonicalize fixes:
- Self-chain decision now counts pending + stale + alert-affected (was pending-only → stalled stale-only queues after iteration 1)
- JSON canonicalization (
sort_keys=True) before commit — eliminates the +9K/−9K diff churn that was choking GitHub's PR renderer
Dispatcher default bump (polkadot-docs #1681): dispatcher's batch-size default 5 → 7 (was overriding the upstream 7).
Drain (877962f → e5a87b6, 9 chained iterations): all 55 stale skills regenerated. project_structure coverage 0 → 43/62 (ceiling ~58/62 — 15 pure-API skills correctly omit the field per schema).
Net state
| Metric | Before | After |
|---|---|---|
| Skills | 55 | 62 |
| Coverage statuses | 4 | 6 |
uncovered pages |
56 (many incorrectly so) | 0 |
| Typical agent-PR diff | ~9K lines | proportional to actual changes |
| Batch size | 5 | 7 |
| Chain handles stale-only | no | yes |
| Race-prone | yes | no |
3 participants
Auto-generated by the monitor-skills workflow.
This PR bundles all changes produced by the skill generation pipeline:
skill_coverage.json— page coverage stateskill_candidates.json— triage backlogref_repo_change_alerts.json— upstream reference repo alertsrepo_state.json— tracked reference repo commit SHAsagent_skills_config.json— generated skills consumed by the mkdocs pluginThe branch accumulates changes across triage and generate batches;
this single PR gets updated instead of creating new ones.
Merging publishes the skills to the docs site.