feat: relevance ranking, evals, chronicle, embeddings, skill drift-check#1782
Open
BarrettHolien wants to merge 1 commit into
Open
feat: relevance ranking, evals, chronicle, embeddings, skill drift-check#1782BarrettHolien wants to merge 1 commit into
BarrettHolien wants to merge 1 commit into
Conversation
Five features grafting the strongest ideas from tobi/qmd (hybrid-search ranking) and garrytan/gbrain (evals, versioned skills, history) onto graphify's structural spine. Each is a module + CLI + tests. graphify's edge: it fuses structural signals (graph proximity, hub centrality, community) that pure-retrieval tools can't compute. 1. Ranking (graphify/ranking.py): rank query results by Reciprocal Rank Fusion over lexical + structural + optional semantic backends, replacing the degree-only render order. New query flags --top-k, --explain, --semantic; also on the MCP query_graph tool. Shared core: serve._resolve_and_rank / rank_query_nodes. 2. Evals (graphify/evals.py): `graphify bench` scores P@k/recall/MRR/nDCG against a JSONL fixture via the real ranking pipeline. --init scaffolds, --save/ --replay gate regressions (exit 1). 3. Embeddings (graphify/embed.py): opt-in `graphify embed` builds a local sidecar (Ollama nomic-embed-text with search_query:/search_document: prefixes, or sentence-transformers; no API). --semantic fuses cosine similarity and seeds traversal when lexical finds nothing. 4. Chronicle (graphify/chronicle.py): `graphify chronicle OLD NEW` (or --rev via `git show`) diffs two graph snapshots: nodes/edges, god-nodes emerged/vanished, community shifts. API-free. 5. Skill drift (graphify/skill_migrations.py): `graphify skill status|check-update` reports installed-skill-vs-package drift (exit 1 for CI) with a migration registry. Skill fragments document the new query flags + a new analysis.md reference; regenerated + blessed all artifacts (skillgen --check/--audit-coverage pass). Version 0.9.12 -> 0.10.0.
Author
|
Note: this supersedes #1760 (same change set, rebased onto current v8 and re-tested). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Five additive features that graft hybrid-search ranking, eval scaffolding, and history tooling onto graphify's structural spine. Each is a self-contained module + CLI command + tests; no existing behavior changes unless the new flags are used.
Ranking (
graphify/ranking.py) — ranks query results by Reciprocal Rank Fusion over lexical + structural (graph proximity, hub centrality, community) + optional semantic backends, replacing degree-only render order. Newqueryflags--top-k,--explain,--semantic; also exposed on the MCPquery_graphtool via a shared core (serve._resolve_and_rank/rank_query_nodes).Evals (
graphify/evals.py) —graphify benchscores P@k / recall / MRR / nDCG against a JSONL fixture through the real ranking pipeline.--initscaffolds a fixture;--save/--replaygate regressions (exit 1) for CI.Embeddings (
graphify/embed.py) — opt-ingraphify embedbuilds a local sidecar (Ollamanomic-embed-textwithsearch_query:/search_document:prefixes, or sentence-transformers; no API calls).--semanticfuses cosine similarity into ranking and seeds traversal when lexical matching finds nothing.Chronicle (
graphify/chronicle.py) —graphify chronicle OLD NEW(or--revviagit show) diffs two graph snapshots: node/edge deltas, god nodes emerged/vanished, community shifts. API-free.Skill drift (
graphify/skill_migrations.py) —graphify skill status|check-updatereports installed-skill-vs-package drift (exit 1 for CI) with a migration registry.Skill fragments document the new query flags plus a new
analysis.mdreference; all skillgen artifacts regenerated and blessed (skillgen --check/--audit-coveragepass).Testing
v8(4c075f9). The only failures in my environment (8 intest_llm_backends/test_ollama*, missing optionalopenaimodule) reproduce identically on cleanv8, so they're unrelated to this change.test_ranking.py,test_evals.py,test_embed.py,test_chronicle.py,test_skill_migrations.py.Notes for maintainers