Skip to content

feat: relevance ranking, evals, chronicle, embeddings, skill drift-check#1782

Open
BarrettHolien wants to merge 1 commit into
Graphify-Labs:v8from
HolienTech:feat/ranking-evals-chronicle-embeddings
Open

feat: relevance ranking, evals, chronicle, embeddings, skill drift-check#1782
BarrettHolien wants to merge 1 commit into
Graphify-Labs:v8from
HolienTech:feat/ranking-evals-chronicle-embeddings

Conversation

@BarrettHolien

Copy link
Copy Markdown

Summary

Five additive features that graft hybrid-search ranking, eval scaffolding, and history tooling onto graphify's structural spine. Each is a self-contained module + CLI command + tests; no existing behavior changes unless the new flags are used.

  1. Ranking (graphify/ranking.py) — ranks query results by Reciprocal Rank Fusion over lexical + structural (graph proximity, hub centrality, community) + optional semantic backends, replacing degree-only render order. New query flags --top-k, --explain, --semantic; also exposed on the MCP query_graph tool via a shared core (serve._resolve_and_rank / rank_query_nodes).

  2. Evals (graphify/evals.py) — graphify bench scores P@k / recall / MRR / nDCG against a JSONL fixture through the real ranking pipeline. --init scaffolds a fixture; --save / --replay gate regressions (exit 1) for CI.

  3. Embeddings (graphify/embed.py) — opt-in graphify embed builds a local sidecar (Ollama nomic-embed-text with search_query:/search_document: prefixes, or sentence-transformers; no API calls). --semantic fuses cosine similarity into ranking and seeds traversal when lexical matching finds nothing.

  4. Chronicle (graphify/chronicle.py) — graphify chronicle OLD NEW (or --rev via git show) diffs two graph snapshots: node/edge deltas, god nodes emerged/vanished, community shifts. API-free.

  5. Skill drift (graphify/skill_migrations.py) — graphify skill status|check-update reports installed-skill-vs-package drift (exit 1 for CI) with a migration registry.

Skill fragments document the new query flags plus a new analysis.md reference; all skillgen artifacts regenerated and blessed (skillgen --check / --audit-coverage pass).

Testing

  • 3,161 passed, 31 skipped on this branch rebased onto current v8 (4c075f9). The only failures in my environment (8 in test_llm_backends/test_ollama*, missing optional openai module) reproduce identically on clean v8, so they're unrelated to this change.
  • New test files: test_ranking.py, test_evals.py, test_embed.py, test_chronicle.py, test_skill_migrations.py.

Notes for maintainers

  • The commit bumps version 0.9.12 → 0.10.0 and adds a CHANGELOG entry — happy to drop those if you prefer to manage releases yourselves.
  • This is intentionally one commit of five cohesive features (ranking is the base the others build on), but I'm glad to split it into smaller stacked PRs if that's easier to review.

Five features grafting the strongest ideas from tobi/qmd (hybrid-search ranking)
and garrytan/gbrain (evals, versioned skills, history) onto graphify's structural
spine. Each is a module + CLI + tests. graphify's edge: it fuses structural
signals (graph proximity, hub centrality, community) that pure-retrieval tools
can't compute.

1. Ranking (graphify/ranking.py): rank query results by Reciprocal Rank Fusion
   over lexical + structural + optional semantic backends, replacing the
   degree-only render order. New query flags --top-k, --explain, --semantic; also
   on the MCP query_graph tool. Shared core: serve._resolve_and_rank /
   rank_query_nodes.

2. Evals (graphify/evals.py): `graphify bench` scores P@k/recall/MRR/nDCG against
   a JSONL fixture via the real ranking pipeline. --init scaffolds, --save/
   --replay gate regressions (exit 1).

3. Embeddings (graphify/embed.py): opt-in `graphify embed` builds a local sidecar
   (Ollama nomic-embed-text with search_query:/search_document: prefixes, or
   sentence-transformers; no API). --semantic fuses cosine similarity and seeds
   traversal when lexical finds nothing.

4. Chronicle (graphify/chronicle.py): `graphify chronicle OLD NEW` (or --rev via
   `git show`) diffs two graph snapshots: nodes/edges, god-nodes emerged/vanished,
   community shifts. API-free.

5. Skill drift (graphify/skill_migrations.py): `graphify skill status|check-update`
   reports installed-skill-vs-package drift (exit 1 for CI) with a migration
   registry.

Skill fragments document the new query flags + a new analysis.md reference;
regenerated + blessed all artifacts (skillgen --check/--audit-coverage pass).
Version 0.9.12 -> 0.10.0.
@BarrettHolien

Copy link
Copy Markdown
Author

Note: this supersedes #1760 (same change set, rebased onto current v8 and re-tested).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant