fix(docs-review): check related docs by name and directory, not just identifier grep

[csoceanu]

Summary

The docs-review skill finds stale docs by grepping for identifiers from the diff. This misses docs that describe the same feature area but don't mention any specific changed identifier — common when a PR adds new behavior rather than renaming existing behavior.

This PR adds a step to check whether documentation files covering the same feature exist based on file name and directory structure, and evaluate them as candidates even without grep matches.

Changes

• docs-review/SKILL.md step 4: after the identifier grep, also check for related doc files by name/directory match
• docs-review/SKILL.md step 5: handle candidates without grep hits by reading section headings in the quick-scan pass
• docs-currency.md: align the sub-agent instructions with the updated skill

Context

An impact analysis of 111 merged PRs found that the skill missed stale docs in 7 PRs (6% false negative rate). In most of these cases, the PR added new user-visible behavior and the related doc file wasn't found because the new identifiers didn't exist in existing docs. This change addresses that gap by matching code files to doc files by name and directory structure, not just identifier grep.

Same change as fullsend-ai/fullsend#3233, opened here per @rh-hemartin's request since agents are migrating to this repo.

[qodo-code-review]

PR Summary by Qodo

Improve docs-review stale-doc detection with name/directory doc matching

✨ Enhancement 📝 Documentation 🕐 10-20 Minutes

AI Description

• Extend docs-review candidate discovery with name/directory matching for related docs.
• Update quick-scan to evaluate candidates without grep hits via section headings.
• Align docs-currency sub-agent instructions with the revised docs-review workflow.

Diagram

graph TD
  A["PR diff"] --> B["Extract identifiers"] --> C["Grep docs"] --> D{"Any matches?"}
  D -->|"Yes"| E["Candidate docs"] --> G["Quick scan"] --> H["Verdicts"]
  D -->|"No / partial"| F["Name/dir match"] --> E

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Maintain an explicit code→docs mapping (manifest/tags)

• ➕ More precise than heuristics; fewer false positives than name/dir matching
• ➕ Easier to enforce/validate in CI once conventions exist
• ➖ Requires ongoing maintenance and conventions adoption across teams
• ➖ Up-front work to bootstrap mappings for existing areas

2. Semantic similarity search over docs (embeddings/LLM)

• ➕ Can detect related docs even when names/paths diverge
• ➕ Potentially better recall for conceptual changes and new behavior
• ➖ Higher operational complexity and cost (indexing, model/runtime)
• ➖ Harder to make deterministic and reviewable than simple heuristics

3. Extend grep strategy with additional signal (API surface, UI strings)

• ➕ Keeps tooling simple and local (still text-based)
• ➕ Improves recall without relying on directory conventions
• ➖ Still misses purely conceptual docs; can increase noise
• ➖ More engineering effort than the proposed incremental heuristic

Recommendation: The PR’s approach (augmenting identifier grep with name/directory-based doc discovery, then using headings for quick evaluation) is a pragmatic, low-cost recall improvement that fits the existing workflow and remains deterministic. Consider an explicit mapping or semantic search only if false negatives remain significant or directory conventions aren’t reliable enough in practice.

Files changed (2) +12 / -3

Documentation (2) +12 / -3

SKILL.mdAdd name/directory related-doc discovery and headings-based quick scan +9/-2

Add name/directory related-doc discovery and headings-based quick scan

• Expands candidate doc selection to include docs inferred by filename/directory similarity to changed code files (even without identifier grep hits). Updates the quick-scan guidance to read section headings for these non-grep candidates before recording a verdict.

skills/docs-review/SKILL.md

docs-currency.mdAlign docs-currency instructions with new related-doc matching step +3/-1

Align docs-currency instructions with new related-doc matching step

• Updates the sub-agent guidance to explicitly include name/directory-based related-doc checks as a complement to identifier-reference grepping, to better catch docs that need updates for new behavior.

skills/pr-review/sub-agents/docs-currency.md

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📜 Skill insights (1)

Context used

✅ Compliance rules (platform): 55 rules

✅ Skills: 4 invoked

  code-review

  code-implementation

  pr-review

  docs-review

1. Protected skills/ files modified 📜 Skill insight § Compliance

Description

This PR modifies files under skills/, which is a protected governance/infrastructure path and must
not be auto-approved; it requires explicit human review. Ensure the PR has appropriate
authorization/justification recorded (ideally via a linked issue) before merging.

Code

skills/docs-review/SKILL.md[R127-130]

+Additionally, for each changed code file, check whether a
+documentation file covering the same feature exists based on file
+name and directory structure. If one exists, is not already in the
+candidate list, and is not modified in this PR, add it.

Relevance

⭐⭐ Medium

Similar compliance suggestions on protected paths/authorization in PRs #25/#29; no explicit
skills/-protected precedent found.

PR-#25
PR-#29

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The compliance checklist marks any modifications under protected paths (including skills/) as
requiring a finding and human approval. This PR changes skills/docs-review/SKILL.md and
skills/pr-review/sub-agents/docs-currency.md, which are both inside skills/.

skills/docs-review/SKILL.md[127-139]
skills/pr-review/sub-agents/docs-currency.md[18-22]
Skill: pr-review

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Protected governance/infrastructure paths were modified under `skills/`, which must be flagged for human review and should include explicit authorization/justification (preferably a linked issue) to avoid accidental auto-approval.

## Issue Context
Compliance requires raising a finding whenever protected paths are modified. If there is no linked issue, add one (or otherwise document explicit authorization) in the PR description.

## Fix Focus Areas
- skills/docs-review/SKILL.md[127-139]
- skills/pr-review/sub-agents/docs-currency.md[18-22]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Headings scan underspecified ✗ Dismissed 🐞 Bug ☼ Reliability

Description

Step 5’s quick-scan instruction for candidates without grep hits says to “read the doc’s section
headings,” but the skill explicitly includes .rst and .adoc docs and does not describe how to
identify headings in those formats.

Code

skills/docs-review/SKILL.md[R136-138]

+in context). For candidates added by name/directory match without
+grep hits, read the doc's section headings to determine if it
+describes behavior changed in this PR. Record a verdict for every

Relevance

⭐⭐ Medium

No historical evidence found for specifying heading-detection rules across .rst/.adoc in skills
instructions.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The skill’s doc discovery and grep explicitly include .rst and .adoc, but the new quick-scan
instruction only says “read section headings” without explaining how to do that across all supported
formats.

skills/docs-review/SKILL.md[79-88]
skills/docs-review/SKILL.md[109-110]
skills/docs-review/SKILL.md[134-138]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The quick-scan path for candidates added without grep hits relies on reading “section headings,” but the skill’s doc set includes `.md`, `.rst`, and `.adoc`. Without format-specific guidance, reviewers may miss headings or apply inconsistent scanning.

## Issue Context
Step 3/4 explicitly include `.rst` and `.adoc` files in the doc corpus and grep script, so step 5 should explain how to quickly find headings across those formats.

## Fix Focus Areas
- skills/docs-review/SKILL.md[87-88]
- skills/docs-review/SKILL.md[109-110]
- skills/docs-review/SKILL.md[134-138]

## Suggested fix
Update step 5 to define a quick heading-extraction approach per format, e.g.:
- Markdown: lines starting with `#`
- AsciiDoc: lines starting with `=`
- reStructuredText: detect common underline/overline patterns (or instruct to read the first ~50 lines / TOC section)

Optionally include a simple command snippet per type (e.g., `grep -nE '^(#|=)'` for md/adoc) and a fallback (“if heading syntax is unclear, read the intro + TOC”).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Doc match heuristic unclear 🐞 Bug ⚙ Maintainability

Description

Docs-review step 4 adds a requirement to include candidate docs by “file name and directory
structure” but does not define a matching heuristic (mapping rules, tie-breaking, or scope), which
can produce inconsistent candidate sets across reviewers/runs.

Code

skills/docs-review/SKILL.md[R127-130]

+Additionally, for each changed code file, check whether a
+documentation file covering the same feature exists based on file
+name and directory structure. If one exists, is not already in the
+candidate list, and is not modified in this PR, add it.

Relevance

⭐⭐ Medium

No historical evidence found requiring a defined name/directory doc-matching heuristic in skills
docs.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The new step requires name/directory-based doc matching but provides no concrete rule for how to do
that mapping. The same vague requirement is repeated in the docs-currency sub-agent instructions,
increasing the chance of inconsistent application.

skills/docs-review/SKILL.md[123-130]
skills/pr-review/sub-agents/docs-currency.md[18-22]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Step 4 introduces “related docs by name/directory” candidate selection, but the instructions don’t specify how to perform the match (how to map code paths to doc paths, how to handle multiple matches, what directories to consider, etc.). This ambiguity makes the skill harder to apply consistently.

## Issue Context
The docs-review process is used verbatim by the `docs-currency` sub-agent; unclear heuristics can lead to inconsistent candidate lists and noisy or missed findings.

## Fix Focus Areas
- skills/docs-review/SKILL.md[127-130]
- skills/pr-review/sub-agents/docs-currency.md[18-22]

## Suggested fix
Add a short, explicit heuristic with examples and tie-breakers, e.g.:
- Normalize changed code file basename (drop extension, `_test`, etc.)
- Prefer docs in the nearest feature directory (e.g., `docs/<feature>/`, or same top-level component dir)
- Match on basename similarity (`foo/bar/baz.ts` → `docs/**/baz*.md|.rst|.adoc`)
- If multiple candidates match, pick the closest path distance, and cap to N (e.g., 1–3) to prevent candidate explosion.
- If no reasonable match is found, explicitly say “skip name/directory match for this code file.”

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[rh-hemartin]

LGTM

[rh-hemartin]

I'll let this sit for 24 more hours to see if you want to submit some corrections or someone else has anything to say.

[github-actions]

Functional tests are running

Authorization passed for this commit. See the Functional Tests workflow for results.

[rh-hemartin]

Hello! @csoceanu you need to decide if the Qodo feedback is relevant and resolve the threads in order for me to merge. (I rebased to get you the functional tests running)

[ben-alkov]

LGTM 🚢, and agree with @rh-hemartin ☝️

[csoceanu]

Thanks @rh-hemartin @ben-alkov ! Resolved all 3 Qodo threads - none require code changes.

[rh-hemartin]

Thanks for your contribution!

[fullsend-ai-retro]

🤖 Finished Retro · ✅ Success · Started 7:12 AM UTC · Completed 7:23 AM UTC
Commit: a7f35a6 · View workflow run →

[fullsend-ai-retro]

PR #55 was a well-motivated, small (12 additions, 3 deletions) human-authored contribution from an external contributor that improved the docs-review skill's recall by adding name/directory-based document discovery alongside the existing identifier grep. The change was backed by data (6% false negative rate from 111 merged PRs) and required zero rework. Two human reviewers approved within 2 days, but the PR took 5 days to merge due to a 2-day gap waiting for the contributor to resolve bot review threads. The fullsend review agent never ran because the dispatch requires PR author write permission for auto-triggering on fork PRs, and no maintainer manually triggered /fs-review or applied ready-for-review. Only Qodo reviewed, producing 1 valid procedural finding and 2 dismissed findings that mischaracterized LLM agent instructions as needing rigid deterministic specifications. The PR also modifies skill behavior without eval coverage — only triage evals exist in this repo.

Proposals filed

• Dispatch should trigger review agent for fork PRs when ok-to-test label is applied (in fullsend-ai/fullsend)
• Add docs-review eval cases covering name/directory-based document discovery (in fullsend-ai/agents)