Skip to content

Commit abff99c

Browse files
committed
chore(lint): add doc link checker, OSS hygiene ratchets, and CI tool pins
- Add `tools/lint/check_doc_links.py`, `check_doc_links_test.py`, and Bazel targets wired into `tools/lint/precommit_gates.py`, `.github/workflows/lint.yml`, and `tools/README.md`, with shared CommonMark `opens_fence`/`INLINE_CODE_RE` in `check_agent_doc_paths.py`, balanced/angle/reference-link parsing, case-insensitive external/file scheme checks, protocol-relative handling, and `forbidden_url` deduplication. - Add `oss_b_todo_references` and `generic_helper_filenames` ratchets to `check_legacy_imports.py` and `legacy_imports_baseline.json` (plus broadened `HandledError` guidance and unit tests). - Pin `ruff==0.16.4`, `mypy==2.4.1`, `helm` `v3.20.1` (`helm version --short` check), and `Check PR Diff Size` warning telemetry in `.github/workflows/lint.yml`. TAG=agy CONV=ba8411ef-c093-4f45-950a-b3bc1899268c
1 parent f710acb commit abff99c

11 files changed

Lines changed: 614 additions & 15 deletions

‎.github/workflows/lint.yml‎

Lines changed: 24 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -89,7 +89,7 @@ jobs:
8989
- name: 'Install Ruff'
9090
run: |
9191
if ! command -v ruff &>/dev/null; then
92-
pip install --no-cache-dir ruff
92+
pip install --no-cache-dir ruff==0.16.4
9393
fi
9494
9595
- name: 'Run Ruff Lint Check'
@@ -101,7 +101,7 @@ jobs:
101101
- name: 'Install Mypy'
102102
run: |
103103
if ! command -v mypy &>/dev/null; then
104-
pip install --no-cache-dir mypy
104+
pip install --no-cache-dir mypy==2.4.1
105105
fi
106106
107107
- name: 'Run Mypy Static Type Check'
@@ -169,6 +169,12 @@ jobs:
169169
run: |
170170
python3 tools/lint/check_agent_doc_paths.py
171171
172+
- name: 'Check Documentation Links & OSS URLs'
173+
env:
174+
XMC_BYPASS_VENV: '1'
175+
run: |
176+
python3 tools/lint/check_doc_links.py
177+
172178
- name: 'Check UI & Workflow File Naming'
173179
env:
174180
XMC_BYPASS_VENV: '1'
@@ -352,9 +358,9 @@ jobs:
352358
- name: 'Install Helm'
353359
if: steps.changes.outputs.helm == 'true'
354360
run: |
355-
if ! command -v helm &>/dev/null; then
361+
if ! helm version --short 2>/dev/null | grep -q '^v3\.'; then
356362
mkdir -p "${RUNNER_TEMP}/helm"
357-
curl -sSfL "https://get.helm.sh/helm-v3.15.2-linux-amd64.tar.gz" \
363+
curl -sSfL "https://get.helm.sh/helm-v3.20.1-linux-amd64.tar.gz" \
358364
| tar -xz -C "${RUNNER_TEMP}/helm" --strip-components=1 linux-amd64/helm
359365
chmod +x "${RUNNER_TEMP}/helm/helm"
360366
echo "${RUNNER_TEMP}/helm" >> "${GITHUB_PATH}"
@@ -377,3 +383,17 @@ jobs:
377383
severity: 'HIGH,CRITICAL'
378384
scan-ref: 'deploy/'
379385
trivyignores: '.trivyignore'
386+
387+
# 8. PR Diff Size Telemetry
388+
- name: 'Check PR Diff Size'
389+
if: github.event_name == 'pull_request'
390+
env:
391+
XMC_BYPASS_VENV: '1'
392+
BASE_REF: ${{ github.base_ref }}
393+
run: |
394+
DIFF_JSON="$(python3 tools/analyze_diff_size.py --base-ref "origin/${BASE_REF}...")"
395+
echo "${DIFF_JSON}"
396+
if [ "$(echo "${DIFF_JSON}" | jq -r '.result.threshold_exceeded')" = 'true' ]; then
397+
REASONS="$(echo "${DIFF_JSON}" | jq -r '.result.exceeded_reasons | join("; ")')"
398+
echo "::warning::PR diff size exceeds review threshold: ${REASONS}"
399+
fi

‎tools/README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -84,8 +84,9 @@ Tooling and automation for release metadata extraction, SemVer calculations, and
8484
### 7. `tools/lint/` (Static Analyzers & Linters)
8585
Offline syntax walkers and static validation tooling:
8686
- [`check_go_determinism`](lint/check_go_determinism/): Ultra-fast (<50ms) Go AST analyzer checking Go test files for `os.Setenv` usage and arbitrary second-level `time.Sleep` calls.
87-
- [`check_legacy_imports.py`](lint/check_legacy_imports.py): Ratchet lint preventing regression of google3 migration leftovers (legacy Go import roots, unresolved `xmanager_cloud/...` module paths, `compat_go_*` targets, `gazelle:resolve` overrides, `third_party/` proto paths, google3 Helm flags, legacy UI/Python paths) against the zeroed [`legacy_imports_baseline.json`](lint/legacy_imports_baseline.json). Fails when any count grows above zero. Runs in pre-commit and the lint CI job. Tracking: [#1066](https://github.com/google/xmc/issues/1066).
87+
- [`check_legacy_imports.py`](lint/check_legacy_imports.py): Ratchet lint preventing regression of google3 migration leftovers (legacy Go import roots, unresolved `xmanager_cloud/...` module paths, `compat_go_*` targets, `gazelle:resolve` overrides, `third_party/` proto paths, google3 Helm flags, legacy UI/Python paths), OSS `TODO(b/...)` IDs under `api/`, `cmd/`, `pkg/`, and `sdks/`, and new generic helper/util/misc/common filenames against [`legacy_imports_baseline.json`](lint/legacy_imports_baseline.json). Fails when any count grows above baseline. Runs in pre-commit and the lint CI job. Tracking: [#1066](https://github.com/google/xmc/issues/1066).
8888
- [`check_agent_doc_paths.py`](lint/check_agent_doc_paths.py): Fails when agent docs (`.agents/**/*.md`, `AGENTS.md`) reference repo paths in code spans or fenced blocks that are not tracked files or directories. Placeholders (`<name>`, `{a,b}`, `$VAR`, `*`, `[`, `...`) only require their directory prefix to exist; runtime-created and illustrative paths live in its `ALLOWLIST` with reasons, and unreferenced entries fail. Runs in pre-commit (when agent docs are staged or files are deleted/renamed; always in `--check` mode) and the lint CI job.
89+
- [`check_doc_links.py`](lint/check_doc_links.py): Validates relative Markdown links (`[label](path)`) against tracked files and directories and rejects non-code `file://`, `corp.google.com`, `http(s)://go/...`, and `//go/...` URLs across all tracked Markdown docs outside `.agents/`. Runs in pre-commit (when non-`.agents/` Markdown docs or doc linters are staged, or files are deleted/renamed; always in `--check` mode) and the lint CI job.
8990
- [`check_file_naming.py`](lint/check_file_naming.py): Enforces UI and workflow file-naming conventions across tracked repository files (`*.ng.html` for HTML files under `cmd/ui/`—templates under `cmd/ui/app/` with `cmd/ui/index.html` allowlisted—`*_test.ts` for Karma unit tests under `cmd/ui/app/` while rejecting `*.spec.ts`/`*_spec.ts`/`*.test.ts` repo-wide outside `cmd/ui/screenshot_tests/`, `*.screenshot.spec.ts` for Playwright visual specs under `cmd/ui/screenshot_tests/`, and top-level `*.test.js` for Node workflow tests under `.github/scripts/`). Runs in pre-commit and the lint CI job.
9091
- [`check_ui_selector_contract.py`](lint/check_ui_selector_contract.py): Parses every tracked `*.py` under `tools/nightly_tests/` (fails on any syntax error; `EXCLUDED_SOURCES` opts files out) and lints each file containing selectors (`*_SEL`/`*_SEL_TMPL` constants, `.locator()`/`.wait_for_selector()` string arguments, or `has_text=` literals; reported as `sources`). Fails when selectors reference `[data-testid="..."]` values or custom tags that no `cmd/ui/app` template or component selector declares (Material `mat-`/`mdc-`/`cdk-` names exempt), when selectors use custom classes or non-standard attributes without `# ui-contract: <reason>`, when code references a `ui_css_selectors` constant that does not exist, or when selectors match user-facing copy via `:has-text(...)`/`has_text=` without a `# ui-contract: <reason>` comment on the same line or a standalone comment line above. Runs on every commit in pre-commit and in the lint CI job (~0.2s; no path gate, so the source list is not duplicated).
9192
- [`precommit_gates.py`](lint/precommit_gates.py): Declarative runner (`--mode fix|check`) and shared git/execution primitives for pre-commit Python lint gates (`check_legacy_imports`, `check_examples_coverage`, `check_matrix_contract`, `check_agent_doc_paths`, `check_file_naming`, `check_ui_selector_contract`).

‎tools/lint/BUILD.bazel‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,27 @@ py_test(
3737
],
3838
)
3939

40+
py_library(
41+
name = "check_doc_links_lib",
42+
srcs = ["check_doc_links.py"],
43+
imports = ["../.."],
44+
deps = [
45+
":check_agent_doc_paths_lib",
46+
"//tools/common:tool_utils",
47+
],
48+
)
49+
50+
py_test(
51+
name = "check_doc_links_test",
52+
srcs = ["check_doc_links_test.py"],
53+
imports = ["../.."],
54+
deps = [
55+
":check_agent_doc_paths_lib",
56+
":check_doc_links_lib",
57+
"//tools/common:tool_utils",
58+
],
59+
)
60+
4061
py_library(
4162
name = "check_file_naming_lib",
4263
srcs = ["check_file_naming.py"],

‎tools/lint/check_agent_doc_paths.py‎

Lines changed: 16 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -108,8 +108,8 @@ def build_path_token_re(roots: Iterable[str]) -> re.Pattern[str]:
108108

109109

110110
_PATH_TOKEN_RE = build_path_token_re(PATH_ROOTS)
111-
_INLINE_CODE_RE = re.compile(r"(`+)(.+?)\1")
112-
_FENCE_RE = re.compile(r"^\s*(`{3,}|~{3,})(.*)$")
111+
INLINE_CODE_RE = re.compile(r"(?<!`)(`+)(?!`)(.+?)(?<!`)\1(?!`)")
112+
_FENCE_RE = re.compile(r"^\s*(?:(`{3,})([^`]*)|(~{3,})(.*))$")
113113
_PLACEHOLDER_RE = re.compile(r"[<{$*\[]|\.\.\.")
114114
_TRAILING_PUNCTUATION = "."
115115

@@ -132,6 +132,14 @@ def is_agent_doc(path: str) -> bool:
132132
)
133133

134134

135+
def opens_fence(line: str) -> str:
136+
"""Returns the opening fence marker when a line opens a CommonMark fenced block."""
137+
fence = _FENCE_RE.match(line)
138+
if not fence:
139+
return ""
140+
return fence.group(1) or fence.group(3)
141+
142+
135143
def closes_fence(line: str, opening: str) -> bool:
136144
"""Returns whether a line closes a fenced block opened with `opening`.
137145
@@ -142,7 +150,8 @@ def closes_fence(line: str, opening: str) -> bool:
142150
fence = _FENCE_RE.match(line)
143151
if not fence:
144152
return False
145-
run, rest = fence.group(1), fence.group(2)
153+
run = fence.group(1) or fence.group(3)
154+
rest = fence.group(2) if fence.group(1) is not None else fence.group(4)
146155
return run[0] == opening[0] and len(run) >= len(opening) and not rest.strip()
147156

148157

@@ -156,11 +165,11 @@ def iter_code_segments(text: str) -> Iterator[tuple[int, str]]:
156165
else:
157166
yield lineno, line
158167
continue
159-
fence = _FENCE_RE.match(line)
160-
if fence:
161-
opening = fence.group(1)
168+
opened = opens_fence(line)
169+
if opened:
170+
opening = opened
162171
continue
163-
for match in _INLINE_CODE_RE.finditer(line):
172+
for match in INLINE_CODE_RE.finditer(line):
164173
yield lineno, match.group(2)
165174

166175

‎tools/lint/check_doc_links.py‎

Lines changed: 216 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,216 @@
1+
#!/usr/bin/env python3
2+
# Copyright 2026 Google LLC
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
"""Lint verifying relative links and OSS URL hygiene in repository Markdown docs.
17+
18+
Scans tracked `*.md` files outside `.agents/` (which `check_agent_doc_paths.py`
19+
checks separately) and fails when:
20+
- A relative Markdown link target does not exist in the git index or escapes the repo.
21+
- A non-code Markdown line contains a `file://` URI, `corp.google.com` host, or
22+
`http(s)://go/...` / `//go/...` corporate shortlink.
23+
"""
24+
25+
from __future__ import annotations
26+
27+
import argparse
28+
import dataclasses
29+
import posixpath
30+
import re
31+
import sys
32+
from collections.abc import Iterator
33+
from pathlib import Path
34+
from typing import Any
35+
from urllib.parse import unquote
36+
37+
_REPO_ROOT = Path(__file__).resolve().parents[2]
38+
if str(_REPO_ROOT) not in sys.path:
39+
sys.path.insert(0, str(_REPO_ROOT))
40+
41+
from tools.common.tool_utils import HandledError, run_tool # noqa: E402
42+
from tools.lint.check_agent_doc_paths import ( # noqa: E402
43+
INLINE_CODE_RE,
44+
closes_fence,
45+
list_tracked_files,
46+
opens_fence,
47+
tracked_paths,
48+
)
49+
50+
_LINK_TAIL_RE = re.compile(r'^\s*(?:(?:"[^"]*"|\'[^\']*\')\s*)?\)')
51+
_REF_DEF_RE = re.compile(r"^\s{0,3}\[[^\]]+\]:\s*(<[^>]+>|[^\s]+)")
52+
_SCHEME_RE = re.compile(r"^[a-z][a-z0-9+.-]+:", re.IGNORECASE)
53+
_FORBIDDEN_URL_RE = re.compile(
54+
r"file://[^\s)>'\"]+|[A-Za-z0-9.-]*corp\.google\.com\b|(?:https?://|(?<![A-Za-z0-9:/])//)go/[A-Za-z0-9_./-]+",
55+
re.IGNORECASE,
56+
)
57+
58+
59+
@dataclasses.dataclass(frozen=True)
60+
class DocViolation:
61+
"""A broken relative link or forbidden internal URL in a Markdown document."""
62+
63+
doc: str
64+
line: int
65+
kind: str
66+
target: str
67+
68+
69+
def is_checked_doc(path: str) -> bool:
70+
"""Return whether a tracked path is a Markdown doc covered by this linter."""
71+
return path.endswith(".md") and not path.startswith(".agents/")
72+
73+
74+
def iter_prose_lines(text: str) -> Iterator[tuple[int, str]]:
75+
"""Yield (1-based line number, line without inline code) outside fenced blocks."""
76+
opening = ""
77+
for lineno, line in enumerate(text.splitlines(), start=1):
78+
if opening:
79+
if closes_fence(line, opening):
80+
opening = ""
81+
continue
82+
opened = opens_fence(line)
83+
if opened:
84+
opening = opened
85+
continue
86+
yield lineno, INLINE_CODE_RE.sub("", line)
87+
88+
89+
def _parse_link_destination(prose: str, start: int) -> str | None:
90+
"""Parse an inline Markdown link destination immediately following `](`."""
91+
i = start
92+
while i < len(prose) and prose[i] in " \t":
93+
i += 1
94+
if i < len(prose) and prose[i] == "<":
95+
end = prose.find(">", i + 1)
96+
if end == -1 or not _LINK_TAIL_RE.match(prose[end + 1 :]):
97+
return None
98+
return prose[i + 1 : end]
99+
depth = 0
100+
j = i
101+
while j < len(prose):
102+
ch = prose[j]
103+
if ch in " \t":
104+
break
105+
if ch == "(":
106+
depth += 1
107+
elif ch == ")":
108+
if depth == 0:
109+
return prose[i:j] if j > i else None
110+
depth -= 1
111+
j += 1
112+
if j > i and depth == 0 and _LINK_TAIL_RE.match(prose[j:]):
113+
return prose[i:j]
114+
return None
115+
116+
117+
def extract_link_targets(prose: str) -> list[str]:
118+
"""Extract inline link, linked-image, and reference-definition targets from a line."""
119+
targets: list[str] = []
120+
pos = 0
121+
while (idx := prose.find("](", pos)) != -1:
122+
dest = _parse_link_destination(prose, idx + 2)
123+
if dest is not None:
124+
targets.append(dest)
125+
pos = idx + 2
126+
ref_match = _REF_DEF_RE.match(prose)
127+
if ref_match:
128+
targets.append(ref_match.group(1))
129+
return targets
130+
131+
132+
def resolve_repo_target(doc: str, raw_target: str) -> tuple[str, str | None]:
133+
"""Resolve a Markdown link target relative to doc.
134+
135+
Returns:
136+
Tuple of (resolved_repo_path, error_kind) where error_kind is None if
137+
the link is external, anchor-only, or resolves cleanly within the repo.
138+
"""
139+
stripped = raw_target.strip("<>")
140+
if not stripped or stripped.startswith("#"):
141+
return "", None
142+
if stripped.lower().startswith("file://"):
143+
return stripped.rstrip(".,;:"), "forbidden_url"
144+
if stripped.startswith("//") or _SCHEME_RE.match(stripped):
145+
return "", None
146+
path_part = unquote(stripped.split("#", 1)[0].split("?", 1)[0])
147+
if not path_part:
148+
return "", None
149+
if path_part.startswith("/"):
150+
joined = posixpath.normpath(path_part.lstrip("/"))
151+
else:
152+
doc_dir = posixpath.dirname(doc)
153+
joined = posixpath.normpath(posixpath.join(doc_dir, path_part) if doc_dir else path_part)
154+
if joined == ".." or joined.startswith("../"):
155+
return joined, "escapes_repo"
156+
return ("" if joined == "." else joined), None
157+
158+
159+
def find_line_violations(
160+
doc: str, lineno: int, prose: str, existing: frozenset[str]
161+
) -> list[DocViolation]:
162+
"""Collect broken-link and forbidden-URL violations from a single prose line."""
163+
violations: list[DocViolation] = []
164+
for raw_target in extract_link_targets(prose):
165+
resolved, err_kind = resolve_repo_target(doc, raw_target)
166+
if err_kind is not None:
167+
violations.append(DocViolation(doc=doc, line=lineno, kind=err_kind, target=resolved))
168+
elif resolved and resolved not in existing:
169+
violations.append(
170+
DocViolation(doc=doc, line=lineno, kind="missing_target", target=resolved)
171+
)
172+
for url_match in _FORBIDDEN_URL_RE.finditer(prose):
173+
url = url_match.group(0).rstrip(".,;:")
174+
if not any(v.target == url and v.line == lineno for v in violations):
175+
violations.append(DocViolation(doc=doc, line=lineno, kind="forbidden_url", target=url))
176+
return violations
177+
178+
179+
def scan_docs(repo_root: Path, docs: list[str], existing: frozenset[str]) -> list[DocViolation]:
180+
"""Scan all tracked Markdown docs outside `.agents/` for link and URL issues."""
181+
violations: list[DocViolation] = []
182+
for doc in docs:
183+
doc_path = repo_root / doc
184+
if not doc_path.is_file():
185+
continue
186+
text = doc_path.read_text(encoding="utf-8", errors="replace")
187+
for lineno, prose in iter_prose_lines(text):
188+
violations.extend(find_line_violations(doc, lineno, prose, existing))
189+
return violations
190+
191+
192+
def check_doc_links(args: argparse.Namespace) -> dict[str, Any]:
193+
"""Validate relative links and public URL hygiene in repository Markdown docs."""
194+
repo_root = Path(args.repo_root).resolve()
195+
files = list_tracked_files(repo_root)
196+
docs = sorted(f for f in files if is_checked_doc(f))
197+
violations = scan_docs(repo_root, docs, tracked_paths(files))
198+
if violations:
199+
raise HandledError(
200+
f"Found {len(violations)} broken Markdown link(s) or internal URL(s).",
201+
"Fix broken relative Markdown links against tracked files/directories and remove "
202+
"local file:// or corporate URLs.",
203+
details={"violations": [dataclasses.asdict(v) for v in violations]},
204+
)
205+
return {"docs_scanned": len(docs), "violations": 0}
206+
207+
208+
def main() -> None:
209+
"""CLI entrypoint."""
210+
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
211+
parser.add_argument("--repo-root", default=str(_REPO_ROOT), help="Repository root.")
212+
run_tool(parser, check_doc_links)
213+
214+
215+
if __name__ == "__main__":
216+
main()

0 commit comments

Comments
 (0)