Skip to content

Commit ab6fbd7

Browse files
committed
chore(lint): add doc link checker, OSS hygiene ratchets, and CI tool pins
- Add `tools/lint/check_doc_links.py`, `check_doc_links_test.py`, and Bazel targets wired into `.githooks/pre-commit`, `.github/workflows/lint.yml`, and `tools/README.md`, with shared CommonMark `opens_fence`/`INLINE_CODE_RE` in `check_agent_doc_paths.py`, balanced/angle/reference-link parsing, case-insensitive external/file scheme checks, protocol-relative handling, and `forbidden_url` deduplication. - Add `oss_b_todo_references` and `generic_helper_filenames` ratchets to `check_legacy_imports.py` and `legacy_imports_baseline.json` (plus broadened `HandledError` guidance and unit tests). - Pin `ruff==0.16.4`, `mypy==2.4.1`, `helm` `v3.20.1`, and `Check PR Diff Size` warning telemetry in `.github/workflows/lint.yml`. TAG=agy CONV=ba8411ef-c093-4f45-950a-b3bc1899268c
1 parent b704614 commit ab6fbd7

10 files changed

Lines changed: 603 additions & 16 deletions

‎.githooks/pre-commit‎

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -187,9 +187,8 @@ check_matrix_contract() {
187187
}
188188

189189
check_agent_doc_paths() {
190-
# In fix mode, only run when agent docs are staged, or when staged deletes or
191-
# renames could orphan a path an agent doc references, to keep the hook fast.
192-
if [ "${MODE}" = "fix" ] && ! staged_paths_match AM '^\.agents/.*\.md$|(^|/)AGENTS\.md$' && ! staged_paths_match D '.'; then
190+
# In fix mode, only run when agent docs, its linter, or staged deletes/renames change.
191+
if [ "${MODE}" = "fix" ] && ! staged_paths_match AM '^\.agents/.*\.md$|(^|/)AGENTS\.md$|^tools/lint/check_agent_doc_paths\.py$' && ! staged_paths_match D '.'; then
193192
return
194193
fi
195194
echo "--> Checking Agent Doc Path References..."
@@ -199,6 +198,17 @@ check_agent_doc_paths() {
199198
fi
200199
}
201200

201+
check_doc_links() {
202+
if [ "${MODE}" = "fix" ] && ! staged_paths_match AM '\.md$|^tools/lint/check_(doc_links|agent_doc_paths)\.py$' && ! staged_paths_match D '.'; then
203+
return
204+
fi
205+
echo "--> Checking Documentation Links & OSS URLs..."
206+
if ! XMC_BYPASS_VENV=1 python3 tools/lint/check_doc_links.py >/dev/null; then
207+
echo "❌ Markdown docs contain broken relative links or forbidden internal URLs (see JSON error above)."
208+
FAILED=1
209+
fi
210+
}
211+
202212
check_file_naming() {
203213
echo "--> Checking UI & Workflow File Naming Conventions..."
204214
if ! XMC_BYPASS_VENV=1 python3 tools/lint/check_file_naming.py >/dev/null; then
@@ -756,6 +766,7 @@ main() {
756766
check_examples_coverage
757767
check_matrix_contract
758768
check_agent_doc_paths
769+
check_doc_links
759770
check_file_naming
760771
check_ui_selector_contract
761772
check_ui

‎.github/workflows/lint.yml‎

Lines changed: 23 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -89,7 +89,7 @@ jobs:
8989
- name: 'Install Ruff'
9090
run: |
9191
if ! command -v ruff &>/dev/null; then
92-
pip install --no-cache-dir ruff
92+
pip install --no-cache-dir ruff==0.16.4
9393
fi
9494
9595
- name: 'Run Ruff Lint Check'
@@ -101,7 +101,7 @@ jobs:
101101
- name: 'Install Mypy'
102102
run: |
103103
if ! command -v mypy &>/dev/null; then
104-
pip install --no-cache-dir mypy
104+
pip install --no-cache-dir mypy==2.4.1
105105
fi
106106
107107
- name: 'Run Mypy Static Type Check'
@@ -169,6 +169,12 @@ jobs:
169169
run: |
170170
python3 tools/lint/check_agent_doc_paths.py
171171
172+
- name: 'Check Documentation Links & OSS URLs'
173+
env:
174+
XMC_BYPASS_VENV: '1'
175+
run: |
176+
python3 tools/lint/check_doc_links.py
177+
172178
- name: 'Check UI & Workflow File Naming'
173179
env:
174180
XMC_BYPASS_VENV: '1'
@@ -354,7 +360,7 @@ jobs:
354360
run: |
355361
if ! command -v helm &>/dev/null; then
356362
mkdir -p "${RUNNER_TEMP}/helm"
357-
curl -sSfL "https://get.helm.sh/helm-v3.15.2-linux-amd64.tar.gz" \
363+
curl -sSfL "https://get.helm.sh/helm-v3.20.1-linux-amd64.tar.gz" \
358364
| tar -xz -C "${RUNNER_TEMP}/helm" --strip-components=1 linux-amd64/helm
359365
chmod +x "${RUNNER_TEMP}/helm/helm"
360366
echo "${RUNNER_TEMP}/helm" >> "${GITHUB_PATH}"
@@ -377,3 +383,17 @@ jobs:
377383
severity: 'HIGH,CRITICAL'
378384
scan-ref: 'deploy/'
379385
trivyignores: '.trivyignore'
386+
387+
# 8. PR Diff Size Telemetry
388+
- name: 'Check PR Diff Size'
389+
if: github.event_name == 'pull_request'
390+
env:
391+
XMC_BYPASS_VENV: '1'
392+
BASE_REF: ${{ github.base_ref }}
393+
run: |
394+
DIFF_JSON="$(python3 tools/analyze_diff_size.py --base-ref "origin/${BASE_REF}...")"
395+
echo "${DIFF_JSON}"
396+
if [ "$(echo "${DIFF_JSON}" | jq -r '.result.threshold_exceeded')" = 'true' ]; then
397+
REASONS="$(echo "${DIFF_JSON}" | jq -r '.result.exceeded_reasons | join("; ")')"
398+
echo "::warning::PR diff size exceeds review threshold: ${REASONS}"
399+
fi

‎tools/README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -84,8 +84,9 @@ Tooling and automation for release metadata extraction, SemVer calculations, and
8484
### 7. `tools/lint/` (Static Analyzers & Linters)
8585
Offline syntax walkers and static validation tooling:
8686
- [`check_go_determinism`](lint/check_go_determinism/): Ultra-fast (<50ms) Go AST analyzer checking Go test files for `os.Setenv` usage and arbitrary second-level `time.Sleep` calls.
87-
- [`check_legacy_imports.py`](lint/check_legacy_imports.py): Ratchet lint preventing regression of google3 migration leftovers (legacy Go import roots, unresolved `xmanager_cloud/...` module paths, `compat_go_*` targets, `gazelle:resolve` overrides, `third_party/` proto paths, google3 Helm flags, legacy UI/Python paths) against the zeroed [`legacy_imports_baseline.json`](lint/legacy_imports_baseline.json). Fails when any count grows above zero. Runs in pre-commit and the lint CI job. Tracking: [#1066](https://github.com/google/xmc/issues/1066).
87+
- [`check_legacy_imports.py`](lint/check_legacy_imports.py): Ratchet lint preventing regression of google3 migration leftovers (legacy Go import roots, unresolved `xmanager_cloud/...` module paths, `compat_go_*` targets, `gazelle:resolve` overrides, `third_party/` proto paths, google3 Helm flags, legacy UI/Python paths), OSS `TODO(b/...)` IDs under `api/`, `cmd/`, `pkg/`, and `sdks/`, and new generic helper/util/misc/common filenames against [`legacy_imports_baseline.json`](lint/legacy_imports_baseline.json). Fails when any count grows above baseline. Runs in pre-commit and the lint CI job. Tracking: [#1066](https://github.com/google/xmc/issues/1066).
8888
- [`check_agent_doc_paths.py`](lint/check_agent_doc_paths.py): Fails when agent docs (`.agents/**/*.md`, `AGENTS.md`) reference repo paths in code spans or fenced blocks that are not tracked files or directories. Placeholders (`<name>`, `{a,b}`, `$VAR`, `*`, `[`, `...`) only require their directory prefix to exist; runtime-created and illustrative paths live in its `ALLOWLIST` with reasons, and unreferenced entries fail. Runs in pre-commit (when agent docs are staged or files are deleted/renamed; always in `--check` mode) and the lint CI job.
89+
- [`check_doc_links.py`](lint/check_doc_links.py): Validates relative Markdown links (`[label](path)`) against tracked files and directories and rejects non-code `file://`, `corp.google.com`, `http(s)://go/...`, and `//go/...` URLs across all tracked Markdown docs outside `.agents/`. Runs in pre-commit (when non-`.agents/` Markdown docs or doc linters are staged, or files are deleted/renamed; always in `--check` mode) and the lint CI job.
8990
- [`check_file_naming.py`](lint/check_file_naming.py): Enforces UI and workflow file-naming conventions across tracked repository files (`*.ng.html` for HTML files under `cmd/ui/`—templates under `cmd/ui/app/` with `cmd/ui/index.html` allowlisted—`*_test.ts` for Karma unit tests under `cmd/ui/app/` while rejecting `*.spec.ts`/`*_spec.ts`/`*.test.ts` repo-wide outside `cmd/ui/screenshot_tests/`, `*.screenshot.spec.ts` for Playwright visual specs under `cmd/ui/screenshot_tests/`, and top-level `*.test.js` for Node workflow tests under `.github/scripts/`). Runs in pre-commit and the lint CI job.
9091
- [`check_ui_selector_contract.py`](lint/check_ui_selector_contract.py): Parses every tracked `*.py` under `tools/nightly_tests/` (fails on any syntax error; `EXCLUDED_SOURCES` opts files out) and lints each file containing selectors (`*_SEL`/`*_SEL_TMPL` constants, `.locator()`/`.wait_for_selector()` string arguments, or `has_text=` literals; reported as `sources`). Fails when selectors reference `[data-testid="..."]` values or custom tags that no `cmd/ui/app` template or component selector declares (Material `mat-`/`mdc-`/`cdk-` names exempt), when selectors use custom classes or non-standard attributes without `# ui-contract: <reason>`, when code references a `ui_css_selectors` constant that does not exist, or when selectors match user-facing copy via `:has-text(...)`/`has_text=` without a `# ui-contract: <reason>` comment on the same line or a standalone comment line above. Runs on every commit in pre-commit and in the lint CI job (~0.2s; no path gate, so the source list is not duplicated).
9192

‎tools/lint/BUILD.bazel‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,27 @@ py_test(
3737
],
3838
)
3939

40+
py_library(
41+
name = "check_doc_links_lib",
42+
srcs = ["check_doc_links.py"],
43+
imports = ["../.."],
44+
deps = [
45+
":check_agent_doc_paths_lib",
46+
"//tools/common:tool_utils",
47+
],
48+
)
49+
50+
py_test(
51+
name = "check_doc_links_test",
52+
srcs = ["check_doc_links_test.py"],
53+
imports = ["../.."],
54+
deps = [
55+
":check_agent_doc_paths_lib",
56+
":check_doc_links_lib",
57+
"//tools/common:tool_utils",
58+
],
59+
)
60+
4061
py_library(
4162
name = "check_file_naming_lib",
4263
srcs = ["check_file_naming.py"],

‎tools/lint/check_agent_doc_paths.py‎

Lines changed: 16 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -108,8 +108,8 @@ def build_path_token_re(roots: Iterable[str]) -> re.Pattern[str]:
108108

109109

110110
_PATH_TOKEN_RE = build_path_token_re(PATH_ROOTS)
111-
_INLINE_CODE_RE = re.compile(r"(`+)(.+?)\1")
112-
_FENCE_RE = re.compile(r"^\s*(`{3,}|~{3,})(.*)$")
111+
INLINE_CODE_RE = re.compile(r"(?<!`)(`+)(?!`)(.+?)(?<!`)\1(?!`)")
112+
_FENCE_RE = re.compile(r"^\s*(?:(`{3,})([^`]*)|(~{3,})(.*))$")
113113
_PLACEHOLDER_RE = re.compile(r"[<{$*\[]|\.\.\.")
114114
_TRAILING_PUNCTUATION = "."
115115

@@ -132,6 +132,14 @@ def is_agent_doc(path: str) -> bool:
132132
)
133133

134134

135+
def opens_fence(line: str) -> str:
136+
"""Returns the opening fence marker when a line opens a CommonMark fenced block."""
137+
fence = _FENCE_RE.match(line)
138+
if not fence:
139+
return ""
140+
return fence.group(1) or fence.group(3)
141+
142+
135143
def closes_fence(line: str, opening: str) -> bool:
136144
"""Returns whether a line closes a fenced block opened with `opening`.
137145
@@ -142,7 +150,8 @@ def closes_fence(line: str, opening: str) -> bool:
142150
fence = _FENCE_RE.match(line)
143151
if not fence:
144152
return False
145-
run, rest = fence.group(1), fence.group(2)
153+
run = fence.group(1) or fence.group(3)
154+
rest = fence.group(2) if fence.group(1) is not None else fence.group(4)
146155
return run[0] == opening[0] and len(run) >= len(opening) and not rest.strip()
147156

148157

@@ -156,11 +165,11 @@ def iter_code_segments(text: str) -> Iterator[tuple[int, str]]:
156165
else:
157166
yield lineno, line
158167
continue
159-
fence = _FENCE_RE.match(line)
160-
if fence:
161-
opening = fence.group(1)
168+
opened = opens_fence(line)
169+
if opened:
170+
opening = opened
162171
continue
163-
for match in _INLINE_CODE_RE.finditer(line):
172+
for match in INLINE_CODE_RE.finditer(line):
164173
yield lineno, match.group(2)
165174

166175

0 commit comments

Comments
 (0)