Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 24 additions & 4 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ jobs:
- name: 'Install Ruff'
run: |
if ! command -v ruff &>/dev/null; then
pip install --no-cache-dir ruff
pip install --no-cache-dir ruff==0.16.4
fi

- name: 'Run Ruff Lint Check'
Expand All @@ -101,7 +101,7 @@ jobs:
- name: 'Install Mypy'
run: |
if ! command -v mypy &>/dev/null; then
pip install --no-cache-dir mypy
pip install --no-cache-dir mypy==2.4.1
fi

- name: 'Run Mypy Static Type Check'
Expand Down Expand Up @@ -169,6 +169,12 @@ jobs:
run: |
python3 tools/lint/check_agent_doc_paths.py

- name: 'Check Documentation Links & OSS URLs'
env:
XMC_BYPASS_VENV: '1'
run: |
python3 tools/lint/check_doc_links.py

- name: 'Check UI & Workflow File Naming'
env:
XMC_BYPASS_VENV: '1'
Expand Down Expand Up @@ -355,9 +361,9 @@ jobs:
- name: 'Install Helm'
if: steps.changes.outputs.helm == 'true'
run: |
if ! command -v helm &>/dev/null; then
if ! helm version --short 2>/dev/null | grep -q '^v3\.'; then
mkdir -p "${RUNNER_TEMP}/helm"
curl -sSfL "https://get.helm.sh/helm-v3.15.2-linux-amd64.tar.gz" \
curl -sSfL "https://get.helm.sh/helm-v3.20.1-linux-amd64.tar.gz" \
| tar -xz -C "${RUNNER_TEMP}/helm" --strip-components=1 linux-amd64/helm
chmod +x "${RUNNER_TEMP}/helm/helm"
echo "${RUNNER_TEMP}/helm" >> "${GITHUB_PATH}"
Expand All @@ -380,3 +386,17 @@ jobs:
severity: 'HIGH,CRITICAL'
scan-ref: 'deploy/'
trivyignores: '.trivyignore'

# 8. PR Diff Size Telemetry
- name: 'Check PR Diff Size'
if: github.event_name == 'pull_request'
env:
XMC_BYPASS_VENV: '1'
BASE_REF: ${{ github.base_ref }}
run: |
DIFF_JSON="$(python3 tools/analyze_diff_size.py --base-ref "origin/${BASE_REF}...")"
echo "${DIFF_JSON}"
if [ "$(echo "${DIFF_JSON}" | jq -r '.result.threshold_exceeded')" = 'true' ]; then
REASONS="$(echo "${DIFF_JSON}" | jq -r '.result.exceeded_reasons | join("; ")')"
echo "::warning::PR diff size exceeds review threshold: ${REASONS}"
fi
3 changes: 2 additions & 1 deletion tools/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,8 +84,9 @@ Tooling and automation for release metadata extraction, SemVer calculations, and
### 7. `tools/lint/` (Static Analyzers & Linters)
Offline syntax walkers and static validation tooling:
- [`check_go_determinism`](lint/check_go_determinism/): Ultra-fast (<50ms) Go AST analyzer checking Go test files for `os.Setenv` usage and arbitrary second-level `time.Sleep` calls.
- [`check_legacy_imports.py`](lint/check_legacy_imports.py): Ratchet lint preventing regression of google3 migration leftovers (legacy Go import roots, unresolved `xmanager_cloud/...` module paths, `compat_go_*` targets, `gazelle:resolve` overrides, `third_party/` proto paths, google3 Helm flags, legacy UI/Python paths) against the zeroed [`legacy_imports_baseline.json`](lint/legacy_imports_baseline.json). Fails when any count grows above zero. Runs in pre-commit and the lint CI job. Tracking: [#1066](https://github.com/google/xmc/issues/1066).
- [`check_legacy_imports.py`](lint/check_legacy_imports.py): Ratchet lint preventing regression of google3 migration leftovers (legacy Go import roots, unresolved `xmanager_cloud/...` module paths, `compat_go_*` targets, `gazelle:resolve` overrides, `third_party/` proto paths, google3 Helm flags, legacy UI/Python paths), OSS `TODO(b/...)` IDs under `api/`, `cmd/`, `pkg/`, and `sdks/`, and new generic helper/util/misc/common filenames against [`legacy_imports_baseline.json`](lint/legacy_imports_baseline.json). Fails when any count grows above baseline. Runs in pre-commit and the lint CI job. Tracking: [#1066](https://github.com/google/xmc/issues/1066).
- [`check_agent_doc_paths.py`](lint/check_agent_doc_paths.py): Fails when agent docs (`.agents/**/*.md`, `AGENTS.md`) reference repo paths in code spans or fenced blocks that are not tracked files or directories. Placeholders (`<name>`, `{a,b}`, `$VAR`, `*`, `[`, `...`) only require their directory prefix to exist; runtime-created and illustrative paths live in its `ALLOWLIST` with reasons, and unreferenced entries fail. Runs in pre-commit (when agent docs are staged or files are deleted/renamed; always in `--check` mode) and the lint CI job.
- [`check_doc_links.py`](lint/check_doc_links.py): Validates relative Markdown links (`[label](path)`) against tracked files and directories and rejects non-code `file://`, `corp.google.com`, `http(s)://go/...`, and `//go/...` URLs across all tracked Markdown docs outside `.agents/`. Runs in pre-commit (when non-`.agents/` Markdown docs or doc linters are staged, or files are deleted/renamed; always in `--check` mode) and the lint CI job.
- [`check_file_naming.py`](lint/check_file_naming.py): Enforces UI and workflow file-naming conventions across tracked repository files (`*.ng.html` for HTML files under `cmd/ui/`—templates under `cmd/ui/app/` with `cmd/ui/index.html` allowlisted—`*_test.ts` for Karma unit tests under `cmd/ui/app/` while rejecting `*.spec.ts`/`*_spec.ts`/`*.test.ts` repo-wide outside `cmd/ui/screenshot_tests/`, `*.screenshot.spec.ts` for Playwright visual specs under `cmd/ui/screenshot_tests/`, and top-level `*.test.js` for Node workflow tests under `.github/scripts/`). Runs in pre-commit and the lint CI job.
- [`check_ui_selector_contract.py`](lint/check_ui_selector_contract.py): Parses every tracked `*.py` under `tools/nightly_tests/` (fails on any syntax error; `EXCLUDED_SOURCES` opts files out) and lints each file containing selectors (`*_SEL`/`*_SEL_TMPL` constants, `.locator()`/`.wait_for_selector()` string arguments, or `has_text=` literals; reported as `sources`). Fails when selectors reference `[data-testid="..."]` values or custom tags that no `cmd/ui/app` template or component selector declares (Material `mat-`/`mdc-`/`cdk-` names exempt), when selectors use custom classes or non-standard attributes without `# ui-contract: <reason>`, when code references a `ui_css_selectors` constant that does not exist, or when selectors match user-facing copy via `:has-text(...)`/`has_text=` without a `# ui-contract: <reason>` comment on the same line or a standalone comment line above. Runs on every commit in pre-commit and in the lint CI job (~0.2s; no path gate, so the source list is not duplicated).
- [`precommit_gates.py`](lint/precommit_gates.py): Declarative runner (`--mode fix|check`) and shared git/execution primitives for pre-commit Python lint gates (`check_legacy_imports`, `check_examples_coverage`, `check_matrix_contract`, `check_agent_doc_paths`, `check_file_naming`, `check_ui_selector_contract`).
Expand Down
21 changes: 21 additions & 0 deletions tools/lint/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,27 @@ py_test(
],
)

py_library(
name = "check_doc_links_lib",
srcs = ["check_doc_links.py"],
imports = ["../.."],
deps = [
":check_agent_doc_paths_lib",
"//tools/common:tool_utils",
],
)

py_test(
name = "check_doc_links_test",
srcs = ["check_doc_links_test.py"],
imports = ["../.."],
deps = [
":check_agent_doc_paths_lib",
":check_doc_links_lib",
"//tools/common:tool_utils",
],
)

py_library(
name = "check_file_naming_lib",
srcs = ["check_file_naming.py"],
Expand Down
23 changes: 16 additions & 7 deletions tools/lint/check_agent_doc_paths.py
Original file line number Diff line number Diff line change
Expand Up @@ -108,8 +108,8 @@ def build_path_token_re(roots: Iterable[str]) -> re.Pattern[str]:


_PATH_TOKEN_RE = build_path_token_re(PATH_ROOTS)
_INLINE_CODE_RE = re.compile(r"(`+)(.+?)\1")
_FENCE_RE = re.compile(r"^\s*(`{3,}|~{3,})(.*)$")
INLINE_CODE_RE = re.compile(r"(?<!`)(`+)(?!`)(.+?)(?<!`)\1(?!`)")
_FENCE_RE = re.compile(r"^\s*(?:(`{3,})([^`]*)|(~{3,})(.*))$")
_PLACEHOLDER_RE = re.compile(r"[<{$*\[]|\.\.\.")
_TRAILING_PUNCTUATION = "."

Expand All @@ -132,6 +132,14 @@ def is_agent_doc(path: str) -> bool:
)


def opens_fence(line: str) -> str:
"""Returns the opening fence marker when a line opens a CommonMark fenced block."""
fence = _FENCE_RE.match(line)
if not fence:
return ""
return fence.group(1) or fence.group(3)


def closes_fence(line: str, opening: str) -> bool:
"""Returns whether a line closes a fenced block opened with `opening`.

Expand All @@ -142,7 +150,8 @@ def closes_fence(line: str, opening: str) -> bool:
fence = _FENCE_RE.match(line)
if not fence:
return False
run, rest = fence.group(1), fence.group(2)
run = fence.group(1) or fence.group(3)
rest = fence.group(2) if fence.group(1) is not None else fence.group(4)
return run[0] == opening[0] and len(run) >= len(opening) and not rest.strip()


Expand All @@ -156,11 +165,11 @@ def iter_code_segments(text: str) -> Iterator[tuple[int, str]]:
else:
yield lineno, line
continue
fence = _FENCE_RE.match(line)
if fence:
opening = fence.group(1)
opened = opens_fence(line)
if opened:
opening = opened
continue
for match in _INLINE_CODE_RE.finditer(line):
for match in INLINE_CODE_RE.finditer(line):
yield lineno, match.group(2)


Expand Down
216 changes: 216 additions & 0 deletions tools/lint/check_doc_links.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,216 @@
#!/usr/bin/env python3
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""Lint verifying relative links and OSS URL hygiene in repository Markdown docs.

Scans tracked `*.md` files outside `.agents/` (which `check_agent_doc_paths.py`
checks separately) and fails when:
- A relative Markdown link target does not exist in the git index or escapes the repo.
- A non-code Markdown line contains a `file://` URI, `corp.google.com` host, or
`http(s)://go/...` / `//go/...` corporate shortlink.
"""

from __future__ import annotations

import argparse
import dataclasses
import posixpath
import re
import sys
from collections.abc import Iterator
from pathlib import Path
from typing import Any
Comment thread
VLukiantsev marked this conversation as resolved.
Resolved
from urllib.parse import unquote

_REPO_ROOT = Path(__file__).resolve().parents[2]
if str(_REPO_ROOT) not in sys.path:
sys.path.insert(0, str(_REPO_ROOT))

from tools.common.tool_utils import HandledError, run_tool # noqa: E402
from tools.lint.check_agent_doc_paths import ( # noqa: E402
INLINE_CODE_RE,
closes_fence,
list_tracked_files,
opens_fence,
tracked_paths,
)

_LINK_TAIL_RE = re.compile(r'^\s*(?:(?:"[^"]*"|\'[^\']*\')\s*)?\)')
_REF_DEF_RE = re.compile(r"^\s{0,3}\[[^\]]+\]:\s*(<[^>]+>|[^\s]+)")
_SCHEME_RE = re.compile(r"^[a-z][a-z0-9+.-]+:", re.IGNORECASE)
_FORBIDDEN_URL_RE = re.compile(
r"file://[^\s)>'\"]+|[A-Za-z0-9.-]*corp\.google\.com\b|(?:https?://|(?<![A-Za-z0-9:/])//)go/[A-Za-z0-9_./-]+",
re.IGNORECASE,
)
Comment thread
VLukiantsev marked this conversation as resolved.
Resolved


@dataclasses.dataclass(frozen=True)
class DocViolation:
"""A broken relative link or forbidden internal URL in a Markdown document."""

doc: str
line: int
kind: str
target: str


def is_checked_doc(path: str) -> bool:
"""Return whether a tracked path is a Markdown doc covered by this linter."""
return path.endswith(".md") and not path.startswith(".agents/")


def iter_prose_lines(text: str) -> Iterator[tuple[int, str]]:
"""Yield (1-based line number, line without inline code) outside fenced blocks."""
opening = ""
for lineno, line in enumerate(text.splitlines(), start=1):
if opening:
if closes_fence(line, opening):
opening = ""
continue
opened = opens_fence(line)
if opened:
opening = opened
continue
yield lineno, INLINE_CODE_RE.sub("", line)


def _parse_link_destination(prose: str, start: int) -> str | None:
"""Parse an inline Markdown link destination immediately following `](`."""
i = start
while i < len(prose) and prose[i] in " \t":
i += 1
if i < len(prose) and prose[i] == "<":
end = prose.find(">", i + 1)
if end == -1 or not _LINK_TAIL_RE.match(prose[end + 1 :]):
return None
return prose[i + 1 : end]
depth = 0
j = i
while j < len(prose):
ch = prose[j]
if ch in " \t":
break
if ch == "(":
depth += 1
elif ch == ")":
if depth == 0:
return prose[i:j] if j > i else None
depth -= 1
j += 1
if j > i and depth == 0 and _LINK_TAIL_RE.match(prose[j:]):
return prose[i:j]
return None


def extract_link_targets(prose: str) -> list[str]:
"""Extract inline link, linked-image, and reference-definition targets from a line."""
targets: list[str] = []
pos = 0
while (idx := prose.find("](", pos)) != -1:
dest = _parse_link_destination(prose, idx + 2)
if dest is not None:
targets.append(dest)
pos = idx + 2
ref_match = _REF_DEF_RE.match(prose)
if ref_match:
targets.append(ref_match.group(1))
return targets


def resolve_repo_target(doc: str, raw_target: str) -> tuple[str, str | None]:
"""Resolve a Markdown link target relative to doc.

Returns:
Tuple of (resolved_repo_path, error_kind) where error_kind is None if
the link is external, anchor-only, or resolves cleanly within the repo.
"""
stripped = raw_target.strip("<>")
if not stripped or stripped.startswith("#"):
return "", None
if stripped.lower().startswith("file://"):
return stripped.rstrip(".,;:"), "forbidden_url"
if stripped.startswith("//") or _SCHEME_RE.match(stripped):
return "", None
path_part = unquote(stripped.split("#", 1)[0].split("?", 1)[0])
if not path_part:
return "", None
if path_part.startswith("/"):
joined = posixpath.normpath(path_part.lstrip("/"))
else:
doc_dir = posixpath.dirname(doc)
joined = posixpath.normpath(posixpath.join(doc_dir, path_part) if doc_dir else path_part)
if joined == ".." or joined.startswith("../"):
return joined, "escapes_repo"
return ("" if joined == "." else joined), None
Comment thread
VLukiantsev marked this conversation as resolved.
Resolved


def find_line_violations(
doc: str, lineno: int, prose: str, existing: frozenset[str]
) -> list[DocViolation]:
"""Collect broken-link and forbidden-URL violations from a single prose line."""
violations: list[DocViolation] = []
for raw_target in extract_link_targets(prose):
resolved, err_kind = resolve_repo_target(doc, raw_target)
if err_kind is not None:
violations.append(DocViolation(doc=doc, line=lineno, kind=err_kind, target=resolved))
elif resolved and resolved not in existing:
violations.append(
DocViolation(doc=doc, line=lineno, kind="missing_target", target=resolved)
)
for url_match in _FORBIDDEN_URL_RE.finditer(prose):
url = url_match.group(0).rstrip(".,;:")
if not any(v.target == url and v.line == lineno for v in violations):
violations.append(DocViolation(doc=doc, line=lineno, kind="forbidden_url", target=url))
return violations


def scan_docs(repo_root: Path, docs: list[str], existing: frozenset[str]) -> list[DocViolation]:
"""Scan all tracked Markdown docs outside `.agents/` for link and URL issues."""
violations: list[DocViolation] = []
for doc in docs:
doc_path = repo_root / doc
if not doc_path.is_file():
continue
text = doc_path.read_text(encoding="utf-8", errors="replace")
for lineno, prose in iter_prose_lines(text):
violations.extend(find_line_violations(doc, lineno, prose, existing))
return violations


def check_doc_links(args: argparse.Namespace) -> dict[str, Any]:
"""Validate relative links and public URL hygiene in repository Markdown docs."""
repo_root = Path(args.repo_root).resolve()
files = list_tracked_files(repo_root)
docs = sorted(f for f in files if is_checked_doc(f))
violations = scan_docs(repo_root, docs, tracked_paths(files))
if violations:
raise HandledError(
f"Found {len(violations)} broken Markdown link(s) or internal URL(s).",
"Fix broken relative Markdown links against tracked files/directories and remove "
"local file:// or corporate URLs.",
details={"violations": [dataclasses.asdict(v) for v in violations]},
)
return {"docs_scanned": len(docs), "violations": 0}


def main() -> None:
"""CLI entrypoint."""
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument("--repo-root", default=str(_REPO_ROOT), help="Repository root.")
run_tool(parser, check_doc_links)


if __name__ == "__main__":
main()
Loading
Loading