Skip to content

chore(lint): add doc link checker, OSS hygiene ratchets, and CI tool pins - #1674

Open
VLukiantsev wants to merge 1 commit into
chore/enable-buf-lint-and-fix-bzl-docsfrom
chore/add-doc-links-and-hygiene-lint-gates
Open

VLukiantsev wants to merge 1 commit into
chore/enable-buf-lint-and-fix-bzl-docsfrom
chore/add-doc-links-and-hygiene-lint-gates

Conversation

@VLukiantsev

@VLukiantsev VLukiantsev commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Add tools/lint/check_doc_links.py (with unit tests and check_doc_links_lib/check_doc_links_test Bazel targets) wired into tools/lint/precommit_gates.py, .github/workflows/lint.yml, and tools/README.md to validate relative Markdown links (including balanced parentheses, <...> angle destinations, linked images, root-relative paths, percent-encoded segments, and reference definitions) and reject non-code file://, corporate hostnames, and shortlink URLs outside .agents/. Share CommonMark backtick-fence info-string opens_fence / _FENCE_RE and exact-backtick-run INLINE_CODE_RE helpers in tools/lint/check_agent_doc_paths.py, extend tools/lint/check_legacy_imports.py and tools/lint/legacy_imports_baseline.json with oss_b_todo_references (api/, cmd/, pkg/, sdks/ at baseline 0) and generic_helper_filenames (baseline 27, exempting deploy/**/templates/*.tpl), pin ruff==0.16.4, mypy==2.4.1, and helm (v3.20.1), and add non-blocking PR diff-size ::warning:: telemetry in .github/workflows/lint.yml.

Motivation & Context

Automated link checking, generic-filename and internal issue-ID TODO ratchets, internal-URL blocks, pinned CI tool versions, and PR size warnings prevent documentation rot and OSS hygiene regressions across concurrent contributions.

Key Changes

Modified/added 11 file(s):

  • tools/lint/precommit_gates.py & tools/lint/precommit_gates_test.py: Register check_doc_links in PYTHON_GATES (gated in fix mode on staged AM *.md, check_doc_links.py/check_agent_doc_paths.py, or D paths; unconditional in --check mode), expand check_agent_doc_paths trigger for check_agent_doc_paths.py, and add unit tests asserting both triggers in test_python_gates_include_doc_link_and_agent_doc_triggers.
  • .github/workflows/lint.yml: Pin ruff==0.16.4, mypy==2.4.1, and Helm v3.20.1 (replacing Helm v3.15.2 and checking helm version --short | grep -q '^v3\.' for ubuntu-latest which pre-installs Helm v4.3.1), add Check Documentation Links & OSS URLs, and add PR-only Check PR Diff Size (tools/analyze_diff_size.py) emitting ::warning:: when thresholds are exceeded.
  • tools/README.md: Document oss_b_todo_references and generic filename ratchets under check_legacy_imports.py and add the check_doc_links.py entry.
  • tools/lint/BUILD.bazel: Add py_library check_doc_links_lib and py_test check_doc_links_test.
  • tools/lint/check_agent_doc_paths.py: Extract shared opens_fence (_FENCE_RE enforcing CommonMark backtick fence info-string rules without extra backticks) and hardened INLINE_CODE_RE enforcing exact inline backtick-run delimiter matching across both doc linters.
  • tools/lint/check_doc_links.py & tools/lint/check_doc_links_test.py: New linter scanning tracked *.md files outside .agents/ and unit tests covering percent-decoding, root-relative links, nested/tilde/unclosed fences, empty/malformed/multibyte inputs, linked images, parenthesized/angle/reference links, forbidden internal URLs, working-tree deletions, and CLI JSON output.
  • tools/lint/check_legacy_imports.py, check_legacy_imports_test.py, & legacy_imports_baseline.json: Add oss_b_todo_references (^(api|cmd|pkg|sdks)/, baseline 0) and generic_helper_filenames (27, exempting deploy/**/templates/**/*.tpl), broaden HandledError guidance, and test boundary cases.

Assumptions

  • generic_helper_filenames is ratcheted at the existing 27-file repository count while exempting deploy/**/templates/**/*.tpl files — ratcheting at 27 avoids mass cross-repo renames inside this linter PR while automatically ratcheting down via pre-commit as files are refactored — swapping one generic filename for another keeps net count 27, non-code generic names outside deploy/**/templates/**/*.tpl fail pre-commit/CI until renamed, while generic .tpl partials under deploy/**/templates/, generic directory names, and tools/shared filenames remain unchecked by this metric.
  • oss_b_todo_references scans only ^(api|cmd|pkg|sdks)/ for parenthesized internal issue-ID TODO references at baseline 0 — those four roots contain public OSS APIs, binaries, packages, and SDKs per the Open-Source TODO standard — internal issue IDs under internal/, deploy/, examples/, test/, or tools/ (or unparenthesized TODO: forms) are not flagged by this metric.
  • check_doc_links.py scans tracked *.md files outside .agents/ (including AGENTS.md and */AGENTS.md), treats /path links as repo-root-relative, skips non-file:// URI schemes and non-corporate protocol-relative //... external links in resolve_repo_target, and triggers in pre-commit fix mode when AM .md, check_doc_links.py, check_agent_doc_paths.py, or D paths are staged — .agents/ owns internal skill/rule conventions checked by check_agent_doc_paths.py — internal URLs or broken links inside .agents/, 4-space indented code blocks, inline code spans, or trailing lines after unclosed fences are not flagged by check_doc_links.py.
  • .github/workflows/lint.yml pins ruff==0.16.4, mypy==2.4.1, and Helm v3.20.1 inside availability/version guards (if ! command -v <tool> for Ruff/Mypy; if ! helm version --short | grep -q '^v3\.' for Helm) — ubuntu-latest pre-installs Helm v4 while pre-installed tools save setup latency when matching — if ubuntu-latest pre-installs a different Ruff/Mypy version on PATH the guard skips the pin, or if a pinned version is unavailable or stricter than local tools, the CI Code Quality & Linting job fails until aligned.
  • Check PR Diff Size runs only when github.event_name == 'pull_request', calls tools/analyze_diff_size.py --base-ref "origin/${BASE_REF}...", and emits a non-blocking ::warning:: annotation — diff size serves as an advisory split prompt rather than a hard merge blocker — threshold-exceeding PRs still pass CI unless gated in review.

Verification & Testing

  • Commands Executed:
    • ./.githooks/pre-commit --check (Buildifier -lint=warn, gofmt, golangci-lint, check_go_determinism, go mod tidy, check_legacy_imports, check_examples_coverage, check_matrix_contract, check_agent_doc_paths, check_doc_links, check_file_naming, check_ui_selector_contract, Prettier, ESLint, tsc, addlicense, Ruff, Mypy, terraform fmt, TFLint, helm lint, test/helm 88 tests, Trivy, actionlint, shellcheck, Non-UI JS ESLint, and Bazel tests)
    • bazel test --test_output=errors //tools/lint:all //cmd/ui:ui_unit_tests
    • node --test .github/scripts/*.test.js
  • Coverage & Test Status:
    • ./.githooks/pre-commit --check passed (320/320 backend Bazel tests pass, Ruff 355 files, Mypy 345 source files, check_go_determinism 504 files, check_doc_links 78 docs / 0 violations, 88/88 test/helm tests), //tools/lint:all (9/9) and //cmd/ui:ui_unit_tests passed, and node --test .github/scripts/*.test.js passed (229/229). CI-only ruff==0.16.4, mypy==2.4.1, helm v3.20.1, and Check PR Diff Size steps run on GitHub pull_request events after push.

Related Issues / Bugs

N/A (extends #1066 legacy-hygiene ratchets; does not close #1066).

TAG=agy
CONV=ba8411ef-c093-4f45-950a-b3bc1899268c

@VLukiantsev
VLukiantsev added this pull request to stack #1675 November 12, 2026 19:46

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new Markdown link linter (check_doc_links.py) to validate relative links and restrict internal Google URLs, integrated via pre-commit hooks and Bazel. It also updates check_legacy_imports.py to flag generic helper filenames and OSS TODO(b/...) references. The review feedback identifies critical issues in resolve_repo_target concerning case-sensitive external scheme checks and incorrect resolution of root-relative links, and suggests adding corresponding unit tests to cover these scenarios.

Comment thread tools/lint/check_doc_links.py Resolved
Comment thread tools/lint/check_doc_links_test.py Resolved
Comment thread tools/lint/check_doc_links_test.py Resolved
@VLukiantsev VLukiantsev changed the title chore: add doc link checker, OSS hygiene ratchets, and CI tool pins chore(lint): add doc link checker, OSS hygiene ratchets, and CI tool pins Nov 13, 2026
@VLukiantsev
VLukiantsev force-pushed the chore/add-doc-links-and-hygiene-lint-gates branch from 95c3e06 to 007bea5 Compare November 13, 2026 01:08
@VLukiantsev

Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new Markdown link linter (check_doc_links.py) with associated tests and Bazel/pre-commit integrations to validate relative links and block forbidden internal URLs. It also updates check_legacy_imports.py to track generic helper filenames and internal TODO(b/...) references, and refines the fence-parsing logic in check_agent_doc_paths.py. The review feedback recommends enhancing the new link linter by decoding percent-encoded characters (e.g., %20) using urllib.parse.unquote to prevent false positives, adding tests for this behavior, and extending the forbidden URL pattern to catch protocol-relative corporate shortlinks.

Comment thread tools/lint/check_doc_links.py Resolved
Comment thread tools/lint/check_doc_links.py Resolved
Comment thread tools/lint/check_doc_links_test.py Resolved
Comment thread tools/lint/check_doc_links.py Resolved
@VLukiantsev
VLukiantsev force-pushed the chore/add-doc-links-and-hygiene-lint-gates branch from 007bea5 to 35c2b17 Compare November 13, 2026 01:24
@VLukiantsev

Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new documentation link and OSS URL hygiene linter (check_doc_links.py) along with its corresponding unit tests, Bazel build targets, and integration into the pre-commit githook. Additionally, it updates the check_legacy_imports.py linter to track and prevent the introduction of new generic helper filenames (e.g., containing 'util', 'helper', 'misc', or 'common') and internal Google bug tracker references (TODO(b/...)) in public directories, updating the baseline configuration accordingly. Refactorings were also made to the Markdown parsing logic in check_agent_doc_paths.py to share inline code and fence block detection. There are no review comments, and I have no feedback to provide.

@VLukiantsev
VLukiantsev force-pushed the chore/add-doc-links-and-hygiene-lint-gates branch 2 times, most recently from 07a15d7 to e4500eb Compare November 13, 2026 09:19
@VLukiantsev
VLukiantsev force-pushed the chore/add-doc-links-and-hygiene-lint-gates branch from e4500eb to 2138a44 Compare November 13, 2026 10:14
@VLukiantsev
VLukiantsev force-pushed the chore/add-doc-links-and-hygiene-lint-gates branch from 2138a44 to 5429e88 Compare November 13, 2026 10:56
@VLukiantsev
VLukiantsev force-pushed the chore/add-doc-links-and-hygiene-lint-gates branch from 5429e88 to 0279c5f Compare November 13, 2026 11:58
@VLukiantsev
VLukiantsev force-pushed the chore/add-doc-links-and-hygiene-lint-gates branch 2 times, most recently from 377bdbf to e825c5c Compare November 13, 2026 13:16
@VLukiantsev
VLukiantsev force-pushed the chore/add-doc-links-and-hygiene-lint-gates branch from e825c5c to 3083e0c Compare November 13, 2026 14:12
@VLukiantsev
VLukiantsev force-pushed the chore/add-doc-links-and-hygiene-lint-gates branch from 3083e0c to 41029bf Compare November 13, 2026 14:58
@VLukiantsev
VLukiantsev force-pushed the chore/add-doc-links-and-hygiene-lint-gates branch 2 times, most recently from b77012e to 1e01195 Compare November 16, 2026 09:17
@VLukiantsev
VLukiantsev force-pushed the chore/add-doc-links-and-hygiene-lint-gates branch from 1e01195 to 6054b14 Compare November 16, 2026 11:14
@VLukiantsev
VLukiantsev removed this pull request from stack #1675 November 16, 2026 11:16
@VLukiantsev
VLukiantsev added this pull request to stack #1702 November 16, 2026 11:16
@VLukiantsev
VLukiantsev force-pushed the chore/add-doc-links-and-hygiene-lint-gates branch from 6054b14 to abff99c Compare November 16, 2026 11:44
@VLukiantsev
VLukiantsev force-pushed the chore/add-doc-links-and-hygiene-lint-gates branch from abff99c to 08cff84 Compare November 16, 2026 12:36
@VLukiantsev
VLukiantsev marked this pull request as ready for review November 16, 2026 19:26
@github-actions
github-actions Bot requested a review from AntonTyb November 16, 2026 19:26
@VLukiantsev
VLukiantsev force-pushed the chore/add-doc-links-and-hygiene-lint-gates branch from 08cff84 to 1a45f99 Compare November 16, 2026 19:57
@VLukiantsev

Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new linter, check_doc_links.py, along with its corresponding unit tests, Bazel build targets, and pre-commit gate integration. This linter validates relative Markdown links and blocks forbidden internal URLs (such as corp.google.com or go/ shortlinks) in non-agent Markdown documentation. Additionally, the existing check_legacy_imports.py linter is updated to track and restrict generic helper filenames and internal TODO(b/...) references, with corresponding updates to its test suite and baseline metrics. The check_agent_doc_paths.py utility was also refactored to expose code and fence parsing logic for reuse. There are no review comments to address, and we have no additional feedback to provide.

@VLukiantsev
VLukiantsev force-pushed the chore/add-doc-links-and-hygiene-lint-gates branch 2 times, most recently from 4ba475e to 8b78ea4 Compare November 16, 2026 20:13
…pins

- Add `tools/lint/check_doc_links.py`, `check_doc_links_test.py`, and Bazel targets wired into `tools/lint/precommit_gates.py`, `.github/workflows/lint.yml`, and `tools/README.md`, with shared CommonMark `opens_fence`/`INLINE_CODE_RE` in `check_agent_doc_paths.py`, balanced/angle/reference-link parsing, case-insensitive external/file scheme checks, protocol-relative handling, and `forbidden_url` deduplication.
- Add `oss_b_todo_references` and `generic_helper_filenames` ratchets to `check_legacy_imports.py` and `legacy_imports_baseline.json` (plus broadened `HandledError` guidance and unit tests).
- Pin `ruff==0.16.4`, `mypy==2.4.1`, `helm` `v3.20.1` (`helm version --short` check), and `Check PR Diff Size` warning telemetry in `.github/workflows/lint.yml`.

TAG=agy
CONV=ba8411ef-c093-4f45-950a-b3bc1899268c
@VLukiantsev
VLukiantsev force-pushed the chore/add-doc-links-and-hygiene-lint-gates branch from 8b78ea4 to e367865 Compare November 17, 2026 11:37

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

chore: eliminate google3 migration compatibility layer

1 participant