Skip to content

validation-map.md

Latest commit

 

History

History
504 lines (441 loc) · 30 KB

validation-map.md

File metadata and controls

504 lines (441 loc) · 30 KB
Raw

Validation Map

This map records the lowest-cost validation commands agents should reach for before broad CI-sized runs.

Discovery

pnpm nx show projects --withTarget test
pnpm nx show projects --withTarget lint
pnpm nx show projects --withTarget e2e

Manual CI Runs

When a PR event does not start checks for the current head, dispatch CI and E2E with gh workflow run ci.yml --ref <branch> and gh workflow run e2e-tests.yaml --ref <branch>. CodeQL also supports gh workflow run codeql-analysis.yml --ref <branch>; this analyzes the selected branch commit instead of the PR merge commit. Verify each run's head SHA before using its result as evidence. Docker validation can use gh workflow run docker.yml --ref <branch> -f push=false.

Unit And Type Checks

Area Command
Angular renderer entry points pnpm run typecheck:web
Electron main process entry points pnpm run typecheck:backend
Spec programs (Jest, perf harness) pnpm run typecheck:spec
Full unit suite (all projects) pnpm run test:unit:ci
EPG data access pnpm nx test epg-data-access
Workspace shell utilities pnpm nx test workspace-shell-util
Shared SQLite schema/connection pnpm nx test database
TV runtime, providers, local storage and LG/Tizen package inputs corepack pnpm nx test tv, corepack pnpm nx run tv:verify-build and corepack pnpm nx e2e tv-e2e; Samsung hardware/signing: Tizen acceptance
Packaging metadata pnpm nx test packaging

typecheck:spec (tools/typecheck/spec-typecheck.mjs) runs tsc --noEmit over every tsconfig.spec.json under apps/, libs/ and tools/, a few programs at a time (--concurrency=N or SPEC_TYPECHECK_CONCURRENCY; a positional argument filters by path), and fails on any diagnostic. ts-jest transpiles with isolatedModules, and tsx and Playwright strip types without checking them, so this is the only check that catches a spec or harness file whose types drifted from the code it exercises. CI runs it in the unit-and-typecheck job after typecheck:ci. Conventions the gate relies on:

  • Spec tsconfigs use module: preserve with moduleResolution: bundler, the same as the library's own tsconfig.json; ts-jest forces CommonJS emit outside ESM mode, so the setting only affects type-checking, and node10 resolution cannot see Angular's exports-only secondary entry points.
  • Each spec program keeps global.d.ts in files so window.electron and the other ambient declarations resolve. A config that sets files lists ../../global.d.ts again; one that does not inherits it from tsconfig.base.json.
  • Libraries tested through tools/testing/run-web-esm-lib-tests.mjs are type-checked by apps/web/tsconfig.spec.json, the config jest.web-esm.workspace.ts hands to ts-jest; add a new ESM-tested library's spec globs there. apps/web/src/jest-esm.d.ts types jest.unstable_mockModule.
  • apps/electron-backend-e2e/tsconfig.spec.json type-checks the performance harness, which test-performance-harness (tsx --test) and the journeys' Playwright runs execute without type checking. Besides module and moduleResolution it sets target: es2022 and lib: ["es2022", "dom"], because the project's commonjs module and the base es2015 target reject the harness's import.meta, BigInt literals and ES2022 library names. It also sets types: ["node"]: the specs import describe and it from node:test, so a Jest or Mocha global fails the check. Its include covers the whole project (src/**/*.ts and the root Playwright configs): the journey harness, the benchmark harness and its specs, and the Electron E2E specs, with no baseline or ignore list.
  • Type test doubles instead of casting to any: jest.Mocked<T>, InstanceType<typeof SomeStore> for signal stores, and Object.defineProperty or a writable mapped type for read-only capability flags.

Lint

pnpm run lint                 # nx run-many --target=lint --all
pnpm nx lint <project>        # single project

The CI workflow (.github/workflows/ci.yml) lints affected projects on PRs (nx affected) and every project on master pushes. This enforces @nx/enforce-module-boundaries (scope/domain/type tag constraints), the legacy bare-alias ban, and the max-lines file-size rule (hard maximum 400 lines for production TypeScript, 1200 for tests; blank lines and comments are not counted). The limits live in tools/eslint/max-lines-config.mjs, which both eslint.config.mjs and the generator import. Files that predate the max-lines rule are baselined in tools/eslint/max-lines-baseline.mjs; after splitting a baselined file below the limit, regenerate the list with node tools/eslint/generate-max-lines-baseline.mjs. Never add new files to the baseline.

Coverage Tiers

Use tools/coverage/coverage-policy.json as the source of truth for coverage ownership. Every project with a test target must be classified in a tier; pnpm run coverage:policy:check (part of coverage:ci) fails CI when a new project is missing from the policy, a listed project no longer exists, or a Tier A entry has no test target. CI runs Tier A with coverage (uploaded to Codecov) and each Tier B/C project's validationCommand (falling back to nx test) without coverage; projects with an e2e target are skipped there because the E2E workflow already runs them on every PR that touches app code. Coverage commands resolve the Nx project graph once per invocation so inferred Jest targets remain classified even when project.json has no explicit test. Project-local Jest configs and the existing workspace-root Angular ESM config keep separate source-glob roots, with coverage reports written to the same workspace output paths and JUnit classnames relative to the repository. Migrated Jest commands keep the workspace working directory for subprocess and file-based test contracts, independently of the Jest config's root directory. Docs-only changes (Markdown, docs/, .plans/, .codex/, .claude/) and apps/website/** changes skip the E2E workflow via paths-ignore — for those PRs no E2E validation runs in CI, which is intentional: they cannot affect app behavior.

Tier A coverage is fail-closed. coverage:unit:ci relays Jest output but exits nonzero on a Failed to collect coverage marker, a missing or invalid project report, or a runtime-owning production TypeScript file absent from that report. It runs projects a few at a time, largest first, with a bounded Jest worker count per project (defaults: min(3, cores - 1) in flight and ceil(cores / concurrency) workers each; override with --concurrency=N, --max-workers=N or TIER_A_CONCURRENCY / TIER_A_MAX_WORKERS). Two cores run one project at a time with two workers. Two in flight would leave each project one worker, which runs Jest in-band, and a big project's single heap can run out on a small machine; for the same reason avoid --max-workers=1. Each project's output is printed as one block when it finishes, and the run ends with the wall-clock total and the longest projects. Spec tsconfigs set isolatedModules: true, so ts-jest transpiles files one at a time instead of type-checking each through a language service (the web configs already ran with diagnostics: false); isolatedModules-incompatible syntax such as a type re-export without export type still fails at load time, and spec type errors are caught by typecheck:spec (see Unit And Type Checks).

The activity SQLite and real Chromium contracts return their actual runtime counters through @iptvnator/shared/testing/runtime-coverage. Esbuild instruments selected original TypeScript files before bundling; paths and locations remain source-based. SQLite writes counters on subprocess exit. Browser tests capture all pages before reload/close and merge their counters before writing the report. The parent merges those reports into Jest without replacing objects cached by existing instrumentation. Missing reports fail the test; assertion failures still propagate. The helper never creates synthetic hits or changes coverage thresholds.

Some electron-backend and database specs run SQLite code in the Electron binary under ELECTRON_RUN_AS_NODE, resolving it with require('electron'). pnpm does not run Electron's postinstall (electron is deliberately absent from onlyBuiltDependencies, see workspace shell), so the first require downloads and extracts the binary. With projects and Jest workers in parallel, one process can exec it while another is still extracting it (spawnSync … ETXTBSY on Linux). When a selected project has a spec that calls createRequire(...)('electron'), coverage:unit:ci first runs tools/testing/electron/ensure-electron-binary.mjs, which downloads the binary once and fails unless it runs and reports the pinned version; CI also runs it as its own step when the Tier A suite is in scope. In a fresh worktree, run it before starting several of these specs at once by other means. The database:test, electron-backend:test and packaging:test targets share the non-cached electron-test-tools:ensure-electron-binary dependency. Nx runs it once before these tests in a task graph, including in the full workspace test run. For a filtered selection known to run under Node alone, use pnpm nx test database --excludeTaskDependencies --testPathPatterns=path-utils. Leave the dependency enabled for the full suites and for specs that spawn Electron.

In CI, a pull request skips the Tier A suite (and the merged-coverage upload) when every changed file is outside Tier A test inputs: git diff compares the checked-out PR merge commit with its first parent, including both paths of any rename. A newer PR push cannot change that diff. tools/coverage/unit-coverage-scope.mjs holds the allowlist (Markdown, docs/, notes and plans, agent guidance, workflows other than ci.yml and build-and-make.yaml, the website, E2E and mock-server apps, release/packaging/skills/performance tooling, and a package.json edit confined to non-coverage:* scripts). Anything else, including files no Nx project owns such as jest.preset.js or tsconfig.base.json, and every {workspaceRoot} input a Tier A test target declares, runs the full suite; master pushes always run it. nx affected is deliberately not used for this decision because a change to an unowned file affects no project. A node test parses every Tier A source for string literals that point at repository files outside the owning project (tier-a-external-references.mjs) and fails if the allowlist would skip any of them, so a new cross-project read cannot be silently exempted. Jest's transform cache is kept in JEST_CACHE_DIRECTORY and persisted with actions/cache: pull requests restore it, while only master pushes (starting from an empty cache, so it holds exactly the current tree) and maintainer dispatches save it. A shared Nx task cache is not used: Nx indexes its local cache in a machine-specific database, so a restored cache folder is never hit, and sharing it safely needs Nx Cloud or another supported remote cache. coverage:merge requires every configured Tier A report before replacing the merged output. Strict health validation also requires the merged Istanbul map itself to contain usable instrumentation for every runtime-owning Tier A file, recomputes its summary, and then applies aggregate and selected critical-file ratchets.

Runtime-owning files are discovered from the TypeScript AST. Specs, declarations, test setup and stubs, generated and environment files, index.ts, type-only files, and pure re-export shims are excluded. Ratchets live under reporting.coverageRatchet in tools/coverage/coverage-policy.json; update them only from a fresh full coverage:ci report when every value stays level or rises, and never lower one to accept a regression. The only exception is an explicitly reviewed production source shrink: minimumCovered may follow a lower total statement count when the PR documents the removed executable statements and fresh coverage proves that the corresponding minimumPercent, every aggregate ratchet, and the remaining behavioral coverage do not decrease.

Tier Rule Validation
A Product/runtime Angular, Electron, backend, data-access, portal, playlist, workspace, playback, EPG, and shared UI code collects source coverage. pnpm run coverage:ci
B Validate behavior without percentage coverage, such as website, packaging, and Playwright E2E projects. pnpm nx test website, pnpm nx test packaging, or the closest E2E target
C Excluded from the source coverage baseline, such as mock servers, test helper libraries, and untested feature shells. Validate through dependent flows, or add focused tests when changing behavior directly

apps/website is an Astro marketing site. Its useful signal is a successful static build plus targeted output checks, not a merged code coverage percentage. Projects with a test target but no specs, such as remote-control-web today, should not be in Tier A until focused specs exist.

For local coverage inspection:

pnpm run coverage:tools:test
pnpm run coverage:unit:ci
pnpm run coverage:merge
pnpm run coverage:health -- --require-report

The merged report is written to coverage/merged/ as HTML, LCOV, Cobertura, and JSON summary output. CI uploads the merged Tier A report to Codecov with the unit flag and keeps the HTML report as a GitHub artifact.

Each Tier A Jest run also writes coverage/<projectRoot>/junit.xml through jest-junit, with file paths as test class names for Codecov Test Analytics. The report lives inside the test target's coverage output so Nx cache restores include it, and concurrent projects cannot overwrite one another's results. CI uploads available JUnit reports with codecov-action@v7 and report_type: test_results, including after test failures, and keeps them in the unit-coverage artifact. Cancelled runs and runs without reports skip the test-results upload. Upload errors are non-blocking for this analytics step; test failures and the existing coverage upload remain blocking. Analytics covers executed Tier A tests only: projects skipped after the first failure and out-of-scope suites do not contribute results.

Passing the local coverage health checks does not guarantee that Codecov's patch check passes: it separately evaluates changed code, including partially covered branches, against the target reported for the current PR head. Inspect the affected files in the coverage report and add scenario coverage for missing paths before pushing; the overall project percentage alone is insufficient.

Codecov uploads require the repository upload token in the GitHub Actions repository secret CODECOV_TOKEN. The workflow passes that secret to the uploader and fails on upload errors; it only uploads after the complete Tier A coverage suite and health checks succeed. The artifact upload still runs on failure to preserve diagnostic reports. Fork pull requests use Codecov's tokenless fork uploads without access to the repository secret. Dependabot pull requests also lack Actions secrets, so their uploads use a dependabot:-prefixed Codecov branch for tokenless authentication.

Keep Codecov's default branch set to master. The README's Shields badge also selects master explicitly in its URL path (/iptvnator/master); a ?branch=master query parameter does not select a branch for that badge. If coverage appears stale, compare the Codecov commit with the GitHub run and check the upload log for authentication errors. Fix the secret rather than changing source exclusions or coverage ratchets to correct a reporting problem.

E2E

Area Command
Web app browser flows pnpm nx run web-e2e:e2e -- --project=chromium
Electron flows pnpm nx run electron-backend-e2e:e2e
Desktop-hosted server pnpm nx run electron-backend-e2e:hosted-server
VOD multi-source pnpm nx run electron-backend-e2e:e2e-ci--src/vod-multi-source.e2e.ts

The Windows hosted server job runs the dedicated Electron target. It checks that the server's instance ID matches the one in its dedicated SQLite database after restarting the desktop with an active source profile.

The Web E2E CI job runs Chromium with --workers=2; retry and timeout settings remain defined by the Playwright configuration. This override applies only to the Web E2E workflow step.

On pull requests, the CI Lint job summarizes which E2E suites project-level Nx affected selects for the PR merge diff. Use the preview to check that the graph includes each suite that consumes the changed code before using it to skip any jobs. The preview is informational. The E2E workflow keeps its existing path filters and matrix, and TV E2E remains a separate local target.

Use atomized E2E targets when available, for example pnpm nx run web-e2e:e2e-ci--src/xtream.e2e.ts.

After changing a Playwright config's webServer list, nx.json target defaults or an E2E project's dependencies, run pnpm run e2e:task-graphs:validate. It builds every Playwright target's task graph with and without CI and fails on the graphs Nx refuses to run; CI runs it in the unit-and-typecheck job. See Xtream mock Playwright integration.

Inside expect.poll, read a changing list in one DOM snapshot (allTextContents() or evaluateAll(), as in apps/electron-backend-e2e/src/sidebar-categories.e2e-support.ts), not by looping over count() with per-row nth(index) reads. Those reads auto-wait, so a row removed mid-loop hangs the predicate until the poll times out instead of letting it retry.

Chromium drops a scroll key pressed in the frame or two after the previous smooth scroll lands but before Blink handles its finish, so the final scrollTop is not a safe point for the next scroll input. Electron specs press scroll keys through pressScrollKey in apps/electron-backend-e2e/src/keyboard-scroll.ts, which waits for scrollend; web specs call waitForScrollIdle from apps/web-e2e/src/e2e-helpers.ts before the next key.

Playwright coverage is measured semantically by tags and critical journeys, not by a source-line percentage. E2E reports should use tags such as @critical, @electron, @web, @xtream, @stalker, @m3u, @search, @epg, @persistence, @settings, @pwa, and @self-hosted.

After an E2E run, generate the semantic summary with:

pnpm run coverage:e2e:summary

The electron-backend-e2e:e2e target owns its electron-backend:build-e2e dependency, including renderer and remote-control assets. CI calls that target without a separate production build before each shard.

CI runs the Electron suite as Playwright shards (--shard=<n>/<total>, split by spec file because the suite is sequential): three shards on Ubuntu, four on Windows, and two on macOS, whose runners are the scarcest and queued longest. Windows uses the additional shard to keep slower runners within the existing 30-minute job limit, including setup; each shard still uses one worker and the same tests, assertions, and retry policy. Each shard uploads playwright-report-electron-<os>-<n>; the follow-up Electron E2E summary job downloads the shards of each OS into their own directory and runs the summary per OS with --input=<directory> and --output-dir=coverage/e2e/<os>. A directory input merges every results.json beneath it and fails when a shard is missing or duplicated, so the summary never reports a partial run as complete. Tests for that merge live in tools/coverage/e2e-shard-reports.test.mjs (pnpm run coverage:tools:test).

The shared Electron launcher waits for Playwright's first window and selects a non-DevTools URL, including a later renderer when DevTools appears first. It then waits for the app root, visible rail, fonts and finite rail entrance animations, without a fixed startup sleep. Visibility comes first because source inventory loading can hide the workspace. electron-app-readiness.e2e.ts covers this delayed startup gate and animation completion. A visible startup failure alert is also ready for tests: failed migration/recovery scenarios need the Electron bridge while the workspace stays hidden. Window-selection and process-lifecycle regression tests run once per OS before the E2E suite. Every test still owns its temporary profile, and restarts still confirm the previous process exited before relaunching.

For local investigation only, Chromium browser V8 coverage can be explored with:

pnpm run coverage:e2e:v8:web

Production PWA update checks use pnpm nx run web-e2e:pwa-updates; its web:build-pwa dependency builds the matching service-worker output first. pnpm nx run web-e2e:test-service-worker validates the pinned worker behavior without a browser. Both are required after changes to update recovery or its Angular worker patch; ordinary dev-server browser flows do not exercise it.

I18n

pnpm run i18n:validate          # checker unit tests, then the check (CI)
pnpm run i18n:check             # the check only: locale drift, then key usage
pnpm run i18n:usage             # key usage only
pnpm run i18n:duplicates        # report: one English string, diverging translations
pnpm run i18n:baseline:update   # deliberate: rewrite the English-identical baseline

The i18n check is non-mutating. It compares every locale file in apps/web/src/assets/i18n/ against en.json and fails on missing or extra keys, empty/non-string values, changed interpolation placeholders (including their multiplicity), or changed inline HTML. The baseline never exempts these errors. It also fails on a locale value that is identical to English unless tools/i18n/identical-en-baseline.json records that exact English text for the same locale and key. The baseline holds values that are legitimately the same in a language (brand and technical names, language autonyms, loanwords such as "PIN") and the untranslated debt that existed when the guard was added, so new keys must ship translated. An entry stops covering its key once the English text changes, so copying reworded English into a locale fails as well.

Baseline entries that are no longer English-identical (translated, removed, or reworded) are reported but do not fail. pnpm run i18n:baseline:update rewrites the baseline from the current locale files, dropping those entries and printing every added one; it writes nothing while any locale is unreadable or has missing or extra keys. Run it only after translating, or for a value that is legitimately identical in that language; review the diff, and never run it in CI. A new locale starts with no baseline entries, so it has to record its legitimate identical values the same way. node tools/i18n/check-drift.mjs --fail-on-identical ignores the baseline for a full translation audit.

The usage check (tools/i18n/check-usage.mjs) fails on a key that renderer code uses but en.json lacks, because ngx-translate shows such a key raw. It reads production .ts and .html files under apps/web/src, apps/remote-control-web/src and libs, without comments, and treats as a use: a quoted key before | translate, or a ternary or fallback branch of a parenthesised operand before it; the first argument of instant, get or stream on a translate service, or of a function named like translate*, marker or t; and a dotted upper-case literal in an en.json namespace, which may also name a group of keys that code completes. The static prefix of a template literal such as `EPG.DIALOG.${name}` must name a group. Keys built entirely at runtime are not checked. CI runs pnpm run i18n:validate in the unit test job.

pnpm run i18n:duplicates (tools/i18n/check-duplicates.mjs) is a report, not a gate: it lists English strings defined under several keys that a locale translates differently. Align drifted wording in the locale files; a group may stay when context needs another grammatical form. --max <n> makes it fail above n groups.

Performance

pnpm nx build web
pnpm run perf:initial-bytes         # breakdown only
pnpm run perf:initial-bytes:check   # measure, then compare with the committed baseline
pnpm nx test performance-tools
pnpm run perf:journeys              # J1 launch, J2 open-source and J3 playback journeys, one dist/performance/journeys/<timestamp>/summary.json
pnpm nx run electron-backend-e2e:journeys-large-library   # J1 on the large-library profile (launch-large); skipped by perf:journeys unless IPTVNATOR_JOURNEY_LARGE_LIBRARY=1
pnpm nx run electron-backend-e2e:large-library-profile    # build or find that profile (cached by content hash under node_modules/.cache)
pnpm run perf:compositing           # tile memory and composited layers per route, dist/performance/compositing/<timestamp>/summary.json

perf:initial-bytes reads the built dist/apps/web/index.html and sums the bytes on the initial path (the J1 counter renderer.initialBytes). perf:initial-bytes:check then fails if the value exceeds tools/performance/journey-baselines.json; baselines only move down. CI runs the same check in the Initial bytes ratchet job of ci.yml for PRs that target master and for master pushes (dispatch it with gh workflow run ci.yml --ref <branch> for a stacked branch). The Performance journeys job checks the journey counters listed with --only in its Check the journey counters against the baselines step against the same file and fails the workflow on a regression; a new journey baseline must be added to that list too, which pnpm nx test performance-tools checks. The weekly performance-ratchet.yml workflow lowers baselines through a bot PR; validate a change to it with gh workflow run performance-ratchet.yml --ref <branch>, which measures but opens no PR off master. Dispatch needs the workflow file on master; before that, see the temporary-trigger note under Weekly tightening in the performance journeys document. perf:journeys builds the electron-performance configuration and runs every journey spec against the Xtream mock: J1 launch, then J2 open-source (a second set of launches, each followed by the click on the portal card), both written to the same summary file. The large-library variant of J1 (launch-large, a generated profile of 182 sources and 2.4M catalog rows) skips itself unless IPTVNATOR_JOURNEY_LARGE_LIBRARY=1, which the journeys-large-library target sets and the weekly performance-ratchet.yml passes through the composite action; pull-request CI does not run it (see "CI cost" in the journeys document). Its probe specs run with pnpm nx run electron-backend-e2e:test-performance-harness, which CI runs in the Unit Tests and Typechecks job of ci.yml on every run. tsx runs them without type checking; pnpm run typecheck:spec electron-backend-e2e checks the whole project, those specs included (see Unit And Type Checks). The electron-backend-e2e command targets call tsx and playwright directly, not through pnpm exec: under pnpm nx, a nested pnpm exec can run from the workspace root instead of the target cwd and miss cwd-relative specs, globs and configs. The contract, what counts and how to add a counter or a journey are in the performance journeys document. perf:compositing measures tile memory and composited layers on the artwork-heavy routes; the deterministic guard is compositing.e2e.ts in the Electron E2E suite (compositing budget).

Logging

Runtime playback and EPG debug logs should use the existing logger or trace helpers instead of unconditional console.log. Electron external-player traces are gated by:

IPTVNATOR_TRACE_PLAYER=1 pnpm run serve:backend

Test impact and completion

Before finishing a feature, bug fix, data-flow or UI workflow change, identify the affected projects and choose unit, integration, E2E, build, lint and manual checks. Bug fixes normally include regression coverage that fails before the fix. If automation is impractical, explain why and report the strongest manual validation. Update fixtures, mocks, routes and E2E flows when behavior changes. Prefer extending the closest existing suite to introducing a parallel suite.

Run targeted unit checks first, then affected E2E for workflows, routing, persistence, playback, portals, settings or import flows. Electron IPC, SQLite, packaged runtime, external players, native files and Electron-only routes require Electron E2E where available, otherwise CDP/manual verification using the debugging guide. Prefer atomized E2E targets before broad suites. Final reports name tests changed, commands/results, and skipped validation with reasons. Docs-only changes need Markdown validation, not app unit/E2E. Tooling validation still requires its own focused tests.

Agent guidance checks

pnpm run agents:validate checks root instruction budgets/imports and guidance navigation links/anchors. pnpm run skills:validate checks skill frontmatter, length, paths, required release skills and Claude symlinks. Both tooling suites run through pnpm nx test repository-skills; syntax checks use pnpm nx lint repository-skills. The CI guidance check runs regardless of the Nx affected set. Semantic preservation of moved contracts is a review task; link validation alone cannot prove it.