EPG Acquisition and Merge — multi-lineup identity, validation, provenance, and deterministic XMLTV #40

Current main status — 2026-08-16

GitHub main at 88b81fb contains delivered local XMLTV functionality across fd1e556, e7c1168, 519828e, 6fa6797, and 88b81fb: local .xml, gzip, and single-entry ZIP ingestion; streaming validation with DTD/external-entity protections and bounded decompressed input; Programme and evidence/provenance records; deterministic merge; deterministic XMLTV export; and Build-Lineup integration with fail-closed publication.

Recent comments that describe source catalogs, preflight or web surfaces, source-metadata reconciliation, SQLite/cache, or 884/884 validation refer to dirty or disposable worktree state and are not implementation evidence for main. Historical comments remain unchanged for auditability; this section is the authoritative mainline status.

Remaining #40 ownership is the broader EPG parent scope: canonical station identity and channel/guide binding; coverage and fit analysis; source quality, freshness, and conflict reporting; provider/catalog and guide-adapter concerns; scheduling/refresh; mirrors and rights; and broader downstream EPG behavior.

#89 owns XMLTV remote acquisition/transport/cache work. #88 owns provider M3U live fetch and remains separate.

Mission

Build a provider-neutral EPG Acquisition and Merge pipeline for traditional linear channels that can fetch, validate, normalize, compare, and deterministically merge multiple XMLTV/guide lineups while preserving station identity, provenance, freshness, and user policy.

Gracenote2XML is a useful research reference for scheduled guide acquisition, account/lineup configuration, multiple-lineup merging, timespan/timezone controls, station/callsign ordering, and client-specific XMLTV output options. ChannelForge must implement a lawful, source-neutral architecture and must not require that container, Gracenote, Zap2it, or any single upstream service.

Acquire guides safely, preserve identity, merge with evidence, and never hide conflicts behind one convenient XML file.

Product outcomes

A user should be able to:

1. Add one or more XMLTV files, URLs, or supported guide-provider connections.
2. Test access and preview lineup/station coverage before saving.
3. Schedule safe refreshes with source-aware intervals and bounded retries.
4. See freshness, completeness, overlap, conflicts, and unmapped stations.
5. Merge sources deterministically according to explicit policies.
6. Preserve stable station identity across lineup renames, number changes, and source refreshes.
7. Publish validated, atomic XMLTV output suitable for supported downstream clients.
8. See an explicit per-source outcome (Fetched, CacheHit, Stale, Failed, or NeedsReview) when an upstream source fails or returns stale data, rather than a silently substituted 'good enough' result.

Guide catalog and fit analysis

Provide a catalog layer for discovering user-authorized public, private, and local guide feeds without confusing catalog metadata with guide truth.

Catalog entries may be organized by:

• country, continent, market, postal/lineup context, language, or theme
• standard station-ID/callsign edition
• IPTV-friendly channel-name-plus-country ID edition
• individual market, country, continent, or combined/full guide
• look-back/look-ahead horizon such as 3, 7, or 14 days
• XML/XMLTV and supported compressed variants
• generation method/source family
• mirror or alternate delivery endpoint

Each entry should expose a machine-readable manifest where available or derived safely:

• stable catalog/source ID
• display name and geographic/market coverage
• channel count and identifier scheme
• sample or complete channel manifest without downloading all programme data when possible
• programme coverage start/end
• generated/updated time, content length, hash, and compression
• source family and mirror relationship
• intended-use, licensing, redistribution, and attribution terms
• expected refresh cadence
• availability/health history
• schema/parser warnings
• supported compatibility profiles

Fit analyzer

Compare catalog manifests against the user's canonical lineup before subscription:

• exact canonical-station coverage
• exact and candidate tvg-id/station-ID coverage
• callsign/name candidates requiring review
• missing lineup channels
• extra guide channels
• market/regional conflicts
• duplicated coverage across selected guides
• estimated programme window and download/storage cost
• rights or intended-use incompatibility

Recommend the smallest suitable guide or combination that meets the user's coverage and horizon needs. A country or market guide should normally be preferred over a continent/full guide when coverage is equivalent, reducing downloads, parsing, storage, and conflict surface.

Never silently switch identifier schemes. Standard station-ID/callsign and IPTV-friendly channelname.countrycode editions must be represented as explicit compatibility choices with a preview of expected mapping changes.

Mirrors and resilience

• model mirrors as alternate delivery endpoints for one logical source
• fail over between mirrors only when terms and content identity permit
• verify content/hash/generation metadata before treating mirrors as equivalent
• do not count agreement between mirrors of the same generated guide as independent evidence
• avoid duplicate downloads and duplicate programme evidence
• preserve the last-known-good endpoint/content during outages
• show mirror divergence as a source-integrity warning

Catalog UX

• search and filter guides by geography, market, language, theme, horizon, format, identifier scheme, rights, and health
• preview channels before adding a guide
• show predicted lineup coverage and unresolved mappings
• compare candidate guides side by side
• copy or save a source definition without exposing credentials
• warn before selecting oversized full guides or incompatible intended-use terms
• keep user/private catalog entries local unless explicitly exported

Source types

Support a common EpgSource boundary for:

• local XMLTV files
• remote XMLTV URLs
• compressed XMLTV variants
• generated XMLTV outputs
• authenticated guide-provider adapters where lawful and maintainable
• multiple lineups/accounts from the same provider
• supplemental sources that contribute only selected stations or programme fields

Every adapter must declare capabilities such as:

• lineup discovery
• station discovery
• programme window limits
• incremental fetch support
• authentication method
• timezone semantics
• source-provided identifiers
• image/metadata fields
• rate-limit and refresh guidance
• terms/licensing notes requiring user acknowledgment where applicable

Normal XMLTV ingestion must not require any provider adapter.

Provider interface

Define a contract equivalent to:

EpgSourceProvider
  id
  capabilities()
  validateConfiguration()
  testConnection()
  discoverLineups()                 # optional
  discoverStations(lineup)          # optional
  fetch(window, cursor, conditionalHeaders)
  checkpoint()
  health()

Requirements:

• bounded time windows and response sizes
• conditional requests where supported
• pagination/incremental cursors where supported
• explicit timezone and locale behavior
• idempotent replay
• structured authentication, authorization, rate-limit, malformed-data, and transient errors
• cancellation, timeouts, backoff, and jitter
• adapter/source version in provenance
• no provider-specific types beyond the adapter boundary

Canonical station identity

Normalize guide channels into stable station candidates using evidence such as:

• source station ID
• Gracenote/TMS or other authoritative IDs when lawfully supplied
• XMLTV channel ID
• callsign
• affiliate/network
• channel number and subchannel
• country, market, postal/lineup context
• language
• normalized display names
• historical identity and user-approved mappings

Rules:

• keep external IDs namespaced by authority/source
• never assume equal numeric IDs from different authorities are equivalent
• callsign/name matching is supporting evidence, not proof across markets
• lineup channel-number changes should not create a new station when stronger identity evidence agrees
• ambiguous station merges require review
• preserve aliases and identifier history with provenance
• allow split, merge, pin, and rollback

This complements the channel identity and Match Inspector work in #25 and #27.

Programme normalization

Preserve and normalize, where available:

• title, subtitle, description
• start/stop timestamps and original timezone/offset evidence
• categories/genres
• episode numbering systems
• new/live/premiere/finale/rerun flags
• ratings and advisories
• credits
• language
• icon/artwork references
• source programme/series IDs
• station identity
• source fetch time and provenance

Do not invent missing programme facts. Output transformations such as appending markers to titles must be optional presentation policies, never modifications to canonical facts.

Multi-lineup merge

Merge station and programme data using explicit, deterministic policies.

Station merge

• detect exact authoritative-ID agreement
• preserve lineup-specific numbers and aliases
• identify duplicate, overlapping, and conflicting station candidates
• allow source/field priority by station, market, or lineup
• retain all contributing sources and reason codes

Programme merge

• identify candidate duplicates by canonical station, time interval, IDs, and normalized programme identity
• distinguish true duplicates from adjacent episodes, overruns, regional variants, and schedule corrections
• resolve field differences by explicit priority, freshness, completeness, and trust policy
• retain contradictory evidence
• avoid counting mirrored copies as independent corroboration
• allow supplemental sources to fill missing fields without replacing stronger canonical values
• never silently discard a conflict that could materially change what the guide shows

Source quality and coverage

Track per source/lineup:

• last attempt and success
• data generation time when available
• coverage start/end
• station count
• programme count
• empty/invalid station rate
• gaps and overlaps
• timestamp validity
• identifier stability
• schema/parse warnings
• agreement and contradiction with other sources
• historical freshness and correctness evidence
• credential/rate-limit/availability state

Present quality as explainable dimensions rather than one opaque score. Source popularity or repetition does not prove correctness.

Refresh scheduling

• choose safe defaults from source guidance, cache headers, and coverage window
• support configurable look-back and look-ahead windows
• refresh before coverage expires
• use conditional fetches and content hashes
• add bounded jitter and exponential backoff
• avoid synchronized refreshes across lineups
• respect provider limits and account/lineup constraints
• on authentication, network, parsing, or merge failure, resolve the source to Failed or NeedsReview; retain the prior validated evidence record for inspection/rollback, but never republish it as current successful output
• validate complete candidate output before replacing active state
• support manual refresh, cancellation, progress, partial success, and degraded mode
• do not require the GUI to remain open

Related unattended operation: #35.

XMLTV validation and security

Treat every guide as hostile input:

• streaming parse for large documents
• prohibit external entity expansion and unsafe DTD/network resolution
• bound nesting, text/attribute sizes, channel/programme counts, decompression ratio, total bytes, and parse duration
• validate timestamp syntax/ranges and reject pathological intervals
• sanitize URLs and image references
• validate gzip/zip content and prevent archive traversal
• reject unsupported content types safely
• redact credentials, signed URLs, cookies, headers, and source paths from logs/diagnostics
• store credentials only in the existing secret mechanism
• apply SSRF and outbound-host policy to remote sources/adapters
• use atomic staging and publication
• preserve a sanitized validation report and source hash, not secrets or unnecessary raw account responses

Rights and provider policy

• document the source, access method, refresh limits, and intended-use constraints
• require the user to supply and authorize their own credentials where needed
• do not bypass paywalls, authentication controls, rate limits, CAPTCHAs, or technical restrictions
• do not ship shared credentials
• do not imply that guide data, station metadata, or artwork may be redistributed merely because it is technically retrievable
• keep proprietary provider adapters optional and independently disableable
• allow lawful user-supplied XMLTV to remain the universal path
• provide deletion/export controls for cached provider data and credentials

Source stewardship and retirement

Guide catalogs and endpoints can outlive, lose, or change maintainers. ChannelForge must model that as an operational and trust-boundary event, not as permission to take over or redistribute the source.

• Record publicly documented project stewardship, repository/archive state, endpoint identity, source family, licensing/intended-use evidence, attribution, expected cadence, last success, and last verified content hash.
• Never store private biographical or social-media details about maintainers.
• Treat a maintainer's absence or death as neither proof that a service is abandoned nor a change to copyright, license, attribution, personal-use, or redistribution terms.
• Distinguish an outage from a stale source, deliberate retirement, archive, successor-operated continuation, unexpected ownership transfer, and suspected compromise.
• Revalidate redirects, domains, certificates, signing keys, repository/package ownership, content structure, and terms before trusting a successor endpoint.
• Do not automatically grant a new owner the prior source's trust score or treat a renamed/rehosted feed as identical without verifiable provenance and content identity.
• Pin optional generators and containers to reviewed versions or immutable digests; do not make a mutable image tag or personal account part of normal operation.
• Provide disable, replace, export, and migration workflows while preserving canonical station mappings and rights-permitted last-known-good data.
• Recommend alternate sources based on coverage, rights compatibility, health, and identifier fit; never silently substitute a source with materially different terms.
• Cross-link lifecycle detection, degraded operation, and notification behavior to Autonomous Operations — scheduled refresh, multi-guide learning, source drift detection, and zero-touch maintenance #35.

Output compatibility profiles

Keep one canonical internal representation and apply explicit output profiles only at serialization time.

Profiles may control:

• channel ordering by number, callsign, display name, or station ID
• display-name ordering/selection
• episode-number element ordering
• optional live/new markers
• timezone/offset serialization
• icon handling
• past/future programme window
• deterministic channel/programme ordering
• compression and filename

Requirements:

• defaults follow XMLTV conventions
• profiles are named, versioned, previewable, and testable
• transformations are visible in the build report
• no client-specific behavior contaminates canonical storage
• identical canonical data and profile version produce byte-stable output where practical

GUI

Provide a guided workflow:

1. Choose source type.
2. Enter location/credentials using secret-safe controls.
3. Test connection.
4. Discover/select lineups or validate the XMLTV source.
5. Preview stations, coverage, timezone, freshness, and warnings.
6. Choose a recommended refresh/window policy.
7. Review overlaps and merge policy.
8. Save and run an initial staged import.

Default health view:

• source status and coverage horizon
• last/next refresh
• stations/programmes imported
• gaps, conflicts, and unmapped stations
• per-source outcome state (Fetched/CacheHit/Stale/Failed/NeedsReview)
• items needing review

Advanced mode may expose field priorities, window controls, serialization profiles, and adapter diagnostics.

Non-goals

• Making Gracenote, Zap2it, Gracenote2XML, or any proprietary guide service mandatory.
• Shipping or sharing upstream-service credentials.
• Circumventing provider access controls or terms.
• Replacing a commercial guide-data license.
• Treating source-specific station IDs as globally universal.
• Flattening conflicting sources without provenance.
• Editing canonical titles merely to satisfy a client display preference.
• Loading untrusted XML with external entities or unbounded decompression.
• Making guide acquisition dependent on Docker.
• Combining sports schedule providers with linear EPG adapters behind one misleading interface; they may share canonical evidence models while retaining distinct source contracts.

Phased delivery

Phase 0 - contracts, XMLTV profile, and threat model

• Define EpgSourceProvider, source capabilities, provenance, error taxonomy, and source lifecycle/trust states.
• Define the canonical station/programme model and supported XMLTV subset/extensions.
• Threat-model XML, compressed input, remote fetches, credentials, and proprietary providers.
• Create sanitized multi-lineup fixtures with conflicts, gaps, malformed data, and timezone cases.

Phase 1 - universal XMLTV ingestion

• Add local/remote/compressed XMLTV sources.
• Implement streaming validation, structural inventory, content hashing, and staged import.
• Add source health, coverage, last-known-good, atomic replacement, and deterministic serialization.

Phase 2 - identity and merging

• Add canonical station resolution with external-ID namespaces and review.
• Implement deterministic station/programme merge policies with conflict preservation.
• Add source quality dimensions, overlap/gap analysis, and field-level provenance.

Phase 3 - scheduling and compatibility

• Add unattended refresh, conditional fetching, coverage-aware scheduling, backoff, and degraded mode.
• Add source retirement/replacement, ownership-change validation, and migration workflows aligned with Autonomous Operations — scheduled refresh, multi-guide learning, source drift detection, and zero-touch maintenance #35.
• Add named output compatibility profiles and preview.
• Add GUI setup, health, merge-review, and serialization-report workflows.

Phase 4 - optional provider adapters

• Define adapter eligibility based on lawful access, stability, documentation, and testability.
• Add selected adapters independently behind the provider contract.
• Add capability/version matrices, credential deletion, rate-limit tests, and offline fixtures.
• Ensure removal of every proprietary adapter leaves universal XMLTV ingestion fully functional.

Acceptance criteria

• Local, remote, and compressed XMLTV sources ingest without a proprietary provider account.
• Catalog entries expose geography, horizon, format, identifier scheme, freshness, source family/mirrors, and rights metadata.
• Users can preview a guide's channel manifest and predicted lineup coverage before downloading full programme data when the source permits.
• Fit analysis recommends the smallest adequate guide set and explains missing, extra, conflicting, and duplicated coverage.
• Standard and IPTV-friendly identifier editions remain explicit compatibility choices and never switch silently.
• Mirrors provide delivery resilience without being counted as independent corroborating sources.
• Multiple lineups merge deterministically while preserving field-level provenance and conflicts.
• External station IDs remain authority-namespaced and ambiguous merges require review.
• Renumbering or display-name changes do not break stable station identity when stronger evidence agrees.
• Programme duplicates, corrections, regional variants, gaps, and overlaps have deterministic tested behavior.
• Source health exposes coverage, freshness, validity, gaps, conflicts, and last-known-good state.
• Refresh scheduling respects coverage, conditional requests, rate limits, backoff, cancellation, and provider-safe defaults.
• Failed refreshes or merges never replace validated active guide state.
• XML parsing disables external entities and enforces size, depth, count, duration, and decompression limits.
• Credentials and sensitive URLs are absent from tracked configuration, logs, fixtures, reports, and diagnostics.
• Output profiles are explicit serialization policies and do not mutate canonical facts.
• Identical canonical data and profile version serialize deterministically.
• Proprietary adapters are optional, removable, and covered by sanitized/offline contract fixtures.
• Documentation states access requirements, refresh guidance, retention/deletion behavior, and rights limitations.
• Normal ChannelForge operation remains possible with only user-supplied XMLTV files.
• Stale, archived, retired, successor-operated, ownership-changed, and integrity-suspect sources have distinct tested behavior.
• A changed domain, repository owner, package publisher, signing key, or redirect cannot silently inherit prior trust.
• Source replacement preserves canonical mappings and valid provenance while retaining all applicable rights and attribution restrictions.

Dependencies and cross-links

• Channel Intelligence boundaries, identifiers, and interoperability: Define Channel Intelligence Engine architecture and interoperability roadmap #27
• Explainable matching and evidence inspection: Build Alias Tester and explainable Match Inspector #25
• Event/sports schedules and canonical event model: Event Intelligence Engine — automatic live-event discovery, schedule verification, and learned EPG generation #28-Automatic event discovery — classify channels, resolve identities, and match verified schedules #30
• Event EPG generation and publication: Event EPG delivery — generate, refresh, explain, and publish live-event lineups #32
• Autonomous refresh, multi-guide comparison, drift, last-known-good, and degraded mode: Autonomous Operations — scheduled refresh, multi-guide learning, source drift detection, and zero-touch maintenance #35
• Staged inventory, canonical storage, provenance, and indexed guide queries: High-Performance Data Store — fast identity, guide, event, provenance, and incremental-build indexing #36
• Stream health and provider-incident patterns: Stream Health Evidence — bounded probes, incident detection, and stable fallback ordering #39
• GUI-first path and safe configuration standards: Define the GUI-first product path #10, Secure local configuration workflow for Build-Lineup.ps1 (no editing tracked provider files) #20, Simplicity Standard — mandatory UX, documentation, configuration, and code clarity gate #33

Research references

• Gracenote2XML - scheduled guide retrieval, multiple account/lineup support, XMLTV merging, configurable coverage windows, timezone/output controls, station/callsign ordering, and downstream compatibility switches.
• epg.guru / epg.fyi - browsable country, continent, combined, themed, legacy, and individual-market guides; standard and IPTV-friendly identifier editions; multiple guide horizons; compressed downloads; channel previews; and explicit personal-use restrictions.
• epg.jesmann.com - earlier/catalog-related EPG service supplied by the same maintainer; retain as historical research context and verify any current source metadata before relying on it.

These are research references only, not dependencies or bundled guide sources. As observed, epg.guru and epg.fyi present the same EPG Hub content and should be modeled as related endpoints/mirrors rather than independent evidence. Their published personal-use restriction must be preserved and surfaced; ChannelForge must not include those guides in a commercial distribution or redistribute them without explicit permission.

Definition of done

ChannelForge can safely acquire, validate, normalize, compare, and merge multiple linear EPG sources into deterministic XMLTV output with stable station identity, complete provenance, coverage-aware refresh, last-known-good protection, and explicit client compatibility-without requiring a proprietary guide service or weakening provider access and rights boundaries.

ChannelForge product premise and scope guardrail

ChannelForge exists to make the user's existing content and services easier to discover and use.

Required product test

Before implementation, this issue must answer yes to the following question:

Does this reduce the effort required to find, understand, choose, or open content from services the user already has?

If the answer is no, unclear, or depends on building a parallel provider product, stop and re-scope the issue.

Scope rules

• Aggregate content and capabilities across providers into a clearer discovery experience.
• Use, link to, or delegate to provider-owned systems wherever available; do not recreate provider products.
• Keep providers as the systems of record for content, playback, recordings, schedules, entitlements, accounts, billing, storage, permissions, and service-specific state.
• Limit ChannelForge-owned behavior to what makes aggregation easier: permitted ingestion, normalization, identity matching, source comparison, presentation, routing, explainability, and safe configuration.
• Preserve provider attribution, access controls, terms, content rights, capability differences, and failure boundaries.
• Prefer the smallest integration that removes meaningful user effort. A deep link is better than a new subsystem when it solves the user's problem.
• Do not add setup, decisions, dashboards, automation, or maintenance burden unless the issue removes more complexity than it introduces.
• Default to a simple path for beginners; expose advanced controls only when they provide real value.

Definition-of-done addition

This issue is not complete until evidence shows that it makes a real user task easier without duplicating a provider capability or creating a second system of record.

Integrate before rebuild. Aggregate without taking ownership. Make things easier.

Current implementation status — 2026-08-16

Status: local XMLTV ingestion, deterministic merge/export, and Build-Lineup publication are implemented on GitHub main; broader EPG identity, catalog, freshness, scheduling, and remote transport work remains open.

Implemented on GitHub main at 88b81fb:

• local .xml, gzip, and single-entry ZIP XMLTV ingestion
• XmlReader streaming parsing with DTD/external-entity protections and bounded decompressed input
• Programme and evidence/provenance records
• deterministic source-aware merge and XMLTV export
• Build-Lineup local XMLTV integration with fail-closed publication

Not implemented on main and remaining under this epic:

• canonical station identity/binding and coverage/fit analysis
• source quality, freshness, conflict, and broader EPG behavior
• provider/catalog/guide-adapter concerns, mirrors, and rights
• scheduling/refresh
• downstream compatibility and broader output behavior
• remote XMLTV transport and disposable fetch cache (tracked under XMLTV EPG fetch/parse/cache — streaming parser and disposable fetch cache #89)

Claims in earlier comments about source catalogs, preflight or web surfaces, source-metadata reconciliation, SQLite/cache, or 884/884 validation were produced from dirty or disposable worktrees and are not evidence of mainline implementation.