[omerdduran]

The list stopped being fed when the hunter crawler was retired, and it froze for 105 days before anyone noticed. This is the pipeline that keeps it fed — one PR a week, never auto-merged.

Sources

crt.sh — new numeric mirrors. Brand tokens are derived from the gambling list itself, so a new brand needs no code change. Queries use %brand% because the mirrors are separate registrable domains, not subdomains. Internal hosts are rejected: a probe of bets10.com returned 138 hostnames and every one was a QA subdomain (api.qa., casinoblack.test.). Each query retries with backoff — crt.sh answered 502 on the first attempt and 200 on the second while I was measuring — and a dead source never stops the others. Candidates must resolve before they are proposed.

liveness — dead-rule pruning. Plain ||domain^ rules are probed with the bounded prober (the one that fixed the release hang). A rule is proposed for removal only after two rounds (≈2 weeks) and a second resolver via DoH, because one resolver's NXDOMAIN is not proof: parked domains, resolver-level filtering and CDN behaviour all look the same. If the second opinion cannot be obtained, nothing is removed.

graveyard — resurrection. A removed domain stays tracked for 90 days and is proposed for re-addition if it answers again. That is exactly how gambling mirrors behave, and it was the flaw in my first design: once deleted, nothing looked again.

New rules younger than 30 days are never pruned.

Output

• data/liveness.json — auditable state (first_seen, last_alive, dead_streak, graveyard with removed_at)
• additions land in a marked block ! ══ OTOMATİK ADAYLAR (crt.sh) ══, sorted, rewritten in place so a second run cannot duplicate them
• the PR body lists every candidate with its source, plus a warning when a new brand is missing from GAMBLING_PATTERNS — otherwise that domain silently lands in the lite list instead of the gambling one

Wiring

• weekly cron (Mon 04:00 UTC) + workflow_dispatch (with tokens / skip_crt inputs)
• shares the main-writes concurrency group, so it cannot race the version bump
• a run with no list change commits only the state, with [skip ci]
• 20 network-free unit tests in make test-unit and in the PR gate

Demonstrated, not just written

path	result
crt.sh	found casibom--747.com — resolves, not in the list
prune	two rules (dolarkurum.com, r.yabancidizi.in) proposed after 2 rounds + DoH agreement, then removed and kept in the graveyard
resurrection	a graveyard entry that answers again is proposed for re-addition
failure	betcup query returned HTTPError → retried, logged, other sources continued

Worth knowing before merging

• The first scheduled run will open its own PR with the casibom--747.com candidate (and possibly casibom missing from the gambling patterns). That data PR is deliberately not part of this one.
• data/liveness.json starts from today's probe: 1 426 domains, 2 marked dead once.