Skip to content

Make --emit-yaml output pasteable, and refuse a --refresh that would do nothing (#380) - #457

Merged
realmarcin merged 4 commits into
mainfrom
emit-yaml-width-380
Aug 7, 2026
Merged

Make --emit-yaml output pasteable, and refuse a --refresh that would do nothing (#380)#457
realmarcin merged 4 commits into
mainfrom
emit-yaml-width-380

Conversation

@realmarcin

Copy link
Copy Markdown
Contributor

Closes #380.

The divergence

apply_to_community dumped blocks at width=4096, one key per line. --emit-yaml used PyYAML's default, so the two long scalars wrapped:

    mapping_source: kg-microbe NCBI2GTDB.tsv.gz; GTDB release latest (built 2026-07-25) [grounded at g__
      rank; 1 GTDB taxa under the NCBI taxon]

--emit-yaml exists to be pasted into a record, so the two paths disagreeing is how wrapped blocks reached the KB.

This is not cosmetic — it is the root cause of the corruption #378 had to fix. Once both shapes exist, a line-level editor has to handle both, and that PR's first version matched exactly six spaces and orphaned the continuations into duplicate keys.

The fix

One constant, DUMP_WIDTH, used by both call sites. What keeps it fixed is a test that renders the same grounding down each path and compares bytes — otherwise a future change to either dump call (a width, a flow style, a sort) reintroduces #380 silently.

The secondary point, also real

--refresh only modifies how --apply treats existing blocks. On its own, or with --ncbi-id/--name where there is no stored block, it ran the ordinary report and exited 0 — leaving the caller believing a re-ground had happened. Argparse enforced neither:

$ gtdb_ground.py --community X.yaml --refresh          # before: normal report, exit 0
$ gtdb_ground.py --name Bosea --refresh                # before: normal report, exit 0

Both are now refused with a message saying what --refresh actually does. --community X --refresh --apply is unaffected, and a test pins that so the guard can't over-reach.

Scope

scripts/gtdb_ground.py only, +22/−2. No KB record changes. Six new tests. just qc green.

⚠️ CI note: GitHub Actions has produced no run on this repository since 14:50 UTC, so this PR will likely show no checks — the same stall blocking #456. Local just qc (lint + full suite + every offline validator) is green on this tree.

🤖 Generated with Claude Code

@realmarcin

Copy link
Copy Markdown
Contributor Author

Review round 1 — the test that mattered was a tautology

The reviewer confirmed the code is sound on every probe, and then found that my headline test proved nothing.

test_both_render_paths_agree rebuilt emit_block's body character-for-character and compared it to emit_block. It passed for any DUMP_WIDTH and never touched apply_to_community — whose dump is a different call: on the inner block, without default_flow_style, with a hand-prefixed indent. Changing the apply side left all six tests green, which is exactly the regression the file's docstring promises to catch.

It now applies a block to a real record and reads back what landed, comparing key-for-key against --emit-yaml and asserting no line wrapped. Verified against main's script: four tests fail, this one among them — before the rewrite, only the orphan check did.

Two smaller ones from the same review:

  • the fixture used a genus-rank lineage, which contains no whitespace and therefore cannot wrap — leaving gtdb_lineage, the scalar the module comment names as a wrapping risk, untested. It's a species lineage now, whose final segment carries a space.
  • the orphan detector keyed on the absence of a colon, which a wrapped scalar can contain. It keys on indentation instead.

What the review cleared

  • No workflow breaks. Repo-wide, the only --refresh invocations already carry --apply (gtdb_coherence.py:247 and a history record); the skill mentions it in prose only; justfile:472 is a generic passthrough; no CI workflow invokes the script.
  • The no-op diagnosis holds. args.refresh is read at exactly three places: the two new guards and the single apply_to_community call. withdraw_ambiguous and apply_status_to_community take no refresh, so refusing those combinations is right too.
  • width=4096 is safe. emit_block only prints to stdout; the KB already carries a 2014-char line, and no linter enforces YAML line length.

just qc green · 1570 passed, 16 skipped.

⚠️ Actions is still stalled repo-wide (no run since 14:50 UTC), so this will likely show no checks — same as #456.

realmarcin added a commit that referenced this pull request Aug 6, 2026
test_both_render_paths_agree rebuilt emit_block's body character-for-character
and compared it to emit_block. It passed for any DUMP_WIDTH and never touched
apply_to_community at all - whose dump is a different call, on the inner block,
without default_flow_style and with a hand-prefixed indent. Changing the apply
side left every test green, which is precisely the regression the file's
docstring promises to catch.

It now applies a block to a real record and reads back what landed, comparing
key-for-key against what --emit-yaml prints and asserting no line wrapped.
Verified: against main's script four tests fail, this one among them, where
before only the orphan check did.

Two smaller ones from the same review. The fixture used a genus-rank lineage,
which has no whitespace and so cannot wrap - leaving gtdb_lineage, the scalar
the module comment names as a wrapping risk, untested; it is a species lineage
now, whose final segment carries a space. And the orphan detector keyed on the
absence of a colon, which a wrapped scalar can contain; it keys on indentation
instead.

Also dropped a no-op assertion I left in the rewrite.

The review found nothing wrong with the code: no repo invocation of --refresh
lacks --apply or --community, so the guards break no workflow; args.refresh is
read at exactly one other place, the sole apply_to_community call, so the no-op
diagnosis holds; and width=4096 on the emit path cannot reach the KB except by
paste, where matching apply is the point.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@realmarcin

Copy link
Copy Markdown
Contributor Author

Cause identified: a GitHub Actions platform outage

Not this repo, not this PR. GitHub's own status API:

$ curl -s https://www.githubstatus.com/api/v2/components.json
  Webhooks: operational
  Actions:  major_outage

Repo and org Actions settings are both {"enabled": true, "allowed_actions": "all"}, so it is not a permissions or quota problem either. No workflow has run on this repository since 14:50 UTC (now past 20:00).

Nothing to do here but wait. This PR is ready: just qc green locally — lint, the full suite, and every offline validator, which between them are a superset of the lint and validate-strict jobs. It should merge on a green run once Actions returns.

realmarcin added a commit that referenced this pull request Aug 6, 2026
CI caught what no local run could. Two tests shell out to gtdb_ground.py, which
exits 1 with 'NCBI2GTDB mapping not found' without the kg-microbe crosswalk.
Locally resolve_kg_microbe_dir always finds it by walking to a sibling
directory, so the failure is invisible here and certain in CI.

They now take the mapping fixture that seven other test files already use for
the same reason. The two guard tests stay unguarded on purpose: the argparse
checks run before the mapping is resolved. Worth stating, because a missing
crosswalk also exits nonzero - so asserting the message and not just the exit
code is what stops those passing for the wrong reason.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
realmarcin and others added 4 commits August 6, 2026 20:01
…do nothing (#380)

apply_to_community dumped blocks at width=4096, one key per line, while
--emit-yaml used PyYAML's default. So gtdb_lineage and mapping_source - the two
long scalars - wrapped onto continuation lines indented deeper than the block
keys. --emit-yaml exists to be pasted into a record, so the two paths
disagreeing is how wrapped blocks reached the KB in the first place.

Not cosmetic: it is the root cause of the corruption #378 had to fix. A
line-level editor then has to handle both shapes, and that PR's first version
matched exactly six spaces and orphaned the continuations into duplicate keys.

One constant, DUMP_WIDTH, used by both. What keeps it that way is a test that
renders the same grounding down each path and compares bytes - a future change
to either dump call reintroduces #380 silently otherwise.

The issue's secondary point, also real: --refresh only modifies how --apply
treats existing blocks, so on its own, or with --ncbi-id/--name where there is
no stored block, it ran the ordinary report and exited 0 - leaving the caller
believing a re-ground had happened. Argparse enforced neither. Both are now
refused with a message saying what --refresh actually does, and the one
combination that works is unaffected.

No KB records change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test_both_render_paths_agree rebuilt emit_block's body character-for-character
and compared it to emit_block. It passed for any DUMP_WIDTH and never touched
apply_to_community at all - whose dump is a different call, on the inner block,
without default_flow_style and with a hand-prefixed indent. Changing the apply
side left every test green, which is precisely the regression the file's
docstring promises to catch.

It now applies a block to a real record and reads back what landed, comparing
key-for-key against what --emit-yaml prints and asserting no line wrapped.
Verified: against main's script four tests fail, this one among them, where
before only the orphan check did.

Two smaller ones from the same review. The fixture used a genus-rank lineage,
which has no whitespace and so cannot wrap - leaving gtdb_lineage, the scalar
the module comment names as a wrapping risk, untested; it is a species lineage
now, whose final segment carries a space. And the orphan detector keyed on the
absence of a colon, which a wrapped scalar can contain; it keys on indentation
instead.

Also dropped a no-op assertion I left in the rewrite.

The review found nothing wrong with the code: no repo invocation of --refresh
lacks --apply or --community, so the guards break no workflow; args.refresh is
read at exactly one other place, the sole apply_to_community call, so the no-op
diagnosis holds; and width=4096 on the emit path cannot reach the KB except by
paste, where matching apply is the point.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pushes during the outage (Actions was in major_outage from ~14:50 to ~22:00 UTC)
produced no workflow run. Empty commit to queue one now that events are being
accepted again.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CI caught what no local run could. Two tests shell out to gtdb_ground.py, which
exits 1 with 'NCBI2GTDB mapping not found' without the kg-microbe crosswalk.
Locally resolve_kg_microbe_dir always finds it by walking to a sibling
directory, so the failure is invisible here and certain in CI.

They now take the mapping fixture that seven other test files already use for
the same reason. The two guard tests stay unguarded on purpose: the argparse
checks run before the mapping is resolved. Worth stating, because a missing
crosswalk also exits nonzero - so asserting the message and not just the exit
code is what stops those passing for the wrong reason.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@realmarcin
realmarcin force-pushed the emit-yaml-width-380 branch from 9efc9b1 to c42aca1 Compare August 7, 2026 03:02
@realmarcin
realmarcin merged commit d30c927 into main Aug 7, 2026
3 checks passed
@realmarcin
realmarcin deleted the emit-yaml-width-380 branch August 7, 2026 03:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

gtdb_ground.py --emit-yaml and --apply dump at different widths, producing wrapped blocks

1 participant