test(integration): cover triggered skill and path-rule content across condensation

[georgeglarson]

HUMAN:

Human reviewed. These tests should help make the skill condensation behavior known

---

AGENT:

Why

#4544 reports that keyword-triggered skills and path-triggered rules are injected once per conversation, on an event condensation is allowed to forget, while the "already activated" marker lives in persistent state and is never cleared. After a condensation the guidance can be gone from everything the model sees while state still reports it as active.

The issue deliberately did not prescribe a mechanism, and asked maintainers to confirm the intended lifecycle instead. These tests run the whole scenario end to end so that question can be settled against observed behavior rather than against a reading of the code.

#4552 proposes a fix for the same issue at the session level. These tests sit a layer above it, in the live c* integration suite it does not touch, so they are complementary rather than competing.

Summary

• c06_triggered_skill_survives_condensation.py: keyword-triggered skill, activated outside keep_first, real work, real summarizing condensation, then the keyword again.
• c07_path_rule_survives_condensation.py: the same shape for a PathTrigger rule, driven by real file edits through the file editor so injection happens in the production callback.
• BaseIntegrationTest gains an overridable agent_context property defaulting to None, mirroring the existing tools and condenser hooks. Without it the harness builds Agent() with no context and no skill can be installed.

Both assert one invariant: after condensation, triggered content is either still in the active view or recoverable when the trigger fires again. Preserve, reconstruct and reactivate all satisfy it, so neither test prejudges the fix.

They fail on current main by design, and the failure message is the diagnosis. The c* suite is optional and non-blocking, so a red case here does not gate anything.

Issue Number

#4544

How to Test

Run against any OpenAI-compatible endpoint:

export LLM_API_KEY=... LLM_BASE_URL=...
uv run python tests/integration/run_infer.py \
  --llm-config '{"model":"openai/<model>","max_output_tokens":8192,"native_tool_calling":true}' \
  --test-type condenser \
  --eval-ids c06_triggered_skill_survives_condensation,c07_path_rule_survives_condensation

Observed on anthropic/claude-opus-5, against 1de2e6d1b:

c06_triggered_skill_survives_condensation -> FAIL
  #4544 REPRODUCED: after condensation the skill body is absent from the
  active view and a fresh trigger does not bring it back, while conversation
  state still reports the skill as activated.
  activated_knowledge_skills=['python_tips'], sentinel_in_view=False,
  retrigger_activated_skills=[], retrigger_extended_content=[]

c07 fails on every run, but lands on either of two verdicts. Across six runs on the same model it gave SUMMARY-ECHO ONLY four times and #4544 REPRODUCED twice:

c07_path_rule_survives_condensation -> FAIL
  SUMMARY-ECHO ONLY, not a contract: the marker appears in rendered text but
  NOT in the injection channel. The agent quoted the guidance back, that
  message reached the summarizer, and the summary carried it.

c07_path_rule_survives_condensation -> FAIL
  #4544 REPRODUCED: after condensation the rule body is absent from the active
  view and touching the matching path again re-injects nothing, while
  conversation state still reports the rule as activated.
  activated_path_rules=['style_rule'], sentinel_in_view=False,
  second_touch_extended=[]

Both verdicts are the same failure. The injection channel is empty either way, so the contract fails; which one you land on depends on whether the model happened to quote the guidance back before condensation. Expect either when running it.

Also run on glm-5.3.

Local checks on the committed tree: ruff check and ruff format --check clean, pyright clean, pytest tests/integration/ 27 passed and 1 skipped.

Video/Screenshots

Type

• Bug fix
• Feature
• Refactor
• Breaking change
• Docs / chore

Notes

Three things in the tests are deliberate and easy to undo by accident.

Verdicts are three-way, not two. A marker found only in rendered text gets its own SUMMARY-ECHO ONLY verdict, which is a failure. Injected guidance sits in the agent's context by design, so a capable model quotes it back, that message reaches the summarizer, and the summary carries the marker through. It reads as survival but is a coin flip: before this distinction existed, c07 went 2 pass / 4 fail across six identical runs on the same model. Presence is therefore measured only in the injection channel (extended_content), which matches the issue's ask that content be retained or recovered deterministically.

c06's sentinel sits past N_CHAR_PREVIEW (500), not merely inside a longer body. __str__ truncates the tail, so a front-loaded marker survives, reaches the summarizer, and gets preserved. An earlier draft did that and passed while the bug was live.

Presence is never measured with str(event). ObservationEvent.__str__ omits extended_content entirely, so that string can never contain path-rule text and would report "gone" regardless of truth. The rendered string is what the summarizer is fed; it is not what the agent sees.

verify_result also checks preconditions first and reports SCENARIO NOT SET UP separately from any verdict. In particular the carrying event must actually be absent from the view after condensation. If it survived, nothing was at risk and the run says so rather than pronouncing on the contract.

[github-actions]

📁 PR Artifacts Notice

This PR contains a .pr/ directory with temporary PR-specific documents. Because this is a fork PR, the directory will be automatically removed from main immediately after merge.

[all-hands-bot]

🤖 OpenHands is reviewing this PR.

Head commit: 86e067dadd0a2398f6140c9f16220790be38af35
View the conversation: https://oss-agent-canvas.ngrok.dev/conversations/73b05a23-3339-4864-ac59-00a77e09bb3c

This comment was posted by an AI agent (OpenHands).

[all-hands-bot]

This review was created by an AI agent (OpenHands) on behalf of the repository maintainers.

Summary

Test-only PR adding two condenser-suite integration tests (c06, c07) that reproduce issue #4544 (triggered skill/path-rule content silently lost after condensation), plus a clean agent_context hook on BaseIntegrationTest.

Risk Assessment: LOW

No production code is touched. Both tests live in the optional, non-blocking c* suite and are designed to FAIL on current main — the requested artifact per the issue acceptance criteria.

Findings

No material bugs, security issues, or design flaws found. The tests are well-constructed:

• base.py: The agent_context property addition mirrors the existing tools and condenser override hooks and is correctly wired into the Agent() constructor.
• c06 (keyword skill): Sentinel placement past N_CHAR_PREVIEW (500) verified — at position 1072, so MessageEvent.__str__() truncation guarantees the marker never reaches the summarizer. Keyword injection correctly flows through send_message() → get_user_message_suffix() with skip_skill_names=activated_knowledge_skills, matching the production dedup path.
• c07 (path rule): Uses the default tool preset (file editor included) so injection happens through the production _maybe_inject_path_rules() callback. Verified that ObservationEvent.__str__() omits extended_content and the Skill validator forces disable_model_invocation=True for PathTrigger — both claims in the test rationale are accurate.
• Three-way verdict system (PASS / SUMMARY-ECHO ONLY / REPRODUCED) is a sound design that avoids false positives from summary echoes while not prejudging the fix mechanism.
• Precondition checks in verify_result (carrier event actually forgotten, scenario actually set up) prevent both false passes and false reproductions.

All technical claims in the PR description were verified against the source code.