[oss] Offline tests for openai_compat and remote_inproc adapters

[fstandhartinger]

What changed

Adds one file, tests/test_openai_compat_adapter.py (358 lines, test-only). No library code, datasets, results or scoring changed.

HTTP is mocked by monkeypatching the module-level http_post_json that each adapter imports. For timeout and HTTP-error cases the tests patch urllib.request.urlopen underneath it instead, so the real base.http_post_json path runs. No sockets are opened.

Coverage, pinned to current behaviour:

• Typed answer: probs, probs_source="verbalized", the served model name winning over the requested one, URL with the trailing slash stripped, auth headers, timeout_s passthrough, and no expected/provenance in the request. Also: a missing API key sends no request, and request_options overrides and removals work.
• Probability extraction: a thought channel before the JSON; with several candidates the last one wins; markdown fences; a {"probabilities": ...} object nested under another key is accepted; labels and values are not validated here (the scorer does that).
• Malformed JSON / missing fields: each case is pinned to its exact error string: invalid or truncated JSON, empty content, wrong top-level key, extra keys, array content, probabilities that is a list or null, null content, list content, no or empty choices, a choice with no message, and a 200 response with an HTML body.
• 429/5xx: the adapter does no retries. Each status (400/401/429/500/502/503/529) leads to exactly one call, with error == "HTTP <status>: <body[:300]>". Retry/stop handling lives in the runner (it stops on 401/403/429), which this PR does not touch.
• Timeout: a ConnectionError from the transport becomes ok=False, status=None. A real socket.timeout through http_post_json gives "TimeoutError: timed out". Other exceptions propagate.
• Tokens/cost: prompt_tokens/completion_tokens are mapped to input_tokens/output_tokens, OpenAI-style names win when both are present, and null usage gives Nones. reserve_estimate returns 0.232 for $2/$8 per million tokens, 0 when both prices are 0, and None when a price is missing.
• remote_inproc: an ok response, a failure reported by the server, missing fields, server_latency_s folded into raw.runtime, 429/5xx with no retry, and a timeout.

Bugs found (recorded with xfail, not fixed)

1. OpenAICompatAdapter.run changes the provider's raw response. res.usage = parsed.get("usage") is the same dict object as parsed["usage"]. Adding input_tokens/output_tokens to it also adds them to DecisionResult.raw, which the runner hashes and stores as preserved raw evidence. test_raw_response_not_mutated is xfail(strict=True); test_raw_usage_currently_mutated pins today's behaviour. A one-line fix would be dict(parsed.get("usage") or {}), but it is left out because this PR must not change library code.

Open doubts

• If choices[0] is not a dict (e.g. [None]) or usage is not a dict, the adapter raises AttributeError. The runner then records only error="AttributeError" and loses the HTTP status and raw body. I pinned this as current behaviour (pytest.raises) rather than marking it as a bug.
• A distribution nested under another key ({"answer": {"probabilities": ...}}) is accepted, while a top-level object with extra keys is rejected. I pinned this as current behaviour; it may or may not be intended.
• The parse-error strings include Python's json messages and could change between Python versions (these ran on 3.13).

Verification

Before (main @ 99783d8), python -m pytest -q:

199 passed, 5 subtests passed in 4.64s

New file alone:

51 passed, 1 xfailed in 0.20s

After, python -m pytest -q:

250 passed, 1 xfailed, 5 subtests passed in 4.28s

Nothing is left for Sandy: no credentials, network or GPU were needed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01N6vx2pRLRQWgVMnaYC1GXL

---

Generated by Claude Code