Prompt T0.5 (spike) — baseline-example anchoring: measure compliance gain vs parroting

[GustavoSena]

Prompt T0.5 (spike) — Baseline-example anchoring: measure compliance gain vs parroting

Part of prompt-building Tier 0. This one is a spike with a decision gate, not a straight feature — it has a real failure mode that must be measured before adoption.

Hypothesis

7B models comply far better when shown one full, valid worked example of the expected output. We already have a generator of exactly such an example: templateFallback(req, ctx) (packages/arbitration-sdk/src/fallback.ts) produces a deterministic, valid, request-specific recommendation. Rendering it into the prompt as a baseline —

BASELINE — a valid, conservative answer for THIS request. Improve on it per the
user's intent (band, fee, split, template choice). If you cannot improve on it,
returning it unchanged is acceptable.
<the fallback JSON>

— should cut malformed-output and violation rates, because the model edits a known-good object instead of composing one from a schema description.

The risk being measured

Parroting. If the model returns the baseline (near-)verbatim most of the time, we are paying enclave latency + ledger cost for fallback-quality output labelled ENCLAVE. The source label stays honest either way (it is a model output), but the "the model adds value" story dies, and Tier 0.3's tiering would collapse to whatever the baseline contains.

Also note: the example must be built from the request's live values (the fallback already is) — never a static example, which would hand the model stale constants to echo (the exact chainId/deadline fabrication failure compose.ts documents).

Method

1. Branch-local prompt variant behind a flag/env (SLUICE_PROMPT_BASELINE=1) in the compose path — no permanent wiring.
2. Fixed set of ≥ 20 compose requests (mix: single-token and two-token budgets, varied prompts incl. conservative/aggressive wording, maxStrategies 1 and 3), run against the live Galileo provider with and without the baseline block. Reuse the harness style of spike/inference-spike.ts / the compose CLI.
3. Record per run: malformed rate (parse failures), violation rate + which invariants, attempts-to-accept, fallback rate, parrot rate (accepted output identical to baseline, or differing only in deadline/whitespace — define the comparator in the spike), latency, prompt+completion token counts.

Decision gate

Adopt (as a follow-up PR) only if BOTH hold, otherwise close with the numbers in the write-up:

• malformed + violation rate drops materially (suggested bar: ≥ 30% relative reduction), AND
• parrot rate stays below ~1 in 3 accepted runs.

Either way, the outcome and numbers go to Notion F2 §9 (prompt contract) — this is a prompt-contract decision, and promptVersion must bump if adopted.

Acceptance criteria

• Spike script + fixed request set committed under spike/ (throwaway quality is fine; reproducibility is not optional).
• The comparison table (both arms, all metrics above) in the issue/PR write-up.
• Go/no-go recorded here and on Notion F2 §9.

Coordination

Run AFTER the in-flight prompt-refactor PR lands (the variant must be built on the surviving prompt builder, or the numbers measure a dead prompt).

References

• packages/arbitration-sdk/src/fallback.ts — the baseline generator
• packages/arbitration-sdk/src/compose.ts — fabrication history, attempt cap
• Notion F2 §9: https://app.notion.com/p/3a8caae5863181609acbcfd69a5db06b

[GustavoSena]

Status: blocked on a funded key, not on code. The rest of Tier 0 has landed as #38 (T0.4) and #39 (T0.1–T0.3), and the prompt-refactor gate this issue waited on cleared when #30/#32 merged — so the only thing still missing is the thing the spike measures with: a funded Galileo key in the run environment.

Two notes that change the spike's shape now that #39 is up:

• The baseline arm should render templateFallback() as it exists after Prompt Tier 0: derive risk appetite, pairing and band tiers (T0.1–T0.3) #39 — the prompt now carries appetite, reference pairing and band tiers, so a baseline measured against the old prompt would be measuring a prompt that no longer exists.
• Worth adding to the metrics table: whether the model echoes the precomputed values (pairing amounts, tier bandBps) or re-derives them. That is the assumption all of Tier 0 rests on, and nobody has measured it against a live 7B yet — arguably more valuable than the baseline question this issue was opened for.

Everything else in the method stands.

[GustavoSena]

Acceptance criterion 1 (script + fixed request set) is done in #40 — packages/arbitration-sdk/spike/baseline-anchoring.ts, 24 requests, two arms, gate evaluated by the script.

Still blocked on a funded Galileo key for criteria 2 and 3 (the comparison table and the go/no-go). When one exists:

ZG_PRIVATE_KEY=0x... npx tsx spike/baseline-anchoring.ts --limit 4   # rehearse
ZG_PRIVATE_KEY=0x... npx tsx spike/baseline-anchoring.ts             # full run, ~96 inferences

Fund the ledger first with npm run fund; the script never funds. Paste the printed markdown table straight into this issue, then record the go/no-go on Notion F2 §9.

The harness also measures echo fidelity — whether the model uses the pairing amounts and band tiers we precompute, or re-derives them. That was not in the original method, and it may matter more than the baseline question: it is the assumption #24/#25/#26 all rest on.