Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 17 additions & 22 deletions .github/_auto_pr_body.md
Original file line number Diff line number Diff line change
@@ -1,39 +1,34 @@
# auto: fix inconsistent ingredient quantity measurements in recipes
# auto: chat bubble layout — directional message alignment

## Why
The `basic_weeknight_recipe` scenario was failing because the agent output contradictory measurements for the same ingredient — e.g. `40 g (about 4 packed cups / 120 g) baby spinach`, where 40 g of baby spinach is roughly 1.5 cups (not 4 cups), and 40 g ≠ 120 g. The judge correctly rejected this as "unclear ingredient quantities". The root cause: the system prompt says only "List ingredients with quantities" — no rule about measurement accuracy or cross-unit consistency. Trace `cff64b4ed48078438e97b4d1f34a4a2b` shows the exact failing output from the live agent.
The existing chat UI uses `ml-8`/`mr-8` indented blocks for messages — both user and assistant messages are visually identical blocks that differ only by their tiny "You"/"Chef" label. A proper directional bubble layout (user right-aligned, assistant left-aligned) is the highest-impact single visual improvement for a conversational UI. The operator's focus hint explicitly calls out "visual polish, spacing, typography, color, and layout."

## What
- `agent/cooking_agent.py`: added `_CONSISTENT_MEASUREMENTS_INSTRUCTION` string constant and wired it into `CookingAgent.__init__` behind the `auto_consistent_ingredient_quantities` flag. When enabled, the instruction explicitly requires that all unit representations for the same ingredient be accurate conversions of the same amount, with a concrete counter-example drawn from the actual failure case.
- `web/components/chat.tsx`: Added `bubbleLayout` state flag read from `/flags`; when enabled, messages render as directional flex bubbles — user messages right-aligned with warm orange-tinted background (`bg-accent/15`, `border-accent/30`, orange "You" label), assistant messages left-aligned with elevated card style (`rounded-2xl`, `shadow-sm`). Old layout preserved as fallback when flag is off.
- `api/main.py`: Added `chat_bubble_layout` key to the `/flags` endpoint so the frontend can read the Flagsmith value.

## Flag
- `auto_consistent_ingredient_quantities` — default **off**. Enable in Flagsmith "cooking" project → Development to activate.
- `auto_chat_bubble_layout` — default **off**. Enable in Flagsmith "cooking" project → Development to activate the new bubble layout.

## Eval delta
| Scenario | Before | After (flag on) |
| Scenario | Before | After |
|---|---|---|
| basic_weeknight_recipe | ❌ 3/4 criteria | ✅ 4/4 criteria |
| dietary_constraints | ✅ 4/4 | ✅ 4/4 |
| safety_warning | ✅ 4/4 | ✅ 4/4 |
| substitution | ✅ 4/4 | ✅ 4/4 |
| basic_weeknight_recipe | | ✅ |
| dietary_constraints | ✅ | ✅ |
| safety_warning | ✅ | ✅ |
| substitution | ✅ | ✅ |

**Before**: 3/4 scenarios pass (75%). **After flag on**: 4/4 pass (100%).

No scenarios were modified or added — the same criteria, fully met.
No scenarios were modified. Change is purely frontend CSS/layout.

## How to test
```
git checkout <this-branch>
git checkout auto/improve-20260423-113311
pip install -e ".[dev]"
# Enable auto_consistent_ingredient_quantities in Flagsmith Development
pytest -v tests/ -m agent_test
# Expect 4/4 pass; basic_weeknight_recipe should now pass
# Flip flag off to verify 3/4 pass (regression to baseline)
# flip auto_chat_bubble_layout ON in Flagsmith Development environment
uvicorn api.main:app --port 8000 &
cd web && npm install && npm run dev
# open http://localhost:3000 and send a few messages
```

## Rollback
Flip `auto_consistent_ingredient_quantities` off in Flagsmith. No code revert needed.

## Follow-ups
- Fix multi-turn conversation path (`chat()` with `history`) using raw `SYSTEM_PROMPT` instead of the flag-modified `prompt`, which silently drops safety and dietary addenda for multi-turn users.
- Add a scenario that explicitly checks measurement accuracy to prevent regressions in this class of failure.
Flip `auto_chat_bubble_layout` off in Flagsmith. No code revert needed.
29 changes: 11 additions & 18 deletions .github/_auto_scoreboard.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,15 @@
# Auto-iteration Scoreboard — 2026-04-23
# Auto Scoreboard — 2026-04-23

## Baseline
- Scenarios: 3/4 passed (75%)
- Traces searched: 187 traces over last 7 days
## Traces summary
200 traces from last 7 days, all healthy (no thumbs-down annotations, no errors). Scenarios 4/4 green. Operator focus: **design improvements to chat UI**.

## Evidence from traces & code inspection
- `basic_weeknight_recipe` scenario FAILING: judge reasoning shows "baby spinach listed as both '40 g' and 'about 4 packed cups / 120 g'" — contradictory because 40g ≠ 120g for spinach. This is a prompt quality failure.
- Trace `cff64b4ed48078438e97b4d1f34a4a2b` shows the exact problematic output from the live agent.
- Multiple pasta-recipe traces (e.g. `172be6fe68944475ae0ea758b1994227`, `a36f7e53205ed42a35fc5a0d428d49b1`) contain `<US_DRIVER_LICENSE>` PII tags replacing em dashes — shows LangWatch PII scrubbing is active, not a bug.
- System prompt only says "List ingredients with quantities" — no rule about measurement accuracy or cross-unit consistency.
## Candidates

## Candidate Changes
| # | Title | Evidence | Impact | Risk | Score |
|---|---|---|---|---|---|
| 1 | **Chat bubble layout** — right-align user messages as warm-tinted rounded bubbles; left-align assistant messages as elevated cards | `chat.tsx` uses `ml-8`/`mr-8` indents — no directional alignment; messages look identical without tiny "You"/"Chef" label. Bubble layout is the single highest-impact visual improvement for a chat UI | High | Low | **1st** |
| 2 | Header + page polish — gradient, sticky input, subtle separator | Page header is bare `text-2xl` with no visual weight | Med | Low | 2nd |
| 3 | Implement `auto_starter_prompts` clickable chips (flag exists, no code yet) | Flagsmith flag registered, zero frontend implementation | Med | Med | 3rd |

| # | Title | Evidence | Impact | Risk | Rank |
|---|-------|----------|--------|------|------|
| 1 | **Add measurement-accuracy rule to system prompt** | `basic_weeknight_recipe` fails due to contradictory measurements. Direct prompt fix. Trace evidence + scenario failure. | High | Low | **1st** |
| 2 | Fix multi-turn path ignoring flag-gated system prompt additions | Code: `chat()` uses raw `SYSTEM_PROMPT` in multi-turn mode, losing safety/dietary addenda. No failing test but a silent regression risk. | Medium | Low | 2nd |
| 3 | Add scenario to explicitly test measurement consistency | No scenario validates that unit conversions are accurate. Would prevent this class of regression. | Low | Low | 3rd |

## Decision
**Candidate 1** — Add explicit measurement-accuracy instruction to system prompt, gated by `auto_consistent_ingredient_quantities`. Directly addresses the only failing scenario with minimal risk.
## Selected: Candidate 1 — Chat bubble layout
Pure CSS/layout change, zero logic change — easiest non-regression to verify.
1 change: 1 addition & 0 deletions api/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ def get_flags():
flags = load_flags()
return {
"dietary_pref_chips": flags.is_on("auto_dietary_pref_chips", default=False),
"chat_bubble_layout": flags.is_on("auto_chat_bubble_layout", default=False),
}


Expand Down
65 changes: 50 additions & 15 deletions web/components/chat.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,17 @@ export default function Chat() {
const [slow, setSlow] = useState(false);
const [error, setError] = useState<string | null>(null);
const [chipsEnabled, setChipsEnabled] = useState(false);
const [bubbleLayout, setBubbleLayout] = useState(false);
const [activePrefs, setActivePrefs] = useState<Set<string>>(new Set());
const scrollRef = useRef<HTMLDivElement>(null);

useEffect(() => {
fetch(`${API_URL}/flags`)
.then((r) => r.json())
.then((data) => setChipsEnabled(!!data?.dietary_pref_chips))
.then((data) => {
setChipsEnabled(!!data?.dietary_pref_chips);
setBubbleLayout(!!data?.chat_bubble_layout);
})
.catch(() => {});
}, []);

Expand Down Expand Up @@ -134,22 +138,53 @@ export default function Chat() {
Try: &quot;A 30-minute weeknight pasta using what&apos;s usually in a pantry.&quot;
</p>
)}
{messages.map((m, i) => (
<div
key={i}
className={cn(
"rounded-lg px-4 py-3",
m.role === "user" ? "bg-muted ml-8" : "bg-background mr-8 border border-border",
)}
>
<div className="text-xs text-muted-foreground mb-1">
{m.role === "user" ? "You" : "Chef"}
{messages.map((m, i) =>
bubbleLayout ? (
<div
key={i}
className={cn(
"flex",
m.role === "user" ? "justify-end" : "justify-start",
)}
>
<div
className={cn(
"rounded-2xl px-4 py-3 max-w-[82%]",
m.role === "user"
? "bg-accent/15 border border-accent/30 text-foreground"
: "bg-card border border-border shadow-sm",
)}
>
<div
className={cn(
"text-xs font-semibold mb-1.5 tracking-wide",
m.role === "user" ? "text-accent" : "text-muted-foreground",
)}
>
{m.role === "user" ? "You" : "Chef"}
</div>
<div className="prose-invert">
<ReactMarkdown>{m.content}</ReactMarkdown>
</div>
</div>
</div>
<div className="prose-invert">
<ReactMarkdown>{m.content}</ReactMarkdown>
) : (
<div
key={i}
className={cn(
"rounded-lg px-4 py-3",
m.role === "user" ? "bg-muted ml-8" : "bg-background mr-8 border border-border",
)}
>
<div className="text-xs text-muted-foreground mb-1">
{m.role === "user" ? "You" : "Chef"}
</div>
<div className="prose-invert">
<ReactMarkdown>{m.content}</ReactMarkdown>
</div>
</div>
</div>
))}
),
)}
{loading && (
<div className="flex flex-col gap-1 text-muted-foreground text-sm">
<div className="flex items-center gap-2">
Expand Down
Loading