Skip to content

Show refined/edited/restyled images in candidates panel #2174

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
SajalChaplot wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Show refined/edited/restyled images in candidates panel #2174

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
bff1d70
Pass action context through image generation actions
builderio-bot Jul 15, 2026
7c92d92
Merge remote-tracking branch 'refs/remotes/origin/main' into ai_main_…
builderio-bot Jul 16, 2026
567ad5f
Add integration tests for image variant tray generation
builderio-bot Jul 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 11 additions & 6 deletions templates/assets/actions/edit-image.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,12 +33,15 @@ describe("edit-image", () => {
});

it("delegates to generate-image as a source-guided full-image edit", async () => {
await action.run({
assetId: "asset-target",
instruction: "Make the background navy",
tier: "fast",
source: "chat",
});
await action.run(
{
assetId: "asset-target",
instruction: "Make the background navy",
tier: "fast",
source: "chat",
},
{ threadId: "thread-9" } as any,
);

expect(generateImageRunMock).toHaveBeenCalledWith(
expect.objectContaining({
Expand All @@ -52,7 +55,9 @@ describe("edit-image", () => {
groundingMode: "off",
includeLogo: false,
tier: "fast",
appendVariant: true,
}),
{ threadId: "thread-9" },
);
});
});
43 changes: 24 additions & 19 deletions templates/assets/actions/edit-image.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { defineAction } from "@agent-native/core";
import type { ActionRunContext } from "@agent-native/core/action";
import { z } from "zod";

import { IMAGE_MODELS, IMAGE_QUALITY_TIERS } from "../shared/api.js";
Expand All @@ -23,25 +24,29 @@ export default defineAction({
),
}),
parallelSafe: true,
run: async (args) => {
run: async (args, context?: ActionRunContext) => {
const asset = await getAssetOrThrow(args.assetId);
return generateImage.run({
libraryId: asset.libraryId,
collectionId: asset.collectionId ?? undefined,
prompt: args.instruction,
aspectRatio: (asset.aspectRatio ?? "16:9") as any,
imageSize: (asset.imageSize ?? "2K") as any,
model: args.model,
tier: args.tier,
intent: "edit",
styleStrength: "balanced",
referenceAssetIds: [],
includeLogo: false,
groundingMode: "off",
subjectAssetId: asset.id,
slotId: args.slotId,
source: args.source,
callerAppId: args.callerAppId,
});
return generateImage.run(
{
libraryId: asset.libraryId,
collectionId: asset.collectionId ?? undefined,
prompt: args.instruction,
aspectRatio: (asset.aspectRatio ?? "16:9") as any,
imageSize: (asset.imageSize ?? "2K") as any,
model: args.model,
tier: args.tier,
intent: "edit",
styleStrength: "balanced",
referenceAssetIds: [],
includeLogo: false,
groundingMode: "off",
subjectAssetId: asset.id,
slotId: args.slotId,
source: args.source,
callerAppId: args.callerAppId,
appendVariant: true,
},
context,
);
},
});
9 changes: 9 additions & 0 deletions templates/assets/actions/generate-image.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,12 @@ export default defineAction({
.describe(
"Internal UI state scope for live candidate slots. Usually omitted; embedded picker UIs pass a browser-tab scope.",
),
appendVariant: z.coerce
.boolean()
.default(false)
.describe(
"Internal: when true, add this candidate to the existing live tray for its scope instead of resetting to a fresh candidate set. Refine/edit/restyle set this so iterative results accumulate alongside prior candidates (newest-first).",
),
dismissible: z.coerce
.boolean()
.default(true)
Expand Down Expand Up @@ -650,6 +656,7 @@ export default defineAction({
sessionId: session?.id ?? null,
threadId: context?.threadId ?? null,
variantScopeId,
appendVariant: args.appendVariant,
prompt: args.prompt,
slotId,
status: "pending",
Expand Down Expand Up @@ -854,6 +861,7 @@ export default defineAction({
sessionId: session?.id ?? null,
threadId: context?.threadId ?? null,
variantScopeId,
appendVariant: args.appendVariant,
prompt: args.prompt,
slotId,
status: "ready",
Expand Down Expand Up @@ -901,6 +909,7 @@ export default defineAction({
sessionId: session?.id ?? null,
threadId: context?.threadId ?? null,
variantScopeId,
appendVariant: args.appendVariant,
prompt: args.prompt,
slotId,
status: "failed",
Expand Down
66 changes: 37 additions & 29 deletions templates/assets/actions/refine-image.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { defineAction } from "@agent-native/core";
import type { ActionRunContext } from "@agent-native/core/action";
import { z } from "zod";

import { parseJson } from "../server/lib/json.js";
Expand Down Expand Up @@ -30,18 +31,21 @@ export default defineAction({
),
}),
parallelSafe: true,
run: async ({
assetId,
feedback,
presetId,
sessionId,
model,
aspectRatio,
imageSize,
slotId,
source,
callerAppId,
}) => {
run: async (
{
assetId,
feedback,
presetId,
sessionId,
model,
aspectRatio,
imageSize,
slotId,
source,
callerAppId,
},
context?: ActionRunContext,
) => {
const asset = await getAssetOrThrow(assetId);
if (sessionId) {
await requireGenerationSessionInLibrary(sessionId, asset.libraryId);
Expand All @@ -55,22 +59,26 @@ export default defineAction({
"",
"Preserve the strongest successful parts of the prior candidate unless the feedback contradicts them.",
].join("\n");
return generateImage.run({
libraryId: asset.libraryId,
collectionId: asset.collectionId ?? undefined,
presetId,
sessionId,
prompt,
aspectRatio: (aspectRatio ?? asset.aspectRatio ?? "16:9") as any,
imageSize: (imageSize ?? asset.imageSize ?? "2K") as any,
model: (model ?? asset.model ?? "gemini-3.1-flash-image") as any,
categories: metadata.category ? [metadata.category] : undefined,
includeLogo: false,
groundingMode: "auto",
sourceAssetId: asset.id,
slotId,
source,
callerAppId,
});
return generateImage.run(
{
libraryId: asset.libraryId,
collectionId: asset.collectionId ?? undefined,
presetId,
sessionId,
prompt,
aspectRatio: (aspectRatio ?? asset.aspectRatio ?? "16:9") as any,
imageSize: (imageSize ?? asset.imageSize ?? "2K") as any,
model: (model ?? asset.model ?? "gemini-3.1-flash-image") as any,
categories: metadata.category ? [metadata.category] : undefined,
includeLogo: false,
groundingMode: "auto",
sourceAssetId: asset.id,
slotId,
source,
callerAppId,
appendVariant: true,
},
context,
);
},
});
19 changes: 12 additions & 7 deletions templates/assets/actions/restyle-image.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,13 +33,16 @@ describe("restyle-image", () => {
});

it("delegates to generate-image with the subject first and restyle intent", async () => {
await action.run({
subjectAssetId: "asset-subject",
prompt: "Make it match the launch campaign",
styleStrength: "strong",
tier: "best",
source: "chat",
});
await action.run(
{
subjectAssetId: "asset-subject",
prompt: "Make it match the launch campaign",
styleStrength: "strong",
tier: "best",
source: "chat",
},
{ threadId: "thread-9" } as any,
);

expect(generateImageRunMock).toHaveBeenCalledWith(
expect.objectContaining({
Expand All @@ -51,7 +54,9 @@ describe("restyle-image", () => {
styleStrength: "strong",
tier: "best",
includeLogo: false,
appendVariant: true,
}),
{ threadId: "thread-9" },
);
});
});
45 changes: 25 additions & 20 deletions templates/assets/actions/restyle-image.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { defineAction } from "@agent-native/core";
import type { ActionRunContext } from "@agent-native/core/action";
import { z } from "zod";

import {
Expand Down Expand Up @@ -37,29 +38,33 @@ export default defineAction({
),
}),
parallelSafe: true,
run: async (args) => {
run: async (args, context?: ActionRunContext) => {
const subject = await getAssetOrThrow(args.subjectAssetId);
const prompt =
args.prompt?.trim() ||
"Apply this library's brand style to the subject image while preserving the subject, pose, composition, and framing.";
return generateImage.run({
libraryId: subject.libraryId,
collectionId: subject.collectionId ?? undefined,
presetId: args.presetId,
sessionId: args.sessionId,
prompt,
aspectRatio: (args.aspectRatio ?? subject.aspectRatio ?? "16:9") as any,
imageSize: (args.imageSize ?? subject.imageSize ?? "2K") as any,
model: args.model,
tier: args.tier,
intent: "restyle",
styleStrength: args.styleStrength,
includeLogo: false,
groundingMode: "auto",
subjectAssetId: subject.id,
slotId: args.slotId,
source: args.source,
callerAppId: args.callerAppId,
});
return generateImage.run(
{
libraryId: subject.libraryId,
collectionId: subject.collectionId ?? undefined,
presetId: args.presetId,
sessionId: args.sessionId,
prompt,
aspectRatio: (args.aspectRatio ?? subject.aspectRatio ?? "16:9") as any,
imageSize: (args.imageSize ?? subject.imageSize ?? "2K") as any,
model: args.model,
tier: args.tier,
intent: "restyle",
styleStrength: args.styleStrength,
includeLogo: false,
groundingMode: "auto",
subjectAssetId: subject.id,
slotId: args.slotId,
source: args.source,
callerAppId: args.callerAppId,
appendVariant: true,
},
context,
);
},
});
64 changes: 64 additions & 0 deletions templates/assets/actions/variant-slots.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -280,4 +280,68 @@ describe("variant slot state", () => {
expect.objectContaining({ slotId: "slot-2", status: "pending" }),
]);
});

it("appends a refined candidate to the existing scope instead of resetting", async () => {
await Promise.all(
["slot-1", "slot-2", "slot-3"].map((slotId, index) =>
upsertVariantSlot({
runId: `run-${index + 1}`,
batchId: "batch-1",
libraryId: "lib-1",
prompt: "First prompt",
slotId,
status: "ready",
assetId: `asset-${index + 1}`,
threadId: "thread-1",
}),
),
);

// A refine is a distinct run with no batchId; without appendVariant this
// would reset the tray. appendVariant keeps prior candidates and adds this.
await upsertVariantSlot({
runId: "refine-run",
libraryId: "lib-1",
prompt: "First prompt\n\nUser feedback:\nmake it warmer",
slotId: "refine-slot",
status: "ready",
assetId: "asset-refined",
threadId: "thread-1",
appendVariant: true,
});

expect(
(appStateByKey["asset-variants:thread-1"] as any).slots.map(
(slot: any) => slot.slotId,
),
).toEqual(["slot-1", "slot-2", "slot-3", "refine-slot"]);
});

it("does not append across libraries even when appendVariant is set", async () => {
await upsertVariantSlot({
runId: "run-1",
batchId: "batch-1",
libraryId: "lib-1",
prompt: "First prompt",
slotId: "slot-1",
status: "ready",
assetId: "asset-1",
threadId: "thread-1",
});

await upsertVariantSlot({
runId: "refine-run",
libraryId: "lib-2",
prompt: "Different library",
slotId: "refine-slot",
status: "ready",
assetId: "asset-refined",
threadId: "thread-1",
appendVariant: true,
});

expect((appStateByKey["asset-variants:thread-1"] as any).slots).toEqual([
expect.objectContaining({ slotId: "refine-slot", status: "ready" }),
]);
});
});
11 changes: 10 additions & 1 deletion templates/assets/actions/variant-slots.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,11 @@ type VariantSlotInput = VariantScopeInput & {
previewUrl?: string;
thumbnailUrl?: string;
error?: string;
// When true, add this candidate to the existing live tray for this scope
// instead of starting a fresh candidate set. Refine/edit/restyle set this so
// iterative results accumulate alongside the prior candidates (newest-first)
// rather than replacing them.
appendVariant?: boolean;
};

let variantStateLock: Promise<void> = Promise.resolve();
Expand Down Expand Up @@ -96,8 +101,12 @@ export async function upsertVariantSlot(input: VariantSlotInput) {
await withVariantStateLock(async () => {
const scopeId = variantScopeIdFor(input);
const previous = await readVariantStateUnlocked(scopeId);
const appendToPrevious =
previous != null &&
input.appendVariant === true &&
previous.libraryId === input.libraryId;
const state =
previous && isSameVariantScope(previous, input)
previous && (isSameVariantScope(previous, input) || appendToPrevious)
Comment on lines +104 to +109

@builder-io-integration builder-io-integration Bot Jul 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Appending a refinement lets in-flight batch completions reset the tray

appendToPrevious retains the prior batch's slots, but the unconditional assignments above replace that retained state's runId/batchId with the refinement's run, whose batch ID is null. If a user refines an already-ready candidate while siblings from the original generate-image-batch are still pending, a sibling's later ready/failed upsert has appendVariant: false and its original batch ID; it no longer matches the state and takes the reset branch, dropping the refinement and other candidates. Preserve the original batch identity for append-only writes or merge late in-flight sibling updates by their original slot/run identity.

Additional Info 
Confirmed by independent code-review agent; focused variant-slots tests passed but do not cover refinement during an in-flight batch.

Fix in Builder

Comment on lines +104 to +109

@builder-io-integration builder-io-integration Bot Jul 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Appending a refinement overwrites the tray prompt

When this branch reuses the existing tray state for appendToPrevious, the unconditional metadata assignments immediately below replace its prompt with the refinement prompt. GenerationResults displays this field as the candidate-panel subtitle, so after refining a candidate the tray loses the original batch prompt and shows the longer feedback text instead. Preserve the original tray prompt for append operations, or track refinement context separately while only replacing metadata for a new generation scope.

Additional Info 
Found by an independent incremental review agent; the added integration test verifies slot preservation but not the prompt shown after append.

Fix in Builder

? previous
: {
runId: input.runId,
Expand Down
Loading
Loading