refactor: complete multi-Actor monorepo layout — move Dockerfile into actors/runner/ - #13
Merged
Merged
Conversation
3 tasks
DaveHanns
force-pushed
the
consolidate-to-single-actor
branch
3 times, most recently
from
July 2, 2026 20:50
95bebfa to
5bfb290
Compare
DaveHanns
changed the title
DaveHanns
force-pushed
the
consolidate-to-single-actor
branch
2 times, most recently
from
July 2, 2026 21:33
fa738f6 to
73d2811
Compare
… actors/runner/ + delete duplicate root .actor/ This repo is a monorepo intended to hold multiple Apify Actors under actors/* (the runner today; an orchestrator and possibly analyzer later). For that layout to be coherent, EVERYTHING Actor-specific must live inside the Actor's own directory alongside its .actor/ manifest and src/ tree — not at the repo root where a reader has to guess which Actor it belongs to. Two moves: 1. Dockerfile → actors/runner/Dockerfile The runner's build config lives with the runner. 2. Delete the duplicate root .actor/ Before this PR, root .actor/ mirrored actors/runner/.actor/ — one Actor had two manifests, and it was unclear to a reader which one the platform actually reads. Only actors/runner/.actor/ survives as canonical. Any future actors/<name>/ Actor gets its own .actor/ alongside its code. Adjustments: - actors/runner/.actor/actor.json — dockerfile path adjusted to point at the new Dockerfile location; dockerContextDir stays "../../.." so the Docker build context is still the repo root (necessary for COPY shared/... to reach the shared/ workspace). - test/run-vm-test.sh, CLAUDE.md, docs/02-decisions.md — path references updated where they clearly meant the runner's build config. Adding actors/orchestrator/ in a future PR becomes purely additive: copy the runner's structure into a sibling directory, no root changes required. Supersedes an earlier attempt on this branch that flattened in the WRONG direction (pulled the runner up to root, would have blocked adding a second Actor). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
DaveHanns
force-pushed
the
consolidate-to-single-actor
branch
from
July 2, 2026 21:46
73d2811 to
ba3c184
Compare
6 tasks
DaveHanns
added a commit
to DaveHanns/apify-evals
that referenced
this pull request
Jul 2, 2026
…chema The Console's dataset "Views" tab is driven by actors/runner/.actor/dataset_schema.json. The shape landed in b17cb10 (alongside judgeMode + OTel) tried to project every metric as a dot-path column and split verdicts into a second view. Nice telemetry, but the tab lost its "at-a-glance" property: overallVerdict got demoted from column 1 to column 5, testPrompt and agentOutput were dropped, and the second Verdicts view rendered each check as one opaque JSON blob. This PR replaces .views with a single Eval Results view — the five columns that answer "what happened on this test": Test # | Verdict | Prompt | Agent Output | Checks verdicts is rendered as an inline array (compact list of check objects) rather than a full unwound table. If someone needs per-check drill-down, the raw dataset item is still one click away in the Data tab; the Views tab optimizes for scanning. Also deletes the stale root-level .actor/dataset_schema.json — the pre-flatten copy that survived PR chocholous#13 and confused everyone about which file was live. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…chema Rolled into this PR from #22 so the multi-Actor layout + view fix can be tested in a single branch. The Console's dataset "Views" tab is driven by actors/runner/.actor/dataset_schema.json. The shape from b17cb10 (judgeMode + OTel) traded scannability for dot-path metric columns; user feedback was "the previous view was much clearer". Replaces .views with a single Eval Results view — the five columns that answer "what happened on this test": Test # | Verdict | Prompt | Agent Output | Checks verdicts is rendered as an inline array (compact list of check objects) rather than a full unwound table. Per-check drill-down is one click away in the raw Data tab, so the Views tab optimizes for scanning. Also deletes the stale root-level .actor/dataset_schema.json — the pre-flatten copy that survived this PR's restructure. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…nd Verdict Checks view Two adjustments after testing the previous 5-column view against a real run: 1. Primary "Eval Results" view: drop the Test # column. With testPrompt visible, the numeric index was redundant — the prompt itself identifies the test. 2. Add a secondary "Verdict Checks" view that unwinds the verdicts array so each check renders as its own row. Columns: Test | Verdict | Check Type | Check | Evidence. Test # retained here because unwound rows blend across tests otherwise. Users can now scan per-check detail on the Views tab without opening raw JSON. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
DaveHanns
force-pushed
the
consolidate-to-single-actor
branch
from
July 3, 2026 09:01
417ca0f to
10f7fc7
Compare
Previous attempt used `unwind: ["verdicts"]` + fields `verdicts.checkType` etc., which Apify Console silently ignored — the view showed 2 dataset rows with only testIndex populated instead of unwinding into per-check rows. Corrected per Apify actor-output-schema docs (see PR body). The Verdict Checks view now produces one row per element of the verdicts array, with columns for Test | Verdict | Check Type | Check | Evidence. Ref: run jaN6GrFkd8yOR1GwH exposed the mis-rendering. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…Name column The unwound Checkpoints view only ever contains one scenario per run's dataset, so the Scenario column was pure noise. Title shortened to "Checkpoints" to match the domain vocabulary (checkpoint scripts + checkpoint field on AgentResult). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ex", omit LLM-judge meta fields Three small tweaks after the last test run: 1. Title "Checkpoints" -> "Tests" — user preference; the view groups check rows by test, and "Tests" reads better as the primary axis. 2. testIndex column label "#" -> "Test Index" — same reason, consistent with the view name. 3. Explicit omit of evalCritique + evalGapSeverity. Apify Console auto-derives columns from every top-level key of the transformed item, so when a verdict happened to have evalCritique populated (LLM judge's meta-feedback about the eval criteria, ~5% of rows in observed data) it leaked into the view as an unformatted column. These fields are for the eval author, not the agent-behaviour triager — they belong in raw JSON only. omit runs after unwind in Apify's transform pipeline (flatten -> pickFields -> unwind -> omit), so this cleanly drops them from the view without affecting anything else. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Complete the multi-Actor monorepo layout in one commit:
Dockerfile→actors/runner/Dockerfile(Actor-specific file where it belongs).actor/— one Actor should have one manifestNet change: 4 file changes + 3 deletions = 7 files touched.
Why
The repo is intentionally a monorepo of Actors under
actors/*. For the layout to make sense, each Actor's.actor/manifest, Dockerfile, and source code all live insideactors/<name>/— not scattered between root and nested dirs. Before this PR, one Actor had two manifests (root.actor/mirroringactors/runner/.actor/) and its Dockerfile at repo root, which reads as "the Actor is here… no wait, over there, but built from way up top".After:
actors/runner/is completely self-contained. Addingactors/orchestrator/in a future PR is purely additive.Layout after merge
Diff
Dockerfile→actors/runner/Dockerfile(rename, content unchanged).actor/(root duplicate) — deleted (3 files: actor.json, dataset_schema.json, input_schema.json)actors/runner/.actor/actor.json—dockerfilefield points at the new locationtest/run-vm-test.sh,CLAUDE.md,docs/02-decisions.md— path references updatedTest plan
npm installclean (verified locally)npm test --workspaces --if-presentpasses (247 shared tests, tsc clean)davehan/apify-evalsafter merge to confirm the platform picks up the new Dockerfile pathHistory note
Supersedes an earlier attempt on this branch that flattened in the wrong direction. Force-pushed twice. Current tip: this commit.
🤖 Generated with Claude Code
Rolled in from #22 (2026-07-03): commit
32f9645— restore simple 5-column dataset view (Test # | Verdict | Prompt | Agent Output | Checks) + drop the stale root-level.actor/dataset_schema.jsonthat this PR's restructure left behind. Reviewers testing the Actor build should now see the intended Views tab layout instead of the busier metric-column version.Views tune-up (2026-07-03): commit
10f7fc7— dropTest #from the primary "Eval Results" view (redundant with the visible prompt) and add a secondary "Verdict Checks" view that unwinds theverdictsarray so each check renders as its own row (columns: Test | Verdict | Check Type | Check | Evidence). Per-check detail is now scannable on the Views tab without opening raw JSON.