The agency benchmark
The explorer measures a tool in isolation. This measures tools and skills as workers inside a real project. Every run is a ticket for one fictional client — Verdant Labs, maker of the Canopy smart hydroponic garden — that fans out into typed, chained deliverables: a plan feeds an architecture diagram, which feeds a build, which feeds the invoice. Each worker sees its teammates' real output; the whole delivery is judged as one body of work.
Does a skill's value carry downstream?
Two arms of the identical ticket, differing by one worker — a skill-taught vs an untaught (@bare) plan or build. Everything after the pivot measures whether that one change propagates. A positive delta means the skill made the rest of the project better, not just its own stage.
mean chain delta · n=3 pairs · 1 skill-helped, 1 skill-hurt
mean chain delta · n=9 pairs · 6 skill-helped, 1 skill-hurt
The plan pivot lifts the whole chain (a better PRD compounds — twice it rescued a later stage from 0 to a full pass); the build pivot is near-flat (a better builder mostly just builds). Directional at this n, with run-to-run variance visible in the tail.
| pivot | ticket | skill arm vs bare | chain Δ | downstream stage deltas |
|---|---|---|---|---|
| build | launch | skill vs bare | 0.00 | billing 0.00 |
| plan | launch | skill vs bare | +0.05 | chart 0.00build 0.00billing 0.00 |
| plan | expansion | skill vs bare | +0.38 | creative +1.00billing 0.00 |
| build | expansion | skill vs bare | -0.20 | billing 0.00 |
| build | launch | skill vs bare | +0.23 | billing 0.00 |
| plan | quote | skill vs bare | 0.00 | billing 0.00 |
| plan | launch | skill vs bare | +0.53 | build +1.00billing 0.00 |
| plan | expansion | skill vs bare | +0.03 | build 0.00billing -0.75 |
| plan | quote | skill vs bare | 0.00 | billing 0.00 |
| plan | launch | skill vs bare | +0.23 | chart 0.00build 0.00billing 0.00 |
| plan | quote | skill vs bare | -0.38 | billing -0.75 |
| plan | expansion | skill vs bare | +0.07 | creative 0.00build 0.00billing 0.00 |
Driver stacks across the same work
Same tickets, same stage workers (rotation held constant) — only the driving model + harness changes. This measures the stack, not the model in isolation: PI-over-OpenRouter for glm and MiMo, Claude Code over ACP for Sonnet.
| template | mimo-v2.5 | glm-5.2 | claude-sonnet-5 |
|---|---|---|---|
| audit | 77%n2 | 67%n5 | 80%n2 |
| expansion | 74%n2 | 67%n5 | 70%n2 |
| incident | 95%n2 | 78%n5 | 93%n1 |
| launch | 81%n3 | 75%n3 | 90%n2 |
| quote | 87%n2 | 69%n5 | 63%n2 |
| status | 36%n2 | 72%n5 | 36%n2 |
Note the status row: both frontier stacks drop sharply on short comms tickets where glm holds steady — a harness-fit effect (heavy harnesses over-engineer trivial work), not a model-quality ranking.
Chains
Every run is a real deliverable chain. Click one to see its stages, workers, and the deterministic coherence checks against the world-file.