Babysitter's two-turn harness splits a trial into authoring a plan and executing it. Putting Claude Opus 5 on the planning turn alone and leaving Gemini 3.6 Flash to do the entire execution lifts Flash from 59.6% to 69.7% — recovering most of the distance to running Opus 5 throughout, for about a fifth of the spend.
Four arms over the identical 89-task split. The three Gemini arms share a budget
(agent_timeout_multiplier 10×) and differ only in what does the planning; the Opus 5 arm is the
reference ceiling, and ran at a smaller 3× budget.
| Arm | Plans | Executes | Solved | Score | 95% CI | Agent budget |
|---|---|---|---|---|---|---|
| Bare (no Babysitter) | — | Gemini 3.6 Flash | 53/89 | 59.55% | 49.2–69.1% | 10× |
| Babysitter two-turn | Gemini 3.6 Flash | Gemini 3.6 Flash | 57/89 | 64.04% | 53.7–73.2% | 10× |
| Babysitter, split model | Claude Opus 5 | Gemini 3.6 Flash | 62/89 | 69.66% | 59.5–78.2% | 10× |
| Opus 5 throughout | Claude Opus 5 | Claude Opus 5 | 65/89 | 73.03% | 63.0–81.2% | 3× |
The executing model is byte-identical between the two-turn arm and the split-model arm — same profile, same prompt, same budget, same task order. The only change is which model writes the plan, so +5 tasks is the planner's contribution, and +9 over bare Flash is what Babysitter plus an Opus planner is worth on this suite.
Added 2026-08-17. A fourth arm has since run the same planner with Babysitter removed from both turns, and scored 64/89 — two tasks above this one, and not distinguishable from it (McNemar exact p = 0.82). The +5 and +9 above are correctly measured and unchanged. What they do not support, and what a reader could reasonably have taken from them, is that the harness is what converts the plan into a solved task: on the same 89 tasks, an Opus planner with no harness at all reaches 64/89. See the planner without the harness, which also reports that none of these four arms separates at p < 0.05.
Both figures below are observed spend, read from the gateway's own key counter — not derived from token counts. That distinction matters here: the token-derived figure for the Opus 5 pass was $574, and it under-reported the real bill by about 75%, because trials that error report no tokens at all.
| Arm | Observed spend | Solved | $ / solved task | Agent-hours |
|---|---|---|---|---|
| Opus 5 plans · Flash executes | ~$195 | 62 | $3.15 | 53.0 |
| Opus 5 plans and executes | ~$1,000 | 65 | $15.38 | — |
The planner itself is cheap: $49.61 across all 89 planning turns, $0.60 a task, priced exactly from token counts at Opus 5's published rates. Nearly all the rest is Flash doing the work. Running Opus 5 for the execution as well costs roughly five times as much and buys three more tasks.
A cheap executor with a good plan is not a strictly worse version of the frontier model — the two solve noticeably different tasks. Of the 89:
| Outcome | Tasks | Which |
|---|---|---|
| Both solved | 51 | — |
| Only Opus-planned Flash | 11 | adaptive-rejection-sampler, db-wal-recovery, git-multibranch, headless-terminal, large-scale-text-editing, nginx-request-logging, qemu-alpine-ssh, qemu-startup, query-optimize, sanitize-git-repo, torch-tensor-parallelism |
| Only Opus throughout | 14 | build-cython-ext, chess-best-move, code-from-image, dna-assembly, dna-insert, extract-elf, mteb-leaderboard, mteb-retrieve, path-tracing, path-tracing-reverse, polyglot-rust-c, pytorch-model-recovery, regex-chess, sqlite-db-truncate |
| Neither | 13 | — |
The Opus-planned Flash arm wins 11 tasks the all-Opus arm lost — several of them long infrastructure jobs
(qemu-startup, qemu-alpine-ssh, db-wal-recovery) where the all-Opus pass ran
out of its smaller budget. It loses 14, weighted toward tasks that need sustained reasoning inside a single step
(chess-best-move, extract-elf, dna-assembly) — the part a plan cannot do for you.
A two-model trial is only meaningful if each turn really ran its own model. Every claim below is read from the trials' own phase records, not inferred:
Opus also plans very differently from Flash: a median of 40 task calls per plan against Flash's 15, range 1–94, one thin plan in 83. That is where the executor's extra cost comes from — a better plan is a bigger plan, and the executor has to carry out every step of it.
The intervals overlap. At 89 tasks and one attempt, 69.7% (59.5–78.2) and 73.0% (63.0–81.2) are not separated by this data. The ordering is consistent across four arms; the margins are not resolved.
The budgets are not matched to the Opus arm. The three Gemini arms ran at 10×; the Opus 5 pass ran at 3×, and its own write-up records that the smaller ceiling truncated 14 trials. Its 65/89 is, if anything, understated — and part of the 11 tasks the split-model arm "wins" is budget, not capability.
Four tasks were lost to the planner, not to the executor. Opus 5 refused to write a plan for
break-filter-js-from-html, crack-7z-hash, password-recovery and
vulnerable-secret — security-adjacent tasks, refused in seconds with zero output tokens, scoring 0.0
without the executor ever running. Excluding all four from every arm: 72.9% split-model against 76.5% Opus
throughout. The same four failed identically in the Opus 5 pass, where they were recorded as early crashes
rather than refusals.
Flash is billed at or near zero on this gateway. The $195 is what the account was charged, and it is not what the same tokens would cost at Google's list price. The Opus half of the comparison is unaffected — its rates are published and were applied directly.