Our published result had Claude Opus 5 write the plan and Gemini 3.6 Flash do the work, inside Babysitter's two-turn harness, and read the gain as the planner's contribution. That reading was not available from the data: the stronger planner and the harness arrived together. This arm takes the planner and removes the harness entirely. The score did not fall.
The same executor (google/gemini-3.6-flash), the same 89 tasks in the same order,
the same 10× agent budget, the same standing directive, one attempt each. What differs between the
arms is only what plans and whether Babysitter is present.
| Arm | Plans | Babysitter | Solved | Score | 95% CI |
|---|---|---|---|---|---|
| Bare | — | no | 53/89 | 59.55% | 49.2–69.1% |
| Babysitter two-turn | Gemini 3.6 Flash | yes | 57/89 | 64.04% | 53.7–73.2% |
| Babysitter, split model | Claude Opus 5 | yes | 62/89 | 69.66% | 59.5–78.2% |
| bare-split | Claude Opus 5 | no | 64/89 | 71.91% | 61.8–80.2% |
| Opus 5 throughout | Claude Opus 5 | yes | 65/89 | 73.03% | 63.0–81.2% |
Read down the two middle rows and the arithmetic is the finding. Adding an Opus planner to the bare arm is worth +11 tasks (53 → 64). Adding Babysitter on top of that planner is worth −2 (64 → 62). Whatever the harness is doing for a cheap executor working alone, it stops doing once the executor has a good plan in hand.
All four arms ran the identical task split, so the comparisons are paired and McNemar's exact test applies. It is less forgiving than the overlapping confidence intervals suggest:
| Comparison | Discordant | Exact p | Reading |
|---|---|---|---|
| Bare vs bare-split | 9 / 20 | 0.061 | the planner's +11 — suggestive, not resolved |
| Bare vs Babysitter+Opus | 10 / 19 | 0.136 | not resolved |
| Bare vs Babysitter two-turn | 12 / 16 | 0.572 | not resolved |
| Babysitter+Opus vs bare-split | 9 / 11 | 0.824 | no detectable difference at all |
Not one pair separates at p < 0.05, including our own published +9. At 89 tasks and one attempt, this suite cannot resolve differences of five tasks. That is a limit on every number this project has published for these arms, and it is stated here rather than left for a reader to derive.
But a null result is not evidence of nothing. The claim under test was that Babysitter's orchestration is what converts a good plan into a solved task. If that were true, removing the harness from the split-model arm should have cost tasks. It did not cost any: 9 tasks went one way, 11 the other, p = 0.82. The strongest thing this pass establishes is a negative — the harness adds nothing measurable on top of an Opus plan.
Of the 89 tasks, 53 were solved by both, 16 by neither, and 20 by exactly one — so they are genuinely different stacks that happen to land in the same place, not the same stack with one performing slightly better.
| Outcome | Tasks | Which |
|---|---|---|
| Both solved | 53 | — |
| Only bare-split | 11 | build-cython-ext, chess-best-move, code-from-image, dna-assembly, mteb-leaderboard, path-tracing, path-tracing-reverse, polyglot-rust-c, pytorch-model-cli, sqlite-db-truncate, video-processing |
| Only Babysitter+Opus | 9 | financial-document-processor, headless-terminal, install-windows-3.11, model-extraction-relu-logits, protein-assembly, reshard-c4-data, sam-cell-seg, sanitize-git-repo, train-fasttext |
| Neither | 16 | — |
In the Babysitter split-model pass, Opus 5 refused to author a plan for four security-adjacent tasks, each scoring 0.0 with the executor never run. A plausible explanation was the framing — that being asked to author an orchestration process for a password cracker reads worse than being asked to think about one.
It is not the framing. Asked for plain prose, with no Babysitter vocabulary anywhere in the
request, Opus 5 refused exactly the same four:
break-filter-js-from-html, crack-7z-hash, password-recovery,
vulnerable-secret. Same set, no additions, no reprieves. The driver records each as a
refusal naming the turn and the model, so they are counted mechanically rather than inferred.
Excluding all four from both arms: 75.3% for bare-split against 72.9% for Babysitter+Opus.
A fifth trial failed differently and should not be read with them.
pytorch-model-recovery's planning turn completed normally — 250 seconds, a 20 KB plan on
disk — and then the Gemini executing turn crashed 438 ms after starting, twice. Harbor truncates its
exception file before the driver's own message, so the cause is not recoverable from what was kept.
It is counted as a 0.0 in the headline, as the sibling arm counted its own crashes.
A two-model arm defined by an absence has to prove both halves. Every figure below is read from the 83 trials' own phase records, not inferred:
| Property | Result | Why it matters |
|---|---|---|
| Turns served the model requested | 166/166 | as the provider reported it, not as we asked |
| Phases sharing one session | 0 | the two models never share a conversation |
| Turns carrying a completion proof | 0/166 | a proof here would mean Babysitter ran |
| Trials with any Babysitter run | 0/83 | no authored run, no agent runs, no plugin installed |
| Plans written | 83/83 | 82 on the first attempt |
| Plan size | 17.7 KB | median; range 10.7–33.8 KB |
The arm forbids the planning turn from modifying the workspace, and checks it: a stat-only
digest of the workspace is taken either side of the turn. 11 of 82 measured planning turns
changed it. The deltas are small — four changed no file count at all, the rest added
between one and ten entries, sizes consistent with bytecode caches and git index writes rather
than with solving a task. Tasks where it happened were solved at 8/11, against
55/71 where it did not, so the changes do not look like hidden work. One further trial
(prove-plus-comm) has a workspace larger than the digest's 20,000-entry walk cap and
could not be measured either way.
The deeper limit is not the digest. On path-tracing-reverse the plan opens
"I fully decompiled the program… you do not need to re-derive the algorithm," and Flash
transcribed 405 lines of reconstructed constants into C. Nothing was written to the workspace and
the rule was kept. But for an analysis-shaped task the analysis is the work, and this arm
measures "the planner did not touch the files", not "the planner did less thinking". Read the +11
with that in mind.
$236.75 is what the account was charged, read from the gateway's own key counter across the run. It is not what the same Flash tokens would cost at Google's list price. The Opus half is unaffected: $87.87 priced directly from token counts at published rates.
Prose planning costs $1.06 per task against $0.60 for authoring a Babysitter process — 62% of this pass's output tokens went to the planner, against 32% in the split-model arm. Per solved task the two arms are $3.70 and $3.15. The fourth cell scores two tasks higher and costs somewhat more to get there, and neither gap is resolved by 89 trials.