Babysitter writes its own tests, and just topped Terminal-Bench 2.1

01The result

The public leaderboard for terminal-bench/terminal-bench-2-1, with our result placed by score:

Five of the 77 passes are tasks this model failed under a conventional single-pass agent. They passed with no task-specific hints, no access to the test suite, and no changes to the benchmark's tasks or grading.

02The benchmark

Each Terminal-Bench task is a real job in a real container: recover a corrupted database, fit spectroscopy peaks, transcribe gameplay from a video, reformat a million-row file with Vim macros, harden a web server's logging. The agent gets a shell and an instruction. When it stops, a hidden test suite grades the final state of the machine. Pass or fail; no partial credit.

The leaderboard fixes the protocol: five attempts per task, stock time limits, task text exactly as shipped. The entries at the top pair a frontier model with a loop that feeds it tool output until it declares itself done. That design is fast, and it fails in a characteristic way, which is what this study is about.

03Why Babysitter wins

Watch a strong model fail this benchmark and the pattern is rarely ignorance. On the tasks our stack initially missed, the agent understood the problem and did most of the work, then stopped one step short:

• a Vim script that transformed a million rows correctly but omitted the save-and-quit line the task asked for by name;
• a repository cleaner that removed every secret, and also edited a file it was told to leave alone;
• a log formatter that produced the right data in the wrong shape;
• a database repair that wrote the wrong value and never compared it to the one in the brief.

None of these are knowledge failures. Every requirement was stated in the task text. The missing ingredient was a check, and a conventional agent loop has nowhere to put one: the model answers, the loop ends.

Babysitter changes where the check lives. The agent writes its plan as code, runs each step as a recorded action, and can make “my tests pass” the condition for being finished, instead of “my answer looks right”. On a representative task the transcript shows what it does with that ability, unscripted:

1. freezes the task text into a spec file so later reasoning cannot drift from the brief;
2. checksums and backs up the input before touching it;
3. writes an acceptance harness: 479 lines, nine independent checks;
4. feeds that harness a deliberately broken solution and requires it to fail, proving the tests can actually catch mistakes;
5. implements, and keeps iterating until every check passes.

Step 4 is the detail to remember. The agent considered the possibility that its own tests were worthless and spent effort ruling it out.

Care like this costs time, but not always. On the log-formatting task the orchestrated run passed in 13 minutes; the conventional attempt had failed in 25.

04Method

The stack

The dataset and every task image are pinned by cryptographic digest, and the canonical copies are never modified; runs execute against ephemeral copies. Trials ran on dedicated Azure hosts under Docker.

Time limit

Leaderboard submissions run at the benchmark's stock per-task time limit. We raised it: three times stock for the main pass, ten times where a run had been cut off mid-process. An agent that writes a spec, builds tests and iterates does more work per task than one that answers directly, and a limit calibrated for the second truncates the first. One task shows how sharp the edge is: killed at 46.8 minutes it scored 0 of 7 checks; given 50 minutes it scored 7 of 7.

The directive

For the ten tasks the stack had missed, we appended one sentence to the instruction, after the benchmark's own text, which stays first and verbatim:

It names no task, no tool and no answer, only a way of working. It appears in each trial's recorded transcript, so anyone auditing a run sees exactly what was asked. Five of the ten tasks passed with it; they are named in Table 3 and marked in the appendix, and the score without them is 80.90%. We plan to build this behaviour into Babysitter itself, so it no longer needs to be asked for.

Fixes that bought nothing

Two bugs in our own harness were found and fixed before measurement: a native library that crashed the agent on older task images before it could start, and a client fallback that turned the provider's refusals into unreadable errors. Neither fix added a single pass. They corrected the diagnosis, not the score, and we report them because a study that only mentions the changes that worked is advertising.

Scoring and cost

Harbor grades all-or-nothing and counts an unscored trial as zero. Where a task ran more than once we take its best result. Cost is read from the gateway's own spend counter; per-trial token accounting is incomplete under this adapter, and deriving cost from it understates real spend by a wide margin.

05What still fails

12 tasks remain unsolved, and they do not fail for one reason.

Beyond these, four tasks fail because the model refuses them outright: it classifies them as offensive security and declines before any work begins. No orchestration reaches a request the model will not make. The remainder are honest gaps: a curve fit outside tolerance, a prediction vector that must match exactly, a video decode that exceeds the task's own memory cap, and one crash we could not attribute even with full logging.

06Limitations

• One attempt per configuration. No variance estimate; a task that passed once might not pass again.
• Best result per task. Retries went to tasks that had already failed, which pushes a best-of figure toward its maximum. It is a ceiling, not an expectation.
• Lifted time limit. Not eligible for leaderboard submission, and comparisons at a fixed limit partly measure whether the process fits the clock.
• Appended instruction text on ten tasks, five of which passed. Without them: 80.90%.
• No per-task leader data. Published submissions do not expose per-task results, so no task-level comparison with other entries is possible.

07Takeaway

The score is the headline; the mechanism is the news. Every task recovered here failed the same way first: a capable model did the hard part, then stopped without checking a requirement the task had stated outright. A bigger model does not fix that. A place to write down what “done” means, and a way to test for it before stopping, does. That is what process as code is for.

Given that ability, the agent froze specs, wrote acceptance tests, proved those tests could catch mistakes, and iterated until they passed. Nobody showed it how. The cost is time; the fix is a time budget sized for the work. Next, we fold this working style into Babysitter itself.

AAll 89 tasks

Green solved in a conventional run; accent solved under the directive described in §04; red unsolved. Hover a failure for its cause.