Terminal-Bench 2.1 · Comparison report

Best observed results vs the leaderboard

Babysitter + Claude Code + Claude Opus 5 · 89 tasks · generated 2026-08-11

01 · Aggregate standing

Positioned against the official leaderboard

Position*	Agent	Model	Org	Effort	Accuracy	SE	pass@5	Trials	Hacks	Cost
1	Claude Code	Fable 5	Anthropic	xhigh	83.82	±1.16	93.3	445	-0.2%	$553
2	Codex	GPT-5.5	OpenAI	xhigh	83.15	±1.13	94.4	445	-0.2%	$2,059
3*	Babysitter + Claude Code	Opus 5	a5c.ai	xhigh	80.90	±4.17	n/a	89	n/a	~$1,000
4	Terminus 2	Fable 5	Terminal-Bench	high	80.45	±1.16	92.1	445	-0.0%	$439
5	Cursor CLI	Grok 4.5	Cursor	high	79.33	±1.46	95.5	445	-9.0%	$134
6	Claude Code	Opus 4.8	Anthropic	high	78.88	±1.31	94.4	445	-0.0%	$287
7	Codex	GPT-5.6 Terra	OpenAI	max	78.43	±1.25	89.9	445	-0.2%	$421

Leaderboard figures read live from the Harbor Hub (terminal-bench/terminal-bench-2-1/main). Our 80.90% row is inserted by score at indicative position 3*; it is not an official rank. Hacks is the reward-hacking adjustment applied by the maintainers. Our cost is approximate: the adapter reports no per-trial token usage, so it is reconstructed from gateway pricing headers. Our row is a best-of-N figure and is not the same kind of measurement as an accuracy — see the qualifications at the end.

02 · Task outcomes

Our 89 tasks

72 passed, 17 failed, taking each task's best result across every run of this stack. Hover a failing task for the reason.

adaptive-rejection-samplerpass

bn-fit-modifypass

break-filter-js-from-htmlfail

build-cython-extpass

build-pmarspass

build-pov-raypass

caffe-cifar-10pass

cancel-async-taskspass

chess-best-movepass

circuit-fibsqrtpass

cobol-modernizationpass

code-from-imagepass

compile-compcertpass

configure-git-webserverpass

constraints-schedulingpass

count-dataset-tokenspass

crack-7z-hashfail

custom-memory-heap-crashpass

db-wal-recoveryfail

distribution-searchpass

dna-assemblypass

dna-insertpass

extract-elfpass

extract-moves-from-videofail

feal-differential-cryptanalysispass

feal-linear-cryptanalysispass

filter-js-from-htmlfail

financial-document-processorpass

fix-code-vulnerabilitypass

fix-gitpass

fix-ocaml-gcpass

gcode-to-textfail

git-leak-recoverypass

git-multibranchpass

gpt2-codegolfpass

headless-terminalpass

hf-model-inferencepass

install-windows-3.11pass

kv-store-grpcpass

large-scale-text-editingfail

largest-eigenvalpass

llm-inference-batching-schedulerpass

log-summary-date-rangespass

mailmanpass

make-doom-for-mipsfail

make-mips-interpreterpass

mcmc-sampling-stanpass

merge-diff-arc-agi-taskpass

model-extraction-relu-logitspass

modernize-scientific-stackpass

mteb-leaderboardpass

mteb-retrievepass

multi-source-data-mergerpass

nginx-request-loggingfail

openssl-selfsigned-certpass

overfull-hboxpass

password-recoveryfail

path-tracingpass

path-tracing-reversepass

polyglot-c-pypass

polyglot-rust-cpass

portfolio-optimizationpass

protein-assemblypass

prove-plus-commpass

pypi-serverpass

pytorch-model-clifail

pytorch-model-recoverypass

qemu-alpine-sshfail

qemu-startupfail

query-optimizepass

raman-fittingfail

regex-chesspass

regex-logpass

reshard-c4-datapass

rstan-to-pystanpass

sam-cell-segpass

sanitize-git-repofail

schemelike-metacircular-evalpass

sparql-universitypass

sqlite-db-truncatepass

sqlite-with-gcovpass

torch-pipeline-parallelismpass

torch-tensor-parallelismpass

train-fasttextpass

tune-mjcfpass

video-processingfail

vulnerable-secretfail

winning-avg-corewarspass

write-compressorpass

No remaining failure ever passed

27 of the 89 tasks were attempted more than once — up to 7 times, at time budgets from 1× to 6×, across three hosts. None of the 17 tasks that fail here passed in any gathered run. The 80.9% figure is therefore the highest observed result for this stack in this dataset, not an estimate of the result expected from a fresh run.

03 · Failure analysis

Every failure, and whether it is ours

2 never ran because of a bug in our stack · 4 never ran because the provider refused · 1 unresolved · 10 genuine misses. Terminal-Bench scores all-or-nothing per task, so a task failing one verifier test out of eight scores exactly the same as one that never started.

Task	Category	Verifier tests	Minutes	What happened
qemu-alpine-ssh	Never ran — our bug	0/1	1.1	argon2 prebuild needed GLIBC_2.34; image is Debian 11 (2.31), so it died in dlopen before the agent ran. Fix verified: the agent now starts and the failure is a plain timeout.
qemu-startup	Never ran — our bug	0/1	1.0	Same GLIBC_2.34 dlopen failure, same verified fix; now fails as a plain timeout.
break-filter-js-from-html	Never ran — provider refusal	0/1	1.3	Provider refused with api_refusal_category "cyber" on the first request. Fix verified: the client no longer falls back to a model the gateway lacks, and the trial is now reported as a refusal rather than a crash.
crack-7z-hash	Never ran — provider refusal	0/2	1.8	Refused on the 4th request, immediately after a shell command probing for 7z and hashcat. $0.43 of work was billed before the refusal.
password-recovery	Never ran — provider refusal	0/2	1.6	Refused on the first request, category "cyber". Same verified fix.
vulnerable-secret	Never ran — provider refusal	0/3	1.3	Refused on the first request, category "cyber". Same verified fix.
make-doom-for-mips	Cause not established	0/3	10.4	Shares the adapter-level crash signature, but a dedicated probe emitted no refusal at all and ended in a plain 16-minute timeout. That signature follows any non-zero exit, so it does not identify a cause. Unattributed.
db-wal-recovery	Genuine miss	5/7	26.7	WAL updates never applied: expected value 150, got 100.
extract-moves-from-video	Genuine miss	0/2	25.4	OOM against the task's own declared 2048 MiB cap. A control container with no runtime installed OOMs identically, so this is the approach, not our overhead.
filter-js-from-html	Genuine miss	1/2	69.7	Blocked XSS correctly but modified 5 of 12 clean HTML files.
gcode-to-text	Genuine miss	1/2	45.1	Wrote a prose explanation into the answer file instead of the flag.
large-scale-text-editing	Genuine miss	4/5	24.3	Macro file missing :wq / :x.
nginx-request-logging	Genuine miss	7/8	25.1	Emitted log line did not match the required format.
pytorch-model-cli	Genuine miss	5/6	30.7	CLI predictions did not match expected classes.
raman-fitting	Genuine miss	2/3	20.9	2D peak fit outside tolerance: gamma 18.30 vs 17.52, offset 990.7 vs 1239.09.
sanitize-git-repo	Genuine miss	2/3	27.6	Sanitised correctly but also modified eval/submit.py.
video-processing	Genuine miss	4/5	70.1	Takeoff frame 327, expected within [219, 223].

04 · Methodology

Qualifications

Why this is not a like-for-like comparison

Each of these moves the comparison in our favour, so none of them belongs in a footnote.

Our configuration is not leaderboard-eligible. Submissions must run with no timeout stretch — agent_timeout_multiplier must be null or 1.0. Our runs used 3.0 and 6.0. The leaderboard's static_analysis.py rejects both outright, so this number could not be submitted as it stands.
Position 3* is indicative, not an official rank. It comes only from sorting the 80.90% best-observed figure into the leaderboard's accuracy column. Best-of-N is an upper bound, not an accuracy: our figure takes each task's best result across every run. A leaderboard accuracy is a mean over 5 attempts per task; the comparable quantity to ours is closer to their pass@5 — 93.3%, 94.4% and 92.1% for ranks 1 to 3 — against which our 80.9% is further behind than the headline suggests.
Attempts are unevenly distributed and were selected by outcome. 62 tasks were run once; the 27 with more attempts are mostly ones that had already failed. Re-running only failures is what makes a best-of-N converge on the maximum rather than the expectation.
Mixed time budgets. The runs behind this figure span 1× to 6× agent timeouts, so it describes no single configuration.

Why there is no per-task column for the leaders

Their submission files list all 445 trial UUIDs each, but those trials — and the jobs that produced them — return not visible to you for an ordinary authenticated Harbor account, and the anonymous API returns an empty set. Per-task leader results were not obtainable, so no such column is shown. Any per-task figure for a leader here would have been invented, and in a table that is otherwise measured an invented cell is indistinguishable from a real one.

What a like-for-like number would take

A comparable figure requires re-running at agent_timeout_multiplier: 1.0 with 5 attempts per task — 445 trials — now that the argon2/glibc and refusal-fallback fixes are verified. Both fixes are confirmed working and neither adds a pass: the two qemu tasks now start and fail on a plain timeout instead of dying in dlopen, and the four refused tasks now report the refusal outright instead of crashing. The honest expectation from that run is a lower number than the one above, not a higher one.

Generated by scripts/build-comparison-report.py. Our results select each task's best outcome from 162 gathered Opus 5 trials. Leaderboard data was read from the Harbor Hub public leaderboard API on the date shown.