Measured Results
Verdict Benchmark Hub
Every number below comes from a saved results file and links to the full benchmark page.
These results were measured by Verdict on its own hardware. Results from sample tasks or test prompts are labelled on each card, and each card shows the date its numbers were recorded and what was held constant.
BigCodeBench: Reasoning × Harness
ToolTerse v5 with reasoning off and on, direct and through Verdict Code. Verified final scores, recorded tokens and separate historical competitor references.
As of 2026-10-28. Held constant: Same 1,140 Complete tasks and model ID on the M5 Max; one replicate. Prompts, output caps and concurrency differ across routes.
Agent Showdown
Verdict Code against Claude Code, Codex, Crush, opencode, aider and Pi, graded by hidden tests. Includes the lane Verdict Code loses.
As of 2026-10-28. Held constant: Same four tasks, limits and model per lane for every agent; 2 runs per task in the cloud, 1 on the local model. No pair is statistically significant.
Project Benchmark (10 Tasks)
Ten focused coding tasks scored by unit assertions, run on local and cloud models. The free local Tool Terse v5 model matches the cloud scores.
As of 2026-10-15. Held constant: Same 10 tasks and 33 unit assertions for every model; one run per model per task.
Blueprint Planning
Verdict's Blueprint planner turns a specification into tasks and dependencies without calling a model. Native coding agents planned the same two specifications.
As of 2026-10-16. Held constant: Same two specifications and scoring rubric for every arm; one run each. Agents used their own native models, so this is not a same-model comparison.
Hardware Concurrency: ON vs OFF
Model-aware parallel slots (ON) against serialized requests (OFF), from one to four concurrent requests, on six resident local models.
As of 2026-09-10. Held constant: Same 6 local models, prompts and machine (Apple M4 Max); only the slot mode changes.
Context Headroom: 66k vs 262k
Four prediction slots at 66k versus 262k total loaded context, with prompts of about 18,000 tokens.
As of 2026-09-10. Held constant: Same model (35B Toolterse v5), prompts and machine; only the loaded context changes.
Opinion & Decision Engine
Three local models each answer every task and Verdict's Decision step picks one. The pick matches the strongest single model; it does not beat it.
As of 2026-10-05. Held constant: Same 148 BigCodeBench-Hard tasks and prompts; one answer per local model.
Try Verdict on Your Own Hardware
Download Verdict to run local open-weights models on your own Mac.