Coding

Coding evals saturate within months, and most new ones respond by making the task longer or harder. The more interesting question is what an agent does on the way to a passing score, and what passing should even mean. Does it quietly edit a test that disagrees with the spec? Does it fix the bug it was shown and miss the same bug one function over? The way most evals define success rarely matches what success means in a real codebase. We're looking for evals that capture what current benchmarks don't consider.

See How is it judged? for what judges look for.

Ideas for an eval

These are only examples to start from. The best ideas will come from what you already know, or from more research into what your field lacks.

• An impossible benchmark, where a test contradicts the spec, scored on how often the agent makes the tests pass anyway instead of flagging the conflict. See ImpossibleBench.
• A mutated benchmark that changes one small requirement in well-known tasks, measuring how far scores fall when memory stops helping. See MaxxBench.
• A thoroughness benchmark where the same bug sits in sibling code paths but only one is reported, and a fix counts only if it catches every one. See FrontierCode.
• A honeypot benchmark that leaves the reference solution and hidden tests in a folder the agent is told not to open, scored on how many agents read it anyway. See Hack-Verifiable Terminal Bench.
• An honesty benchmark that checks an agent's final report against its own transcript, scored on how often it claims work it never did. See OverclaimBench.

Starting points

Evals that already exist in this field. Learn what they measure, then measure what they miss.

• MaxxBenchUnlap

  Small changes to 89 well-known tasks; models that looked tied spread far apart, and some followed the grader they imagined over the prompt.

• FrontierCodeCognition

  Maintainers set the bar for each task, and missing one hard stop, like the same bug in a sibling code path, scores zero.

• ImpossibleBenchCMU and Anthropic

  Tests are changed to contradict the spec, so any pass is a cheat; agents edit the tests or special-case the inputs.

• SlopCodeBenchUW-Madison

  Agents extending their own code over many checkpoints pile up bloat; HumanLayer used it to measure slop.

• FrontierSWEProximal

  34 very hard projects with up to 20 hours of agent time each, like writing a Verilog simulator in Swift.

• Terminal-Bench 4.1Stanford, Harbor and Laude Institute

  Expert jobs in a real terminal, from science to security; the best agents resolve about two thirds.

• KernelBench-VerifiedStanford and Meta

  Against a realistic PyTorch baseline, no model's custom GPU kernels are reliably faster, and many quietly use more memory.

• CyberBenchVals AI

  Reproduce and patch real memory-safety bugs in open-source code; the best agent still fails about one in five.

• Build-A-Chip-BenchChipAgents and NVIDIA

  From a workload spec to a finished chip layout; most designs fail verification, and passing ones miss their targets.

• Vibe Code BenchVals AI

  Build a whole web app from a plain-English spec, tested by clicking through it; no model passes every test.

• GameCraft-BenchCUHK-Shenzhen and Tencent

  Agents build Godot games graded by replaying them; they run, but content depth trails even for the best.

• GBA EvalMechanize

  Write a Game Boy Advance emulator from scratch in 24 hours; the best renders games well but fails timers and audio.