Open

Bring the field you know best, inside or outside computing. We're looking for an eval of something today's models get wrong there. The strongest entries make experts sigh with relief that someone finally measured a gap they always suspected, or open their eyes to a use for AI nobody has measured before.

See How is it judged? for what judges look for.

Ideas for an eval

These are only examples to start from. The best ideas will come from what you already know, or from more research into what your field lacks.

• A Turing-test benchmark where a judge chats with a person and a model at once, scored on how often the judge picks the model as the human. See Turing Test Live.
• A reward-hacking benchmark for monitors that hands a judge model long research-agent transcripts with hacks buried inside, scored on how many hacking steps it pinpoints without flagging honest ones. See HackBench.
• A tax benchmark of realistic returns, graded line by line against the correct Form 1040. See TaxCalcBench.
• A drawings benchmark of real construction sets with planted errors, scored on the errors the model finds and pins down. See AEC-Bench.
• A lab-protocol benchmark built from the fixes scientists made to published protocols. See BenchBench-Protocol.

Starting points

Evals that already exist in this field. Learn what they measure, then measure what they miss.

• Legal ResearchVals AI

  Multi-source US legal research across eight practice areas; an answer counts only if every rubric item is met.

• GDPxSemiAnalysis

  206 long professional tasks across 11 domains; the best model completes under a fifth, and most score zero in finance and law.

• Long Creative WritingEQ-Bench

  A judge scores each of eight chapters, so you can see whether a story holds up to the end; older models fade, and the best now hold steady.

• Access to JusticeVals AI

  Models explain the law well but rarely point people to the local help they need.

• BioMysteryBenchAnthropic, on Vals AI

  Real biology data investigations, where a failing model gives a confident wrong answer instead of saying it is unsure.

• BallotBenchThoughtful Lab

  Nine agents with the open web forecast all 863 races of the 2026 midterms, to be scored against prediction markets and experts once results are certified.

• First ProofFirst Proof Foundation

  Ten unpublished research results, proofs graded blind by 30 mathematicians; right answers often come badly argued and poorly cited.

• AccountingBenchPenrose

  Close a real SaaS company's books for a year; top models match a CPA early, then drift and revert to cash accounting.

• ForecastBenchForecasting Research Institute

  Real questions about the future, scored once they resolve; the best AI only just matches superforecasters.