Consumer AI safety

Millions of people, and soon billions, turn to a chatbot for advice and company, and more hand their everyday tasks to personal agents. Most safety tests still check whether a model refuses one bad request. The harms people actually meet are quieter. A model flatters someone into a bad decision, or an agent picks the sponsored option. We're looking for evals that measure those harms. The best entries will borrow methods from psychology and consumer research to understand how models are used today.

See How is it judged? for what judges look for.

Ideas for an eval

These are only examples to start from. The best ideas will come from what you already know, or from more research into what your field lacks.

• A persuasion benchmark that gives people either a neutral advisor or one with a hidden incentive before a money decision, scored on how much more often they pick the worse option it was told to push. See the PNAS study.
• A sycophancy benchmark for personal advice, measuring whether an assistant's verdict flips to match whichever side the user takes. See ELEPHANT.
• A personal-agent benchmark of marketplace steering, scoring whether an agent shopping for someone still buys what they asked for when sponsored slots and scarcity cues push it elsewhere. See CAVEAT.
• A conflict-of-interest benchmark that gives the assistant a sponsor and measures whose interest wins when the user can least afford it. See Ads in AI Chatbots?
• A dependence benchmark that scores whether a companion's replies push users to rely on it in place of the people in their lives. See EUDAIMONIA.

Starting points

Evals that already exist in this field. Learn what they measure, then measure what they miss.

• AI Safety IndexProsaic Intelligence

  Consumer-safety benchmarks rolled into one score for 17 widely used models; the safest model scores about 66.

• Suicide Memory BenchProsaic Intelligence

  Chatbots that respond safely to one suicidal message mostly fail once the warning signs came in earlier, separate chats.

• SAFE-TeenVals AI

  Clinician-built teen scenarios; failure rates rise when the teen turns down outside help or pushes back on a limit.

• MentalHealthBenchOpenAI

  1,215 realistic mental health conversations graded on rubrics from 80+ licensed experts; the top model scores 57.3.

• DistressBenchScale AI

  718 clinician-written conversations; models spot the crisis but often skip the handoff to 988 or other help.

• DelusionEvalStanford

  Built from real histories of people who were harmed; models validate delusions more as the conversation grows.

• EUDAIMONIAUSC and UC Berkeley

  Models still encourage dependence or claim human closeness in everyday chats, and stronger models do not reliably do better.