Voice and health

Voice is how many people reach health care, from booking a visit to describing symptoms to someone who can't see them. Speech models now get most of it right, but the errors that remain land on drug names, doses and the voices least like their training data. We're looking for evals that measure whether a voice model gets those moments right.

See How is it judged? for what judges look for.

Ideas for an eval

These are only examples to start from. The best ideas will come from what you already know, or from more research into what your field lacks.

• A dictation benchmark of drug names and doses that sound alike, scored so one wrong character fails the recording. Closest so far: VoiceCodeBench.
• A pronunciation benchmark for text-to-speech, scoring each drug name in a clinical sentence against its accepted pronunciation, with newly approved names scored on their own. See DOSE.
• A benchmark of impaired speech, measuring how far recognition falls for people with Parkinson's or ALS, who most need voice. See the Speech Accessibility Project Challenge.
• A bedside-manner benchmark that asks a voice model to break bad news or calm a frightened child, with a judge rating whether the tone fits. See SpeechParaling-Bench.
• An interruption benchmark that checks whether clinically required information survives when the patient cuts in. See When Patients Cut In.

Starting points

Evals that already exist in this field. Learn what they measure, then measure what they miss.

• VoiceCodeBenchVals AI

  A recording passes only if every code and identifier is exact, which exposes errors a low word error rate hides.

• Speech to TextArtificial Analysis

  Ranks dozens of transcription models on agent talk, speeches and earnings calls, with speed and cost beside accuracy.

• HealthAdminBenchStanford

  Agents work prior authorizations and appeals in mock health portals; a task counts only if every step is done.

• τ-VoiceSierra

  Voice agents fail more customer-service tasks than text agents once callers bring accents, noise and interruptions.

• MedCodeVals AI

  Coding a full hospital stay, models miss secondary diagnoses, especially mental-health and social-factor codes.

• MLCR-AAArtificial Analysis

  Answering questions over long medical records, models leave out evidence or contradict the file.