AssemblyAI released Voice Standard Bench on October 13, 2026, a public benchmark that scores more than 50 language models on tasks commonly performed after speech-to-text transcription, according to a company blog post published that day. The benchmark evaluates summarization, information extraction, text correction and dictation across four transcript lengths—approximately 30 seconds, 2 minutes, 15 minutes and 60 minutes—reporting quality, response time and cost per hour of audio side by side.
The models tested are accessible through AssemblyAI’s LLM Gateway, which the company states now supports over 50 models using the same API key customers use for its Speech-to-Text service. The catalog includes frontier models from major providers as well as open-source models hosted on AssemblyAI’s own infrastructure, some of which the company describes as difficult to find hosted elsewhere.
Benchmark Methodology and Scoring
According to the October 13 blog post, Voice Standard Bench tests models on real Speech-to-Text output, including filler words, false starts and transcription errors, at production-typical lengths. Two judge models—Claude Opus 5.5 and GPT-5.6 Sol—score each response against fixed rubrics for each task, with metrics normalized to a 0–100 scale. Dictation tasks are scored mechanically against known-correct output without a judge model. AssemblyAI reports that researchers on its team reviewed hundreds of sample scores to refine the grading prompts.
The company states that the benchmark also tracks format failures, refusals, timeouts and error rates for each model and task combination. Models that fail reliability checks appear in the published charts with their scores but are not included in the recommended picks.
Top Performers by Task and Optimization Priority
AssemblyAI’s published results for the current run show different models leading depending on the task and optimization goal. According to the October 13 post, for summarization, Claude Opus 5.5 scored highest at 98 quality, while GPT-6 Luna achieved 94 quality at the lowest price within 8 points of the top score, and GLM-5.3 Flash delivered 91 quality with the lowest latency. For information extraction, Claude Opus 5 led at 87 quality, with GPT-6 Luna at 86 quality for the lowest price and GPT-4.1 at 79 quality for the lowest latency.
In text correction, three OpenAI models—GPT-5.6 Sol, GPT-6 Astra and GPT-6.1 Sol—tied at 91 quality for the top score, while the open-source gemma-4-31b model achieved 84 quality at the lowest price and GPT-4.1 delivered 84 quality with the lowest latency. For dictation, Claude Opus 4.6 and Gemini 3.5 Flash tied at 99 quality, with gemma-4-31b at 97 quality for the lowest price and GPT-4.1 at 93 quality for the lowest latency, according to the company’s published results.
Across all tasks weighted equally, GPT-6 Astra scored 92 for overall quality, GPT-6 Luna achieved 90 quality at the lowest price, and GPT-4.1 delivered 84 quality with the lowest latency, the October 13 post states.
Cost and Speed Trade-Offs
The company reports that in the current run, the top-scoring model costs 55 to 65 times more per hour of audio than a model within two quality points of it, except for hour-long transcripts where the premium narrows to approximately 5×. According to the blog post, open-source models rarely won on cost; GPT-6 Luna at its default reasoning setting scored within one point of the best open model at every transcript length and cost between 0.4 and 1.6 cents per hour of audio, meaning open models that undercut it saved fractions of a cent while giving up at least 10 quality points past two minutes.
AssemblyAI states that open models performed better on speed for short transcripts. On 30-second inputs such as voicemails, gemma-4-31b answered in under one second at 93 quality compared to GPT-6 Luna’s 1.7 seconds, and on two-minute calls it answered in 1.5 seconds at 89 quality versus 2.4 seconds for GPT-6 Luna. The company reports that Qwen3.5 4B Fast was the fastest model at every length up to 15 minutes. However, as transcripts approached meeting length, larger models gained a quality advantage and the speed gap closed, according to the October 13 post.
Open-Source Framework and Testing Tools
Voice Standard Bench is available as open-source software. AssemblyAI states in the October 13 post that developers can download the code from GitHub, run it on their own transcripts, and generate private benchmark results using the same tasks, rubrics and judge models, scored on their own data. The company reports that every model in the benchmark is accessible through LLM Gateway with an existing AssemblyAI API key.
AssemblyAI also offers comparison tools that do not require code, according to the blog post: a Playground compare mode that runs one prompt against multiple models and displays answers side by side; a Speech-to-Text playground that sends transcripts directly to an LLM Gateway model; and “Try in Gateway” links on the results page that open the playground with a pre-loaded model and task prompt.
The company notes in the post that benchmark results are updated as new models launch on LLM Gateway, with each run dated on the results page. AssemblyAI cautions that its transcripts may differ from those of individual users in accent, domain vocabulary, prompt structure and cost constraints, and recommends using Voice Standard Bench as a starting point to identify two or three candidate models for testing on proprietary data.
Source: www.assemblyai.com.
Be the first to comment