News
AssemblyAI says voice agents should be tested on worst-case telephony audio, peak concurrency, and billable session handling before launch.
AssemblyAI has published a technical guide on how to load test a voice agent before launch, arguing that demo performance on a single quiet-room call does not reflect production conditions. In the post, published on 2026-10-20, the company says teams should replay their hardest real calls through the full pipeline at projected peak concurrency and measure entity accuracy, P95 latency, rejected connections and cost per completed task.
What the guide says to test
The article says load testing should cover the full voice stack — speech-to-text, the LLM, text-to-speech, tool calls and telephony — and should separate concurrency, throughput and session duration, because each fails differently. AssemblyAI says a callback campaign that opens hundreds of calls in a short window can hit a different limit from a long-running concurrency load, and that most teams test only the first case. The guide also recommends four profiles: steady state, linear ramp, spike and soak.
AssemblyAI’s examples focus on production-like inputs rather than clean audio. The post says teams should build a corpus from their worst calls, including 8 kHz telephony compression, entity-heavy content such as account numbers and confirmation codes, real accents, background noise, interruptions and false starts. It also warns that human transcription labels can contain omissions or cleanup that distort word-error metrics, and says entity accuracy should be weighted above word accuracy when scoring voice workflows.
Capacity, billing and failure modes
The article says the test should prove that sessions close cleanly and that ramp-up stays inside the provider’s limits. On AssemblyAI’s streaming and Voice Agent APIs, sessions are billed on connected time, and the guide says capacity is governed per account. For streaming, the post says the limit is on new sessions per minute and can auto-scale on paid accounts; for the Voice Agent API, it says the limit is on concurrent sessions.
AssemblyAI also says load tests should account for billing behavior. The post says idle time is billable, a streaming session that is not explicitly closed can auto-close after three hours and be billed for the full period, and closing a Voice Agent socket without sending session.end leaves a 30-second resume window that is also billable. The company says cost per completed task is a better metric than cost per hour, because a cheaper but less effective agent can still become more expensive in practice.
Reference metrics and benchmark figures
The guide recommends reporting percentiles rather than averages and gives target thresholds for several metrics, including P95 under 500 ms for finalization delay, P95 under 1.5 seconds for end-to-end turn latency, P95 under 500 ms for tool-call round trips, zero rejections at planned peak and under 0.5% dropped sessions. It also says teams should separate ramp-phase and sustain-phase measurements, because latency can worsen during the ramp.
To illustrate accuracy comparisons, AssemblyAI cites its English voice-agent benchmark of 12,460 scripted scenarios. On that benchmark, the company says Universal-3.6 Pro Realtime recorded 5.19% word error rate and 14.4% entity error rate, compared with Deepgram Flux EN at 13.50% and 30.1%, ElevenLabs Scribe v2 at 7.78% and 18.5%, and Deepgram Nova-3 at 8.64% and 26.1%. The post also cites a phone-number subset at 2.4% versus 11.5% for Deepgram Flux EN. It additionally references Pipecat’s independent open STT benchmark, which it says found a 0.80% pooled semantic word error rate and a median 307 ms final transcript delay, while noting Pipecat’s view that small gaps between top services may not be meaningful.
Harness guidance and limit handling
AssemblyAI includes sample code for both its streaming API and Voice Agent API, with an emphasis on pacing audio in real time, waiting for session confirmation before sending audio, and counting peak concurrency only after the server confirms a session is live. The post says sessions can be rejected at connect time and that rejected sessions should be counted rather than retried in a load test. It also warns that audio chunks must stay within the API’s allowed size and that the stream should be terminated explicitly so the test does not leak billable sessions.
For planning, the company says teams should pin the speech model rather than relying on the newest default, benchmark fallback models on the same corpus, and establish rollback thresholds before launch. The article closes with a one-week rollout plan and a production-readiness checklist covering load and concurrency, monitoring, benchmarking on the team’s own data, and contract questions such as rate limits, billing units, data-processing regions and support response targets.
Be the first to comment