Hugging Face blog releases code-switching benchmark for speech agents
Original title:Can Voice Agents Handle Bilingual Customers? Benchmarking Frontier ASR on Code-Switched Speech
AI summary
The Hugging Face blog released a benchmark for speech agents handling code-switched speech. The dataset covers four language pairs — Spanish-English, French-English, Canadian French-English, and German-English — built around HR and IT service management scenarios. Seven ASR systems were evaluated using word error rate, semantic word error rate, and answer error rate: AssemblyAI Universal 3-Pro, Deepgram Nova 3 Multilang, ElevenLabs Scribe V2, Gemini 3 Flash, Mistral AI Voxtral Small 24B-2507, Nvidia Parakeet TDT 0.6b V3, and OpenAI Whisper Large V3 Turbo. Key findings: the cost of transcribing code-switched speech varies by language pair and model; ElevenLabs Scribe V2, Gemini 3 Flash, and AssemblyAI Universal 3-Pro performed best across all metrics. The dataset and test harness are released open-source via AU-Harness.
Key Takeaways
- The Hugging Face blog released a benchmark for speech agents handling code-switched speech.
- The dataset covers four language pairs — Spanish-English,French-English,Canadian…
- Seven ASR systems were evaluated using word error rate,semantic word error rate,and answer error…
- 6b V3,and OpenAI Whisper Large V3 Turbo.
- Key findings: the cost of transcribing code-switched speech varies by language pair and model
Quick Facts
- Published
- Updated
- Source
- Hugging Face 社区博客(混合发现)
- Category
- Papers & Research
- AI score
- 67
如果你在给多语言客户做语音Agent,这篇博客直接把主流ASR的code-switching能力测了一遍,ElevenLabs Scribe V2目前最强,还开源了数据集,拿来就能测自己的模型。
Related entities
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: Hugging Face blog releases code-switching benchmark for speech agents. June 10, 2026. https://new.nat6.net/en/news/cmq7289uo001rsle3dbj1unbi/