Papers & Research

Hugging Face blog releases code-switching benchmark for speech agents

Original title:Can Voice Agents Handle Bilingual Customers? Benchmarking Frontier ASR on Code-Switched Speech

AI summary

The Hugging Face blog released a benchmark for speech agents handling code-switched speech. The dataset covers four language pairs — Spanish-English, French-English, Canadian French-English, and German-English — built around HR and IT service management scenarios. Seven ASR systems were evaluated using word error rate, semantic word error rate, and answer error rate: AssemblyAI Universal 3-Pro, Deepgram Nova 3 Multilang, ElevenLabs Scribe V2, Gemini 3 Flash, Mistral AI Voxtral Small 24B-2507, Nvidia Parakeet TDT 0.6b V3, and OpenAI Whisper Large V3 Turbo. Key findings: the cost of transcribing code-switched speech varies by language pair and model; ElevenLabs Scribe V2, Gemini 3 Flash, and AssemblyAI Universal 3-Pro performed best across all metrics. The dataset and test harness are released open-source via AU-Harness.

Key Takeaways

  • The Hugging Face blog released a benchmark for speech agents handling code-switched speech.
  • The dataset covers four language pairs — Spanish-English,French-English,Canadian…
  • Seven ASR systems were evaluated using word error rate,semantic word error rate,and answer error…
  • 6b V3,and OpenAI Whisper Large V3 Turbo.
  • Key findings: the cost of transcribing code-switched speech varies by language pair and model

Quick Facts

Published
Updated
Source
Hugging Face 社区博客(混合发现)
Category
Papers & Research
AI score
67
Read the original Read the full article at the original source
Why it matters

如果你在给多语言客户做语音Agent,这篇博客直接把主流ASR的code-switching能力测了一遍,ElevenLabs Scribe V2目前最强,还开源了数据集,拿来就能测自己的模型。

Related entities

Source & copyright

This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.

Original article · AIHOT entry

How to Cite

When citing this article, keep the title, date, and permalink, and credit Orange News as the source.

Orange News: Hugging Face blog releases code-switching benchmark for speech agents. June 10, 2026. https://new.nat6.net/en/news/cmq7289uo001rsle3dbj1unbi/

View all →