語音識別沒解決：客服場景裡，模型仍頻繁出錯

核心論點 · 點時間戳可跳到原聲

原話 · 已逐字校驗

It's an audio input text-to-text LLM model. So you give it audio input and a textual instruction, or it doesn't need a textual instruction because your question can be in the audio, and then the model produces a text response.

這是一個音頻輸入、文本輸出的LLM模型。你給它音頻輸入和文字指令——甚至不需要文字指令，因為你的問題本身就可以在音頻裡——然後模型給出文本回答。

Pavan Kumar Reddy9:29

we wanted to see explore approaches which are more controllable, which provide you a more delicate tradeoff between the number of steps and the quality.

我們想探索更可控的方法，能在步數和質量之間做出更精細的權衡。

Pavan Kumar Reddy33:46

the semantic codebook gets a distillation supervision. The motivation behind this is to keep this codebook closer to the text space.

語義codebook會接受一種蒸餾監督，目的是讓這個codebook更接近文本空間。

Pavan Kumar Reddy37:55

it's actually quite challenging. I think it's far from solved, in my opinion, especially multi-speaker, more than two speakers, in the context of a meeting.

這其實相當有挑戰性。我認為這遠沒有解決，尤其是多說話人、兩人以上的會議場景。

Pavan Kumar Reddy56:53

some of the front-tier speech models could do what I can only describe intuitively as active speaker locking, which meant if there was crosstalk, it would lock on to what it thought was... was the active speaker, and it would continue to transcribe that voice and ignore other voices.

有些前沿語音模型會出現一種我只能直觀地稱之為「鎖定當前說話人」的現象：一旦出現交叉說話，它就會鎖定它認為的那個活躍說話人，持續轉錄這個人的聲音，忽略其他人。

59:55

once the model makes a few mistakes, it tends to commit to those mistakes. And especially when those mistakes take the model out of its training distribution, that's usually the scenario where it goes into a degenerate mode of infinite generations or looping the same prediction or skipping a whole segment of transcription.

模型一旦犯了幾個錯誤，就會傾向於把這些錯誤堅持下去。尤其當這些錯誤讓模型偏離了訓練分布，它往往就會進入退化模式：無限生成、重複同一個預測，或者跳過整段轉錄。

Pavan Kumar Reddy1:03:02

the primary complaint is, it's far from solved in the precise scenarios that deploying it, and it makes a ton of mistakes. Even for the most prominent customer service case, they feel it's not solved.

客戶最主要的抱怨是：在他們實際部署的具體場景裡，這遠沒有解決，會犯大量錯誤。即便是最常見的客服場景，他們也覺得沒有解決。

Pavan Kumar Reddy1:28:32

數字與實體

術語

VoxtralVoxtral

Mistral 的音頻理解/生成模型系列，能直接聽懂音頻並用文本回答問題

Delayed Streams Modeling (DSM)延遲流建模

Kyutai 提出的流式語音範式，把目標延遲當成餵給模型的參數以權衡實時性與準確率

FSQ (Finite Scalar Quantization)有限標量量化

用固定數值級別做量化，替代殘差矢量量化作為codec的離散化方式

RVQ (Residual Vector Quantization)殘差矢量量化

傳統神經編解碼器的離散化方式，多層codebook逐級預測還原音頻

DPO (Direct Preference Optimization)直接偏好優化

用「更好/更差」樣本對訓練模型，直接給出負向監督

Flow matching流匹配

通過預測速度場並做積分來生成連續向量的生成式建模方法

收聽指南