语音识别没解决：客服场景里，模型仍频繁出错

核心论点 · 点时间戳可跳到原声

原话 · 已逐字校验

It's an audio input text-to-text LLM model. So you give it audio input and a textual instruction, or it doesn't need a textual instruction because your question can be in the audio, and then the model produces a text response.

这是一个音频输入、文本输出的LLM模型。你给它音频输入和文字指令——甚至不需要文字指令，因为你的问题本身就可以在音频里——然后模型给出文本回答。

Pavan Kumar Reddy9:29

we wanted to see explore approaches which are more controllable, which provide you a more delicate tradeoff between the number of steps and the quality.

我们想探索更可控的方法，能在步数和质量之间做出更精细的权衡。

Pavan Kumar Reddy33:46

the semantic codebook gets a distillation supervision. The motivation behind this is to keep this codebook closer to the text space.

语义codebook会接受一种蒸馏监督，目的是让这个codebook更接近文本空间。

Pavan Kumar Reddy37:55

it's actually quite challenging. I think it's far from solved, in my opinion, especially multi-speaker, more than two speakers, in the context of a meeting.

这其实相当有挑战性。我认为这远没有解决，尤其是多说话人、两人以上的会议场景。

Pavan Kumar Reddy56:53

some of the front-tier speech models could do what I can only describe intuitively as active speaker locking, which meant if there was crosstalk, it would lock on to what it thought was... was the active speaker, and it would continue to transcribe that voice and ignore other voices.

有些前沿语音模型会出现一种我只能直观地称之为「锁定当前说话人」的现象：一旦出现交叉说话，它就会锁定它认为的那个活跃说话人，持续转录这个人的声音，忽略其他人。

59:55

once the model makes a few mistakes, it tends to commit to those mistakes. And especially when those mistakes take the model out of its training distribution, that's usually the scenario where it goes into a degenerate mode of infinite generations or looping the same prediction or skipping a whole segment of transcription.

模型一旦犯了几个错误，就会倾向于把这些错误坚持下去。尤其当这些错误让模型偏离了训练分布，它往往就会进入退化模式：无限生成、重复同一个预测，或者跳过整段转录。

Pavan Kumar Reddy1:03:02

the primary complaint is, it's far from solved in the precise scenarios that deploying it, and it makes a ton of mistakes. Even for the most prominent customer service case, they feel it's not solved.

客户最主要的抱怨是：在他们实际部署的具体场景里，这远没有解决，会犯大量错误。即便是最常见的客服场景，他们也觉得没有解决。

Pavan Kumar Reddy1:28:32

数字与实体

术语

VoxtralVoxtral

Mistral 的音频理解/生成模型系列，能直接听懂音频并用文本回答问题

Delayed Streams Modeling (DSM)延迟流建模

Kyutai 提出的流式语音范式，把目标延迟当成喂给模型的参数以权衡实时性与准确率

FSQ (Finite Scalar Quantization)有限标量量化

用固定数值级别做量化，替代残差矢量量化作为codec的离散化方式

RVQ (Residual Vector Quantization)残差矢量量化

传统神经编解码器的离散化方式，多层codebook逐级预测还原音频

DPO (Direct Preference Optimization)直接偏好优化

用「更好/更差」样本对训练模型，直接给出负向监督

Flow matching流匹配

通过预测速度场并做积分来生成连续向量的生成式建模方法

收听指南