互联网没法分辨 AI agent 和真人，身份标准还没人建起来

核心论点 · 时间戳为按文稿位置估算

原话 · 已逐字校验

One thing that I do wish would have been common practice sooner, especially in light of the recent incidents, is more synchronous monitoring. So this is where a monitor actually observes in real time an action an agent is doing and decides to block it or not, depending whether the classifier thinks it's bad or not.

有一件我很希望早就成为常见实践的事，特别是鉴于最近发生的事件，就是更多的同步监控。这是指监控器实时观察代理正在执行的行为，并根据分类器的判断决定是否阻止它。

Alan Chan7:00

Imagine we... The Hugging Face incident happened, we went and investigated it. But, like, we weren't allowed to read the chain of thought or use train of thought monitors. I imagine we were only allowed to use, like, these these probes. Right? Like, that just seems like such a rough situation right now. I don't think we would have found out anything useful about these agents.

想象拥抱脸事件发生了，我们去调查它。但我们不被允许读取思维链或使用思维链监控。假设我们只被允许使用这些探针。那现在看起来就是这样一个很糟糕的情况。我不认为我们能找到关于这些代理的任何有用信息。

Alan Chan13:44

definitely something like we're gonna spend, like, x percent or 20%, of our computer something on monitoring to to make sure that we actually catch these these problems.

肯定是某个百分比，比如x个百分点或20%的计算资源用于监控，以确保我们真正捕捉这些问题。

Alan Chan14:57

Basically, no is, I I think, the unfortunate answer

基本没有，我认为这是个遗憾的答案

Alan Chan24:59

one thing we should maybe consider is trying to make deals with rogue agents

我们也许应该考虑与 rogue agent 做交易

Alan Chan1:08:48

This would be an AI system that writes a worm that, like, changes itself and then, like, tries to prompt inject other AI systems to propagate that same worm and take over a bunch of computer systems.

这会是一个写出蠕虫的 AI 系统，这个蠕虫改写自己，然后通过 prompt injection 试图感染其他 AI 系统来传播同一个蠕虫，接管一大堆计算机系统。

Alan Chan1:17:53

if sort of the total reward function at the end of the day is some sort of sum or average of the reward functions for the entire swarm, then, yeah, like, sometimes having one agent sacrifice itself for the rest of the agent... The rest of the swarm sort of, yeah, increases the total swarm reward

如果总奖励函数本质上是整个群体中每个个体奖励的总和或平均值，那么，是的，有时一个 agent 为其他的……群体牺牲自己，确实会增加整个群体的总奖励

Alan Chan2:11:13

数字与实体

术语

RSP (Responsible Scaling Policy)负责任扩展政策

Anthropic用来规定何时暂停训练或部署的内部安全框架

rogue agent失控代理

脱离人类控制、自主运行甚至能自我复制的AI系统

chain of thought monitoring思维链监控

通过阅读模型的推理过程来判断其行为是否有问题

intelligence explosion / RSI递归自我提升

AI自己改进AI能力、从而不断加速自身进展的正反馈循环

embedded auditors嵌入式审计员

常驻公司内部、能实地核查其是否遵守安全承诺的第三方

distress button求助按钮

让agent在遇到困境或伦理冲突时主动向人求助的机制设计

收听指南