互聯網沒法分辨 AI agent 和真人，身份標準還沒人建起來

核心論點 · 時間戳為按文稿位置估算

原話 · 已逐字校驗

One thing that I do wish would have been common practice sooner, especially in light of the recent incidents, is more synchronous monitoring. So this is where a monitor actually observes in real time an action an agent is doing and decides to block it or not, depending whether the classifier thinks it's bad or not.

有一件我很希望早就成為常見實踐的事，特別是鑑於最近發生的事件，就是更多的同步監控。這是指監控器實時觀察代理正在執行的行為，並根據分類器的判斷決定是否阻止它。

Alan Chan7:00

Imagine we... The Hugging Face incident happened, we went and investigated it. But, like, we weren't allowed to read the chain of thought or use train of thought monitors. I imagine we were only allowed to use, like, these these probes. Right? Like, that just seems like such a rough situation right now. I don't think we would have found out anything useful about these agents.

想像擁抱臉事件發生了，我們去調查它。但我們不被允許讀取思維鏈或使用思維鏈監控。假設我們只被允許使用這些探針。那現在看起來就是這樣一個很糟糕的情況。我不認為我們能找到關於這些代理的任何有用信息。

Alan Chan13:44

definitely something like we're gonna spend, like, x percent or 20%, of our computer something on monitoring to to make sure that we actually catch these these problems.

肯定是某個百分比，比如x個百分點或20%的計算資源用於監控，以確保我們真正捕捉這些問題。

Alan Chan14:57

Basically, no is, I I think, the unfortunate answer

基本沒有，我認為這是個遺憾的答案

Alan Chan24:59

one thing we should maybe consider is trying to make deals with rogue agents

我們也許應該考慮與 rogue agent 做交易

Alan Chan1:08:48

This would be an AI system that writes a worm that, like, changes itself and then, like, tries to prompt inject other AI systems to propagate that same worm and take over a bunch of computer systems.

這會是一個寫出蠕蟲的 AI 系統，這個蠕蟲改寫自己，然後通過 prompt injection 試圖感染其他 AI 系統來傳播同一個蠕蟲，接管一大堆計算機系統。

Alan Chan1:17:53

if sort of the total reward function at the end of the day is some sort of sum or average of the reward functions for the entire swarm, then, yeah, like, sometimes having one agent sacrifice itself for the rest of the agent... The rest of the swarm sort of, yeah, increases the total swarm reward

如果總獎勵函數本質上是整個群體中每個個體獎勵的總和或平均值，那麼，是的，有時一個 agent 為其他的……群體犧牲自己，確實會增加整個群體的總獎勵

Alan Chan2:11:13

數字與實體

術語

RSP (Responsible Scaling Policy)負責任擴展政策

Anthropic用來規定何時暫停訓練或部署的內部安全框架

rogue agent失控代理

脫離人類控制、自主運行甚至能自我複製的AI系統

chain of thought monitoring思維鏈監控

通過閱讀模型的推理過程來判斷其行為是否有問題

intelligence explosion / RSI遞歸自我提升

AI自己改進AI能力、從而不斷加速自身進展的正反饋循環

embedded auditors嵌入式審計員

常駐公司內部、能實地核查其是否遵守安全承諾的第三方

distress button求助按鈕

讓agent在遇到困境或倫理衝突時主動向人求助的機制設計

收聽指南