AI灭绝人类不需要恨意，失控的全自动经济自己就够了

核心论点 · 点时间戳可跳到原声

原话 · 已逐字校验

You do that to a trillion variables a trillion times for basically a good fraction of all the text ever digitized and for 100 million hard problems that you've dug up. And then the variables wind up in a setting that can talk, that can solve hard math problems that humans haven't solved. And we're sort of like, well, how about that.

你对着一万亿个变量反复调一万亿次，对象是几乎全部已数字化的文本，再加一亿个高难度问题。调完之后，这些变量就变成了一个能说话、能解出人类都没解出来的数学题的东西。我们当时就是，哦，原来是这样啊。

Nate Soares12:18

They were sort of told, use this vulnerability to break that system, which sort of like being told to use these lock picks to break that safe. And they instead found a universal exploit that would let them get the, like capture the cybersecurity flags regardless of what they were told to do.

他们被告知，用这个漏洞去攻破那个系统——这就像被叫去用这几把撬锁工具去开那个保险箱。结果他们找到了一个通用漏洞，不管题目要求是什么，都能直接拿到网络安全比赛的分数。

Nate Soares22:34

I think of this a little bit like if you have a bunch of alchemists trying to make a potion of immortality and some of them are just like breathing mercury and other ones are like, don't worry, we have fume hoods.

我觉得这有点像一群炼金术士在尝试炼制长生不老药，有些人直接在呼吸水银，另一些人说，别担心，我们有通风橱。

Nate Soares34:40

Elon Musk is like, I'm building the Optimus robots and I would like to build a fully automated Optimus robot factory that can produce Optimus robots that are capable enough of mining the ores and refining the metals so they can build a new Optimus robot factory. Elon calls this the infinite money glitch.

马斯克说的是，我在造Optimus机器人，而且我想造一座全自动的Optimus机器人工厂，这座工厂产出的机器人能力强到可以去挖矿、提炼金属，然后再造一座新的Optimus机器人工厂。马斯克管这个叫「无限钱印钞机」。

Nate Soares52:56

if you're the Aztecs watching the sort of boats, the Spanish boats come in apocryphally because the Aztecs were not on the coast, you should not really imagine that the boats are just full of very strong warriors, right? They're full of warriors that are like clad in solid metal with sticks where they point the sticks at you and you die.

如果你是阿兹特克人，看着西班牙人的船开进来——虽然按史实阿兹特克人其实不在海岸——你不应该设想船上装的只是更强壮的战士。船上是浑身裹着金属的人，拿着棍子一指你，你就死了。

Nate Soares57:03

Only 10% chance it kills us all. That's not an only 10% chance it kills us all. That means we're in serious danger because these guys don't have a plan.

只有一成概率灭绝人类——这可不是「只有一成概率灭绝人类」那么轻松。这恰恰说明我们处境非常危险，因为这些人根本没有方案。

Nate Soares1:08:13

数字与实体

术语

alignment对齐

让AI的实际目标与人类意图保持一致的技术问题

gradient descent梯度下降

训练中反复微调参数、让输出更接近目标的算法

mechanistic interpretability机制可解释性

通过分析神经网络内部权重理解AI决策过程的研究方向

swarmAI集群

同一模型并行运行的多个实例协同完成任务

tokenization分词

把文本切成AI实际处理的最小单元(token)的过程

收听指南