AI滅絕人類不需要恨意，失控的全自動經濟自己就夠了

核心論點 · 點時間戳可跳到原聲

原話 · 已逐字校驗

You do that to a trillion variables a trillion times for basically a good fraction of all the text ever digitized and for 100 million hard problems that you've dug up. And then the variables wind up in a setting that can talk, that can solve hard math problems that humans haven't solved. And we're sort of like, well, how about that.

你對著一萬億個變量反覆調一萬億次，對象是幾乎全部已數字化的文本，再加一億個高難度問題。調完之後，這些變量就變成了一個能說話、能解出人類都沒解出來的數學題的東西。我們當時就是，哦，原來是這樣啊。

Nate Soares12:18

They were sort of told, use this vulnerability to break that system, which sort of like being told to use these lock picks to break that safe. And they instead found a universal exploit that would let them get the, like capture the cybersecurity flags regardless of what they were told to do.

他們被告知，用這個漏洞去攻破那個系統——這就像被叫去用這幾把撬鎖工具去開那個保險箱。結果他們找到了一個通用漏洞，不管題目要求是什麼，都能直接拿到網絡安全比賽的分數。

Nate Soares22:34

I think of this a little bit like if you have a bunch of alchemists trying to make a potion of immortality and some of them are just like breathing mercury and other ones are like, don't worry, we have fume hoods.

我覺得這有點像一群鍊金術士在嘗試煉製長生不老藥，有些人直接在呼吸水銀，另一些人說，別擔心，我們有通風櫥。

Nate Soares34:40

Elon Musk is like, I'm building the Optimus robots and I would like to build a fully automated Optimus robot factory that can produce Optimus robots that are capable enough of mining the ores and refining the metals so they can build a new Optimus robot factory. Elon calls this the infinite money glitch.

馬斯克說的是，我在造Optimus機器人，而且我想造一座全自動的Optimus機器人工廠，這座工廠產出的機器人能力強到可以去挖礦、提煉金屬，然後再造一座新的Optimus機器人工廠。馬斯克管這個叫「無限錢印鈔機」。

Nate Soares52:56

if you're the Aztecs watching the sort of boats, the Spanish boats come in apocryphally because the Aztecs were not on the coast, you should not really imagine that the boats are just full of very strong warriors, right? They're full of warriors that are like clad in solid metal with sticks where they point the sticks at you and you die.

如果你是阿茲特克人，看著西班牙人的船開進來——雖然按史實阿茲特克人其實不在海岸——你不應該設想船上裝的只是更強壯的戰士。船上是渾身裹著金屬的人，拿著棍子一指你，你就死了。

Nate Soares57:03

Only 10% chance it kills us all. That's not an only 10% chance it kills us all. That means we're in serious danger because these guys don't have a plan.

只有一成概率滅絕人類——這可不是「只有一成概率滅絕人類」那麼輕鬆。這恰恰說明我們處境非常危險，因為這些人根本沒有方案。

Nate Soares1:08:13

數字與實體

術語

alignment對齊

讓AI的實際目標與人類意圖保持一致的技術問題

gradient descent梯度下降

訓練中反覆微調參數、讓輸出更接近目標的算法

mechanistic interpretability機制可解釋性

通過分析神經網絡內部權重理解AI決策過程的研究方向

swarmAI集群

同一模型並行運行的多個實例協同完成任務

tokenization分詞

把文本切成AI實際處理的最小單元(token)的過程

收聽指南