Google Invented the Transformer, Then Handed the LLM Era to OpenAI
Google held the Transformer, TPUs and the most compute at once, yet the burden of search advertising and infighting between two AI teams let it hand the most critical track — language models — to OpenAI.
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
Google's underlying problem was solved once and for all
In 2000, Google's index had not updated for five months, and engineers searched for days without finding the cause. Jeff Dean and Sanjay compared a batch of seemingly unrelated errors side by side and discovered that a single bit in machine memory would mysteriously flip — to save money, Google's servers used cheap memory without error correction. They did not ask for more expensive machines; instead they added a layer of insurance at the software level, automatically isolating any machine that produced errors. This is the most typical way the two work: not fixing failures at the surface, but going deep into the underlying layer to solve this class of problems once and for all, so that other engineers no longer have to consider whether the machine itself might fail.
Building a chip specifically for neural networks
Jeff Dean used speech as a single estimate, made a two-page PPT for management: in the future, every Android user speaks for three minutes a day, and just processing that speech would require Google's global data centers to expand two to three times. His solution was to build a chip specifically for neural networks, ignoring everything else. This went against his long-held view on chips — specialized hardware is usually not a good idea, because by the time it is built the algorithms have already changed generations — but neural networks were the exception: image recognition and speech dictation are both fundamentally the same kind of matrix multiplication. To shorten the timeline, the chip was made as a plug-in card that fit into the server space originally reserved for hard drives, and from project start to installation in data centers took only 15 months.
The phone call Hassabis made to Hinton
Hinton held an auction by email from a hotel room, with buyers including Google, Microsoft, Baidu, and the fourth being Hassabis's DeepMind. DeepMind had little cash on hand, and what Hassabis bid was $10 million in company stock, roughly one-fifth of the entire company. The price quickly exceeded what he could afford, and after withdrawing he called Hinton and said that although he had only meant to offer ten million to buy you, in his heart he felt your true value was $50 million. Hinton took that in; Microsoft executives privately raised their offer by $10 million and he was tempted for a moment, but ultimately declined. Google finally paid $44 million to buy Hinton and two students.
Others look at the hood, he wants to see the engine
Google's due diligence team flew to London, and Jeff Dean was among them, tasked with verifying for Google whether what DeepMind claimed was actually true. At the time DeepMind had only 50 people and no revenue, yet the price offered was twice what Google usually paid for companies of the same size, and many at the negotiating table found it absurd; Jeff Dean also thought it was a bit expensive. After the PPT was shown, he made one request: he wanted to see the source code. Demos can be packaged, but code is hard to fake. The researcher responsible for writing the code panicked, because when doing research, code is written only to get experiments running as quickly as possible, and is usually messy and hard to read. Hassabis later called this moment crossing the Rubicon — once the code was handed over, there was no turning back.
The more successful AlphaGo became, the less likely independence was
DeepMind wanted to use the Alphabet reorganization to spin out from Google, and negotiations lasted over a year, with terms even put on paper. But Pichai said on the phone that the Other Bets column was reserved for moonshot projects far from Google's homepage, and AI no longer belonged in that category — it was becoming Google's most core technology. Hassabis's past strength was to first win a battle big enough, then use the victory to trade for the next stage's筹码, but this time the筹码 changed in the opposite direction: the more successful AlphaGo became, the more important AI was inside Google, the closer it was to the core of the homepage, and the less likely it could be placed in Other Bets. In April 2021, Hassabis announced at an all-hands meeting that the push for independence was over.
Throw away the entire search index — that suggestion was too crazy
Just after the Transformer paper was written, one of its authors, Shazeer, began thinking about its future and found Google executives with an almost crazy suggestion: throw away the index Google Search had used for so many years, switch to training a giant neural network with the new architecture, relearn and reorganize all the information Google holds, and answer whatever users ask. Even his co-authors on the paper thought the idea was too crazy, and the proposal went nowhere. Around 2018 he and colleagues trained a chat model called Mina, which could talk philosophy, chat about TV shows, and even make puns, and the more the two worked on it the more they felt it had promise. Over the next few years they filed three applications, including opening it to external researchers, integrating it into the voice assistant, and doing a public demo, and all three were rejected for the same reason: it did not comply with the company's AI principles.
The same company, yet guarding against the other reading its folders
DeepMind had no data center of its own; Google's data center was a giant shared compute pool, where money-making teams like Search, YouTube and Google Cloud and money-burning teams like Google Brain and DeepMind shared the same pool, and to train models you had to queue for quota. DeepMind and Google Brain fought year-round over talent and results, and most of all over machines. At the end of 2017, to let a machine learn chess, just playing against itself occupied 5,000 TPUs at once; the next year Google Brain used 64 TPUs in total to train BERT. After GPT-3 came out, both teams' goals became language models, and DeepMind had to change folder names so Google Brain would not know the parameter scale it was aiming for, so 280B was changed to 'mole'.
The opponent has one sharp knife, Google has a drawer of knives
In the second half of 2025, the biggest piece of AI business shifted to the enterprise market, and what enterprises spent the most on was not chat but writing code; half of this market fell to Anthropic, and the other half to Cursor. Google's problem was not that it failed to see the programming direction, but that it did too much, scattered across departments — just for external products, five departments produced six or seven coding tools, with overlapping functions. Google internally stipulated that in principle only its own AI tools could be used for writing code, and using third-party tools required special approval, but left a loophole: if you could prove business need, you could apply for an exception. Several former employees said that several teams at Google DeepMind responsible for training Gemini all applied for Cloud Code, on the grounds that 'the best engineers should use the best tools'.
In their own words · checked verbatim
We are going to make the greatest invention in human history. He will redo every product of today from scratch. If you insist that I pick one product, of course I can, but the fact that you ask that question shows you still haven't understood what we are actually doing.
我们要做出人类历史上最大的发明 他会把今天所有的产品 全部都重新做一遍 你如果非要我挑一个产品出来 当然可以 但你会这么问问题 就说明你还没有明白 我们到底在做什么
Once the code is handed over, there is no turning back. The group of people who understand this field best in the world had seen his entire hand.
代码一旦交出去 就再也没有回头之路了 全世界最懂这一行的一群人 把他的底牌看了一个遍
If the people you bring in are both powerful and able to understand this technology, then they will not be content to sit on the sidelines and give you advice forever. The so-called advisors may very well end up being your opponents.
如果你请来的是既有权势 又看得懂这项技术的人 那么他们不会甘心一直坐在场边给你提意见 所谓的顾问 最后很有可能就是你的对手
The question was never whether they saw this path at the time, but why, having clearly seen it, they ended up doing nothing.
问题从来不是当时他们有没有看到这条路 而是明明已经看见了 为什么最后什么都没有做
At age 12, Hassabis decided that the most worthwhile thing in his life was to apply his brain to science. At age 46, he stood before a room full of scientists and said, we cannot only do science anymore.
12岁那年 哈萨比斯决定 他这一辈子最值得做的事情 就是把脑力用在科学上 46岁这一年 他站在一屋子的科学家面前说 我们不能只做科学了
He wanted to once again have a small team of only about ten people, focused on doing just one thing.
他想要重新拥有一支 只有十来个人 只专注做一件事情的小团队
Figures
| Number of processor cores in Google Brain's cat-face experiment | 16000 | 9:07 |
| Time from Google's custom chip project start to data center installation | 15 months | 14:11 |
| Price Google paid to acquire DeepMind | about $600 million+ | 36:32 |
| Global viewers of AlphaGo vs. Lee Sedol | 200 million | 41:34 |
| DeepMind's losses in 2018 and 2019 combined | nearly £1 billion | 1:28:12 |
| Number of TPUs used by Google Brain to train BERT | 64 | 1:28:12 |
| Number of TPUs occupied by DeepMind for chess | 5000 | 1:28:12 |
| Amount Google paid to buy back Shazeer's team | about $2.7 billion | 1:52:23 |
Glossary
- Transformer
- A neural network architecture proposed in a 2017 Google paper that replaces word-by-word passing with an attention mechanism, enabling large-scale parallel training.
- TPU
- Google's custom chip designed specifically for neural networks, installed in data centers in 2015.
- Reinforcement Learning
- A method that lets an agent learn by trial and error through actions and adjust its strategy based on feedback; AlphaGo used it to learn Go.
- Sparse Model
- Splitting a model into many small networks and calling only one or two at a time, so the total parameter count is large but the computation per call is small.
- Chain of Thought
- Letting a model demonstrate its reasoning process before giving an answer, which can raise accuracy several times over.
- NeoLab
- An industry term for a new lab founded after leaving a big company that puts research ahead of product.
How to listen
Suited to founders and investors watching the LLM competitive landscape, especially those who want to understand why technical leadership does not equal product leadership, and how organizational infighting eats away at a first-mover advantage.
From 19:16 to 24:23, the part about Hassabis's childhood and the founding of DeepMind can be fast-forwarded; its information density is lower than the second half.