The world is too loud. Read what matters.

晚点聊 LateTalk

Pre-training innovations are plateauing, but bringing Scaling to AI drug design is just beginning

ByteDance's protein structure design model trained a 900-trillion-parameter model on just 26.5 million structure data points, while public structure data contains 1.3 billion—compute, not data, is AI drug design's current biggest bottleneck.

AI drug designScalingPre-trainingByteDanceProtein structure designStartup funding

The video won't play here. Listen to the audio instead:

The second half—covering training failure costs, ByteDance's distillation ban, and wet lab business models—is most information-dense; early structural details can be skipped, but don't skip the entire episode.

The argument · tap a timestamp to hear it

2:52

Three independent ByteDance teams race to dominate AI drug design

Gu Quanquan leads AI drug design under ByteDance Research; two other independent teams exist elsewhere—one on small molecules, one that shifted from small molecules to proteins, creating the ProteinX series. These teams report to different divisions and operated largely separately until Try2 appeared. Recognizing the need to quickly pivot to stay competitive, Gu's team shifted from SeedFold to SeedPro2 within six months to overtake Try2. The teams' work began overlapping, but ByteDance didn't force a merger—instead letting the fastest-moving team emerge naturally.

— Gu Quanquan
14:04

Try2 transforms protein design from rare wins to consistent success

Try2, an antibody and nanoantibody design model from Try Discovery Lab, achieved a 10%-20% design success rate, dramatically exceeding David Baker's lab's prior 0.1% rate. Though its structural prediction didn't match AlphaFold3, it showed that modest improvements in structure prediction can yield real pharmaceutical value when aimed at design. ByteDance followed with SeedFold (exceeding AlphaFold3) and SeedPro2, achieving 70%-80% success rates with affinity dozens of times higher than Try2.

— Gu Quanquan
28:20

Model scale expands both capability and the cost of failure

‘Training collapse’ (训飞) refers to sudden Loss or gradient swings, or downstream metrics plateauing for days—indicating failure requiring debugging and restart. Gu reports this became common across Chinese tech companies and startups mid-2024 after model scale and training tokens jumped from 10 trillion to tens of trillions. Longer training cycles demand higher optimizer, normalization, and stability standards; any overlooked detail can compound into massive losses. A billion-parameter model failure costs roughly tens of millions of dollars, making this an increasingly capital-intensive game.

— Gu Quanquan
37:00

ByteDance bans distillation to solve problems, not follow solutions

Until at least his departure, ByteDance prohibited training on distilled data. The tradeoff: near-term capability gaps. The payoff: genuinely mastering high-quality data generation, then producing it perpetually rather than depending on others' distilled outputs. Gu likens this to football recruitment—signing foreign players boosts immediate results but doesn't build youth academies and long-term competitiveness. Distillation maintains top-tier status; refusing it pursues world-first leadership.

— Gu Quanquan
46:28

Pre-training feels stale because organizational consensus matters more than raw speed

Gu admits finding pre-training ‘somewhat boring’ (有点无聊) in later stages, but stresses this reflects organizational risk-aversion, not technical plateau. Many proposals had been debated for months; the chosen path was often months-old before execution began. Major decisions require 90% consensus; if 10% (including Gu) spotted the direction sooner, he still couldn't rush ahead—failure costs too high, team cohesion too precious. This is the price of institutional caution, not lack of technical frontier.

— Gu Quanquan
55:04

AI drug discovery is data-rich and compute-starved: Scaling's unrealized frontier

Public protein structure data totals 1.3 billion samples; SeedFold trained on just 26.5 million, reaching 900 trillion parameters. Most AI drug teams operate at single-digit-thousands of GPUs, far below 10,000-card scale. Data still outpaces available compute. This imbalance is Gu's thesis: sufficient hardware investment builds defensible moats—otherwise AI drug companies resemble traditional biotech with hundreds of GPUs, undifferentiated. Compute, not data, is the bottleneck.

— Gu Quanquan
1:02:51

Wet labs must be in-house or GPU clusters sit idle

Wet labs cannot be outsourced to CROs: designed molecules need in-house verification to count. High-throughput validation must keep pace with Scaling's output rate; outsourced dependencies create bottlenecks that idle compute clusters. Hardware costs roughly triple when buying vs. renting; self-building pays off over three years. Building a wet lab base costs roughly millions in the US—far cheaper than compute clusters—yet investors often shock at compute spend, overlooking the lab's true economic advantage.

— Gu Quanquan
1:22:28

Academia's friendliness masks mediocrity; industry competition exposes real gaps

Gu compares academia's discouragement of same-direction competition to wuxia sects claiming superiority while avoiding confrontation. In fast-moving AI, that courtesy breaks: public forums like Arena reveal true capability instantly. Industry thrives on head-to-head competition (all targeting GPT/Claude) and surfaces real gaps faster than academia's ‘doing different things’ (做不一样的东西) approach, which masks universal mediocrity beneath a veneer of differentiation.

In their own words · checked verbatim

‘Last June we saw Try Discovery Lab release their Try2 model for antibody design, achieving 10%-20% success rates—far higher than David Baker's lab's 0.1%.’

然后我们后来是到了去年6月份的时候 看到了Try Discovery Lab 他们发布Try2那个模型 然后它设计抗体 Band的成功率达到了10%20% 比之前David Baker实验室的千分之一的成功率要高不少

Gu Quanquan5:03

‘If any detail isn't done right or is done wrong, you can end up with significant failures.’

如果哪个地方 细节上没做好 或者做错了 那可能它就会出现一些 比较大的一些失败

Gu Quanquan28:20

‘That's tens of millions in losses, so this is an increasingly capital-intensive game.’

可能就几千万美金 所以这是一个越来越重资产的游戏

Gu Quanquan29:20

‘Short-term capability gaps are the cost, but the benefit is building it thoroughly—solving the fundamental problem—so you become stronger afterward.’

好处的话 可能就是说在某些方面 可能你模型能力在短期之内 可能会有些差距 这个是代价 但是好处就是说 你把这个事情做扎实了 真正从本质上 把这个问题找到了它的原因了 然后把这个东西解决了 那你之后 那你就变得更强了

Gu Quanquan37:26

‘To be world-class, you must figure out how others generate high-quality data yourself—that's the fundamental thing.’

你要做世界上第一最牛 那你必须要figure out别人是怎么样 生成这种高这样数据的 这个我觉得是一个就是说最本质的事情

Gu Quanquan38:26

‘Personally, by the later stages I found the work itself somewhat boring—genuinely a bit dull.’

就我自己比如说到了后期 我对这个事情本身是有点 觉得有点无聊 我确实有一点点无聊

Gu Quanquan46:35

‘When I told him I need 1,200 dollars in compute spend over six months, he was directly shocked.’

但是我直接跟他说 我算力六个月就要一千二美金的时候 别人直接就被shock到了

Gu Quanquan1:05:47

‘That's the downside you described—doing things differently masks the reality that everyone's a bit mediocre.’

那你刚才说的 其实是它 你觉得可能不利的一面 就是我们以一种 差异化的方式 可能掩盖了 大家都有点菜的 这个现实

Man Qi1:22:28

Figures

SeedFold parameters900 trillion (AlphaFold: 400 trillion)7:03
Try2 antibody design success rate10%-20% (David Baker lab: ~0.1%)5:03
SeedPro2 success rate and affinity70%-80%, affinity roughly dozens of times higher than Try27:03
Billion-parameter model training failure lossroughly tens of millions of dollars29:20
In-house wet lab infrastructure costroughly millions of dollars (US)1:05:47

Glossary

Training collapse
Sudden loss or gradient swings during pre-training, or stalled downstream metrics, requiring debugging and retraining.
Try2
Antibody and nanoantibody design model from Try Discovery Lab, achieving dramatically higher success rates than prior methods.
Scaling Law
Predicting and improving model capability by expanding model and data scale.

How to listen

Who it's for

Investors evaluating AI drug design/AI for Science opportunities, and engineers and founders seeking to understand large model pre-training organization structures and training failure costs.

Skip

Skip 32:52-35:25: garbled transcription with no substantive content.