Background agents will eat 90% of compute, so inference should trade latency for throughput
The next wave of AI inference is not chat but background agents, and they will eat 90% of compute. Sale Research therefore optimizes for throughput rather than latency, and will even accept 80% availability, in exchange for prices that fall by orders of magnitude — and it argues Nvidia's performance-per-watt advantage is overrated.
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
Every inference company races on latency; agents need endurance instead
He positions Sale Research as a ‘token factory’: serving open-source large language model tokens at prices nothing else in the market can match, and supporting the building of agents. They also offer Sale Box, a cloud virtual machine for long-running agents, built specifically for agents that run for hours, days, or even weeks. The company's theme is ‘abundance’ — delivering intelligence to as many people as possible at a sustainable cost. The rise of open models is an important tailwind, because customers want control and sovereignty over their intelligence. Existing inference companies all focus on low-latency inference, but agents need work that is more durable and runs over longer horizons.
— Neil MovvaBackground work ends at 90% because no human sits in the loop
He predicts that by the end of this year, background and real-time workloads will each account for 50%, but that over the long run background work will be 90%. The reason is that background tasks have no human in the loop, so token consumption is unbounded, whereas human attention is finite. Deep research and cybersecurity, for example, are better suited to running in the background; Parallel Web Systems wants to index the entire internet and monitor changes in real time, a task that requires intelligence at scale. The background market is unbounded, and that is the decisive advantage.
— Neil MovvaChatbots are forcing GPUs off their best operating point
A GPU is a throughput machine and likes to run at full load, but interactive chatbots demand low latency, which forces the GPU off its best operating point. The most profound change of the coming year is the shift from chatbots to proactive or background agents, at which point building a throughput-centric stack makes far more sense. The best latency is no latency at all — when you wake up, the work has already been done overnight. This shift changes what the entire inference system is optimized for.
— Neil MovvaThe internet was a one-time subsidy; data is no longer free
The internet was a one-time data subsidy: roughly 30 trillion tokens of high-quality text, or about 300 trillion under a broad definition, and models have already read the whole internet several times over. Future data will come from models improving themselves on verifiable tasks, using reinforcement learning environments — coding, math problems — that let the model iterate on its own. Random human feedback no longer has value, because models have already surpassed the average human; what is needed is expert-level feedback. Data is no longer free, and that is the new bottleneck on scaling.
— Neil MovvaInference does not need giant data centers; 1MW is plenty
Building a data center with 100,000 GPUs is much harder than building one with 10,000. In the US, a 100MW data center is already difficult, 1GW is close to impossible, 10MW is barely feasible, and 1MW is plenty. The market still assumes that centralized mega data centers are required, but inference workloads suit distributed 1MW data centers, and he is willing to buy those small pools of compute. He would even accept 95%, or even 80%, availability, because background agents are not latency-sensitive: a robust control plane can migrate around a single point of failure, and what you get in return is extreme price-performance.
— Neil MovvaDistillation cannot be stopped, so the only moat is the speed of staying ahead
If users own the output of their interactions with AI and publish it to GitHub, then over the long run distillation is unavoidable. The diffusion of information cannot be stopped; the only question is how fast it happens. So the key question is how long the value of a 3-to-6-month lead can last. Enterprise adoption is slow — many are still running Opus 4.6 or 4.7 — and they will not adopt the frontier quickly. That means a model's moat lies in the speed at which it keeps extending its lead, not in absolute capability.
— Neil MovvaRoute around HBM and offload the KV cache to flash
The view that most makes his friends think he has some superhuman ability is routing around HBM, and more extreme still, offloading to flash. He is keen to change model architectures so that offloading the KV cache to flash becomes more efficient. The inference community designs its systems around serving at 1 to 10 tokens per second, and that is the north star. At OpenAI's pricing, a trillion tokens costs at least $5 million, so a cost improvement of 3 to 6 orders of magnitude is needed. The world's demand for intelligence will always be there; the challenge is how the product community provides the entry point to it.
— Neil MovvaLosing TSMC costs at worst a 2x gap in performance per watt
From Hopper to Blackwell to Rubin, the gain in performance per watt on Bfloat16 multiplies has not been large; and from TSMC 5nm to 4nm to 3nm, performance per watt does not change dramatically either. Even if TSMC were lost, the best Western process — Intel, for instance — is at worst a 2x gap in performance per watt, far less severe than people imagine. Chip founders have to understand 3 to 5 supply-chain bottlenecks: TSMC wafer capacity, HBM capacity, advanced packaging, and possibly power. Designs should be spiky, picking the directions Nvidia has not prioritized — betting on an HBM shortage, for example.
— Neil MovvaIn their own words · checked verbatim
I love this market because it's unbounded. There's no human in the loop. So you can consume as many tokens as you like in the background versus human attention span.
Neil Movva8:00
I like the phrase that Internet was a one time subsidy on data. We got it for free. It's extremely high quality about 30 trillion tokens of high quality text, 300 trillion tokens. if you take a wider view on what qualifies as a good text and we've basically looked at it all already.
Neil Movva36:48
I like to say we write kernels in the whiteboard we go to the whiteboard, we describe what we think the machine should be doing then we succcly describe that in the natural language to a model and then the model is able to do the execution
Neil Movva41:05
I will buy like5% up time. And the reason for that is because it's background engine thing is things running in the background. You don't care.
Neil Movva54:14
I want abundant tokens and diverse harnesses. I want everyone to build their own harness, every company, every user, even make the agent your own.
Neil Movva1:08:23
There is always demand for intelligence in the world. The on ramps to that intelligence are our challenge as a product community.
Neil Movva1:10:31
Figures
| Forecast split of background vs. real-time work | 50/50 by the end of this year, 90/10 long term | 8:00 |
| Internet text token count | about 30 trillion high-quality, about 300 trillion broadly defined | 36:48 |
| Chip units in an NVL72 | 72 | 43:15 |
| Nvidia Blackwell chip output this year | 5 million | 1:00:44 |
| Cost of a trillion tokens at OpenAI pricing | at least $5 million | 1:10:31 |
| Cost improvement required | 3 to 6 orders of magnitude | 1:10:31 |
| Performance-per-watt gap between Intel and TSMC | at worst about 2x | 1:11:34 |
Glossary
- KV cache
- Memory that caches the keys and values in Transformer attention, avoiding recomputation during generation.
- HBM (high-bandwidth memory)
- Memory packaged together with the GPU, providing extremely high bandwidth but limited capacity.
- SRAM (static random-access memory)
- Extremely fast but small-capacity memory, typically used as on-chip cache.
- Tensor Core
- The hardware unit on a GPU dedicated to accelerating matrix multiplication, the compute core of deep learning.
- Harness
- The agent framework or toolchain built around a model; here, a user's own custom agent workflow.
How to listen
AI infrastructure founders, compute investors, model deployment engineers, and product leaders who care about inference cost and turning AI into products.