Stronger models make dirty data more lethal: cleaning is worth more
Jane Street's alt-data team finds that once data leaks future information into the past, stronger models become more dangerous; nailing down 'what was known at the time' is worth more than tuning the model.
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
Patterns get competed away once discovered
Eric defines this as an anti-inductive environment: in trading, patterns you find are often quickly digested by others and the market, then stop working; there is little knowledge that can accumulate like physical laws. Because of this, the signal-to-noise ratio in trading data is naturally poor—the whole market mechanism is to squeeze out extractable signals until what remains approaches a random walk. To keep winning, you must keep finding new facts, not cling to old patterns.
— EricPrices can go negative; don't hardcode assumptions
On April 20, 2020, CME crude futures near expiry settled at a negative price—just that one day, and the final settlement later turned positive. The reason wasn't that oil had no value, but that storage at the delivery point, Cushing, was nearly full and no one wanted to take physical delivery. Order entry and risk systems that long assumed prices must be positive failed on the spot: some couldn't place orders, others didn't know how to handle negative returns. Eric's conclusion: a large part of trading is understanding the messy details of reality—storage capacity, contract multipliers, quote units—and systems that get these details right are what keep risk limits meaningful.
— EricDomain experts don't know how good tools can be
For data and systems to be truly usable, the bottleneck is often not writing code but translation. Eric observes that traders have high tolerance for painful workflows, worn down by Excel and manual processes until they don't know they deserve better tools; conversely, software engineers who don't understand the trading domain can only build to a narrow spec. So his job is to explain the real mechanics of the domain to engineers, and 'how good it can get' to traders. At Jane Street, every desk now has such an embedded developer, a role that grew out of this need.
— EricParking spots don't equal sales
A classic alt-data example is counting cars in Walmart parking lots, assuming more people means better business. Eric says this example is actually bad: you want to infer spending, but spots only tell you foot traffic, which is indirect; about 20% of Walmart's shopping happens online, and cities like New York have no lots to count. More direct signals sit on the transaction side: card issuers, Visa/Mastercard networks, and credit union processing systems know how much people actually spend, and can sell anonymized aggregate statistics to buy-side firms.
— EricCorrected data turns backtests into time machines
When deciding whether to subscribe to a data feed, the vendor may have already corrected past errors; backtesting on the corrected pretty data tests not what you would have actually received at the time, but the right answer you only learn later. Eric says that's like simulating having a time machine. Jane Street's solution is to save every raw data file and the local timestamp of arrival, so no matter how many transformations are applied, you can reconstruct 'what we actually knew at that moment.' This is the same logic as distrusting exchange timestamps and trusting only your own data arrival time.
— EricLLMs know the future; keep them away from historical data
AI has two impacts on alt data: stronger predictive models raise the value of good data, and LLMs make it easier to extract features from text. But there's a causal trap—using the latest LLM to featurize old documents brings in later events: it knows Enron's eventual fate and may have seen subsequent revisions of a financial report in its training data. That creates new future information leakage, especially when processing historical data; you must pin down 'which model version at which point in time.'
— EricStronger models make data cleaning more valuable
A common expectation is that as neural networks get stronger, feature engineering and manual cleaning will be replaced. Eric thinks the opposite: once models can do more themselves, you need less manual feature work, but the data fed in must be more correct. If data isn't point-in-time and leaks future information into the past, stronger models will learn that problem; different models also have varying tolerance for outliers and dirty patches. Net, cleaning data becomes more valuable, not less.
— EricUsage-based pricing tempts teams to nickel-and-dime
Jane Street runs its data warehouse in its own machine room, not purely out of control. Eric's observation: moving the same queries to a cloud vendor priced by read volume or compute time gets so expensive you think 'this money could buy a bunch of hardware.' More importantly, usage-based pricing distorts behavior—teams constantly optimize bills and reshape workloads to fit the billing model; with self-hosting, cost becomes a one-time hardware investment, and teams spend time on research and queries, not saving money.
— EricIn their own words · checked verbatim
the patterns that you find often get discovered and competed away over time
Eric2:06
the domain experts like who are not themselves software engineers like don't understand how good things can be if we build the right software for it.
Eric32:33
Like the thing you're getting at is like how many people showed up which is different from how much they're spending.
Eric59:04
you're not actually studying like what trades you would have done if you'd been subscribing to the data set. You're uh simulating what you would have done if you had a time machine, right?
Eric1:04:08
if you use like a really up-to-date LLM to like that knows stuff about old data and trying to, you know, use it to featurize some old data, like you're going to get very confusing things happen because the LLM is smart. It knows things about the future.
Eric1:08:10
It seems like an enormous amount of like human judgment is incredibly important.
Eric1:12:27
Figures
| Jane Street data engineers count | First in 2023, now about 20+ | 43:42 |
| Walmart online shopping share | about 20% | 59:04 |
| Crude futures negative settlement event | April 20, 2020, one day closed negative, final settlement turned positive | 19:21 |
| Data engineer intern training cycle | 2026 interview, 2027 summer, 2028 start, 2029 fully productive | 1:22:40 |
| Desk dev coverage | Every desk at Jane Street has an embedded developer | 23:25 |
Glossary
- alt data
- External data used for investment decisions beyond traditional exchange quotes, such as satellite imagery and consumer aggregates.
- anti-inductive environment
- An environment where patterns get competed away once discovered, so knowledge is hard to accumulate.
- point-in-time data
- Data that records only 'what was known at the time' and contains no future information.
- desk dev
- An engineer embedded with a trading desk who combines domain understanding with systems development.
How to listen
CTOs building data teams, quant practitioners evaluating data vendors, and engineers who believe large models will automatically clean data.
Pure tech backgrounds can skip to the data engineering part after 33 minutes, but don't miss the negative oil price at 19 minutes.