Stronger Models Make Dirty Data Deadlier: Cleaning Is Worth More, Not Less
Jane Street's alternative data team found that once data leaks future information into the past, the stronger the model, the more dangerous it gets; pinning down "what was known at the time" is worth more than tuning the model.
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
Once a pattern is found, competition erases it
This is what Eric defines as an anti-inductive environment: in trading, the patterns you find are often quickly absorbed by other people and the market, and then stop working; knowledge that can accumulate over the long term like a law of physics is rare. Because of this, the signal-to-noise ratio of trading data is inherently ugly — the whole mechanism of the market is to keep squeezing out extractable signal, and what's left gets closer and closer to a random walk. To keep winning, you have to keep finding new facts, not guard old patterns.
— EricPrices can go negative, so don't hard-code assumptions into the system
On April 20, 2020, the settlement price of near-expiry crude oil futures on the CME closed negative — just for that one day, and the final settlement price later returned to positive. The reason wasn't that oil had no value, but that storage space at the delivery point, Cushing, was nearly full and nobody wanted to take physical delivery. Order and risk systems that had long assumed prices must be positive failed on the spot: some people couldn't place orders, some didn't know how to handle negative returns. Eric's conclusion is that a large part of trading is understanding the dumb details of reality: tank capacity, contract multipliers, quote units — only when the system gets these details right can you draw a ceiling on risk.
— EricDomain experts don't know how good good tools can be
For data and systems to be genuinely usable, the bottleneck is often not writing code but translation. Eric observes that traders have a high tolerance for painful workflows, worn down by Excel and manual processes to the point where they don't know they deserve better tools; conversely, software engineers who don't understand the trading domain can only build to a narrow spec. So his job is to explain the domain's real mechanics to engineers, and to explain "how good it could get" to traders. Jane Street now has such an embedded developer on every desk, and that role grew out of exactly this.
— EricThe number of parking spaces is not the sales figure
The classic example of alternative data is counting cars in Walmart parking lots — the fuller the cars, the better the business. Eric says this example is actually terrible: what you want to infer is consumer spending, but parking spaces only tell you foot traffic, very indirectly; about 20% of Walmart shopping happens online, and a city like New York has no parking lots to count. The more direct signal is on the transaction side: card-issuing banks, the Visa/Mastercard networks, and credit unions' processing systems know how much people actually spent, and can sell anonymised aggregate statistics to buyers.
— EricCorrected data turns a backtest into a time machine
When deciding "should we subscribe to this dataset," the vendor may already have corrected past errors; backtesting on the pretty corrected data doesn't test the data you would actually have received back then, but the right answer you only learned later. Eric says this is equivalent to simulating that you have a time machine. Jane Street's solution is to keep every piece of raw data and the local timestamp at which it was received, so that no matter how many transformations you do later, you can reconstruct "what exactly we knew at that moment." This is the same logic as distrusting exchange timestamps and trusting only your own data's arrival time.
— EricLLMs know the future, so keep them off historical data
AI hits alternative data on two levels: stronger predictive models raise the value of good data, and LLMs make it easy to extract features from text. But watch out for a causal trap — using the latest LLM to featurize old documents drags in what happened later: it knows how Enron ended, and it may have seen later revisions of this financial report in its training data. This amounts to creating a new leak of future information, and when handling historical data you especially need to pin down "which point-in-time model was used."
— EricThe stronger the model, the more data cleaning is worth
A common expectation is that as neural networks get stronger, feature engineering and manual cleaning will be replaced. Eric's view is the opposite: once models can do more on their own, you can do less manual feature work, but the data you feed in has to be more correct. If the data isn't point-in-time and quietly leaks future information into the past, stronger models will learn that problem; different models also have different tolerances for outliers and patches of dirty data. All told, cleaning data is worth more, not less.
— EricUsage-based billing pushes teams to keep pinching pennies
Jane Street runs its data warehouse in its own data centre, and it isn't purely about control. Eric's observation: move the same queries to a cloud vendor billed by reads or compute time, and the price is high enough that you'd feel "this money could buy a pile of hardware." More important is the distorting effect of the cost model — usage-based billing makes teams constantly optimise the bill and reshape workloads to accommodate the billing; once self-hosted, cost becomes a one-off hardware investment, and the team spends its time on research and queries themselves rather than on saving money.
— EricIn their own words · checked verbatim
the patterns that you find often get discovered and competed away over time
Eric2:06
the domain experts like who are not themselves software engineers like don't understand how good things can be if we build the right software for it.
Eric32:33
Like the thing you're getting at is like how many people showed up which is different from how much they're spending.
Eric59:04
you're not actually studying like what trades you would have done if you'd been subscribing to the data set. You're uh simulating what you would have done if you had a time machine, right?
Eric1:04:08
if you use like a really up-to-date LLM to like that knows stuff about old data and trying to, you know, use it to featurize some old data, like you're going to get very confusing things happen because the LLM is smart. It knows things about the future.
Eric1:08:10
It seems like an enormous amount of like human judgment is incredibly important.
Eric1:12:27
Figures
| Number of data engineers at Jane Street | The first one only arrived in 2023; today about 20-plus | 43:42 |
| Walmart's online shopping share | About 20% | 59:04 |
| Negative crude oil futures settlement event | Closed negative for one day on April 20, 2020; the final settlement price turned positive | 19:21 |
| Data engineer intern development timeline | Interview in 2026, summer in 2027, join in 2028, fully productive in 2029 | 1:22:40 |
| Desk dev coverage | Every desk at Jane Street has an embedded developer | 23:25 |
Glossary
- alt data
- External data used for investment judgments but not traditional data such as exchange quotes, e.g. satellite imagery and consumer aggregates.
- anti-inductive environment
- An environment where patterns get competed away once discovered, making knowledge hard to accumulate over the long term.
- point-in-time data
- Data that records only "what was already known at the time" and contains no future information.
- desk dev
- An engineer embedded in a trading desk who combines domain understanding with systems development.
How to listen
CTOs who want to build a data team, quants evaluating data vendors, and engineers who believe large models will clean data automatically.
Purely technical listeners can skip to the data engineering after 33 minutes, but don't miss the negative oil price at 19 minutes.