Large models' most insidious error in long documents: silently omitting information
Benchmark testing reveals a paradox: leading models excel at extracting from long documents, yet silently lose information in ways production systems can't detect.
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
The founder pivoted to documents even after securing a YC offer
Starting in memory augmentation, Reducto received a YC offer but decided to pivot on day one. The reasoning: agents would increasingly handle high-stakes domains like medical records, where accuracy determines patient outcomes. A misread checkbox isn't a small error—it can change treatment decisions. This insight drove the pivot from memory infrastructure to documents as the foundational bottleneck.
— Adit AbrahamAI agents' self-correction outperforms human annotation on structured extraction
Reducto's Deep Extract framework doesn't just extract; it makes agents verify their own output—checking invoice line totals against declared totals, for instance. On many structured extraction benchmarks, agents outperform human annotators. The reason: human taggers skip rows, take shortcuts, or fill partial answers. These errors became embedded in ‘ground truth’ datasets. Adit calls this the first time he's seen agents genuinely beat the human baseline.
— Adit AbrahamAt one cent per page, processing everything becomes the default strategy
Reducto's $0.01-per-page pricing emerged after processing 7 billion pages. At this cost, economics flipped: instead of choosing which documents deserve high-precision treatment, you run them all through. The company now processes billions of pages monthly and faces the next challenge—maintaining cost and quality at hundred-billion-page scale. Adit argues past AI products were retrieval-gated chatbots over limited data, but real agents need economic freedom to search widely without cost anxiety.
— Adit AbrahamFrontier model labs won't build document infrastructure because they're racing to AGI
Asked why leading model companies don't solve document processing, Adit points to misaligned incentives: AGI labs are in an arms race for super-intelligence, while document processing—however hard—is infrastructure work. Specialization wins. A general vision encoder must handle selfies-to-Ghibli synthesis and document reading; Reducto's model only needs to excel at one task. This focus delivers extreme cost and speed advantages over frontier models at the same accuracy.
— Adit AbrahamLong-document extraction fails on recall, not precision—models silently skip content
Reducto's independent benchmark with MicroOne produced a counterintuitive finding: on long-document extraction, leading models achieve good precision—what they extract is usually right. They fail on recall: models omit information without flagging anything is missing. Deep Extract solves this by adding agent validation rules that know when to double-check. This silent failure is nearly impossible to catch in production.
— Adit AbrahamPublished benchmarks are marketing noise; measure on your own data distribution
Adit critiques how AI benchmarks became marketing theater. Take a public benchmark, rotate documents one degree and add light blur—the vendor degrades while competitors hold steady. Most users never test these perturbations. His advice: use LLMs as judges, test against your actual data distribution rather than published leaderboards. When everyone claims SOTA, benchmarks become useless for real decisions.
— Adit AbrahamOne word wrong—chest pain vs. pain—and an insurance claim gets denied forever
Reducto's medical customer faced a real case where OCR confused ‘chest pain’ for ‘pain.’ When doctors queried patient history by keyword, nothing matched; the claim was denied. Understaffed medical systems can't manually verify OCR output across thousands-of-page records. In this scenario, Reducto found OCR accuracy sometimes fell below 50%. A single term error cascades through systems designed to trust extracted data.
— Adit Abraham80% of Medicare appeals overturn original decisions, exposing systematic approval failure
Millions of Medicare claims face prior-authorization denials yearly; 80% of appeals ultimately reverse the decision. This isn't operator error—it's systemic failure in paper-based workflows. Similar gridlock appears in USCIS immigration backlogs. Adit's insight: if accuracy, cost, and latency approach zero-friction, frontier models could automate these reviews and eliminate the appeals process itself. The reversal rate is evidence of process breakdown waiting for AI.
— Adit AbrahamIn their own words · checked verbatim
nobody thinks about complexity when they upload a document to ChatGPT, right? You just expect it to work.
Adit Abraham4:10
it's the first time where I find that agents can be better at structured extraction than humans on many tests.
Adit Abraham9:19
Today, literally this morning, we crossed our seven billionth page.
Adit Abraham11:24
I remember one of my friends came to dinner with a shaved head. They'd all shaved their heads for a million in ARR.
Adit Abraham15:30
They would silently drop information. That's the kind of thing that is really, really hard to catch in production.
Adit Abraham22:36
benchmarks have become marketing over time. And the downstream side effect is it's really hard to know what is good in your context when everything is like state of the art all the time.
Adit Abraham24:43
We were talking about individual cases where because the initial pipeline read something as pain instead of chest pain, somebody's claim gets rejected
Adit Abraham32:49
one of the craziest stats I learned is that in 80% of appeals, the original decision was overturned, which should be shocking because clearly there's some inefficiency in how the decision was originally made.
Adit Abraham33:49
Figures
| Pages Reducto has processed | 7 billion | 11:24 |
| Reducto's price per page | $0.01 | 11:24 |
| Supported file formats | 40+ | 26:43 |
| U.S. paper tax filings | 16% | 35:52 |
| Medicare appeal reversal rate | 80% | 33:49 |
| Traditional OCR accuracy range | 70–80% | 8:17 |
| Reducto accuracy (most scenarios) | high 90s% | 9:19 |
| Medical records OCR (some cases) | <50% | 32:49 |
Glossary
- Vision Language Model (VLM)
- A model that understands both images and text semantics, replacing template-based OCR systems.
- Intelligent Document Processing (IDP)
- The traditional industry term for document processing; Reducto sees its mission as broader infrastructure work.
- Harness
- A framework surrounding a model for verification, validation, and self-correction—not a single model call.
- Recall
- The percentage of information that should be extracted and actually is; long-document extraction's metric most likely to fail silently.
How to listen
Product managers and entrepreneurs building enterprise AI for document-heavy industries: contracts, insurance policies, medical records.
The opening entrepreneurial pivot story is somewhat broad; jump to 4:10 to start with model technology.