From Black-Hole Research to Housing AI: 90% Accuracy in Production Equals Failure
Papers praise LLM systems at 90% success rates; EliseAI processes roughly 60 million housing and medical calls yearly, where 90% reliability means millions of failures, so the team picks between LLM and classical machine learning per problem type instead of betting on one tool.
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
90% accuracy in production is catastrophe, not a passing grade
A 90% success rate looks respectable in research papers. But EliseAI handles roughly 60 million housing and medical calls yearly; that 10% error rate translates to millions of failed conversations directly affecting people's lives. This gap between theoretical performance and production consequence drove Mario from theoretical physics into production AI—problem complexity didn't shrink, it grew.
— Mario MartoneNot every problem deserves an LLM solution
Many newcomers treat LLM as a universal tool, but LLM performs poorly where you need determinism—identical inputs must yield identical outputs—and precise probability estimation. EliseAI's tenant-retention predictor is built on proprietary deep neural networks, not LLM, because they have massive historical data; classical machine learning is more accurate and faster. Teams must diagnose the problem first, then pick the tool.
— Mario MartoneEach additional nine in reliability is a summit by itself
Academic papers discuss 90% reliability for LLM as a solid achievement. But EliseAI's 60 million yearly calls mean 90% reliability triggers millions of complaint conversations rippling into real tenant and patient lives. This is the chasm between benchmark and consequence, which is why the team assembled both academics and engineers to obsess over evaluation methodology.
— Mario MartoneSmall-sample testing reveals nothing about method differences
Test five or six sample conversations and most approaches look equally good. Run the same methods on a million conversations and their quality diverges sharply. EliseAI applies scientific method to product: systematic evaluation at true scale, not gut feel or handpicked samples.
— Mario MartoneInternal benchmarks beat external hype for predicting production
EliseAI's speech system is entirely self-built, so they own every bottleneck. When new speech-to-text, text-to-speech models, or hyped benchmarks appear online, the team doesn't trust published results. They run it internally against real data immediately. They have already automated to Slack + Kubernetes for experiments and are moving toward full automation.
— Mario MartoneRental automation from search to move-in is already possible
EliseAI has automated the entire rental chain: browsing listings online, booking viewings, using access codes for AI-narrated walkthroughs, signing the lease, and moving in—all without manual handoff. Maintenance requests are mostly automated too—the AI judges urgency, contacts vendors, schedules visits. The only step requiring human labor is the actual repair work itself, which awaits a robot.
— Mario MartoneSubsidized housing represents your highest-leverage automation opportunity
Healthcare and housing account for roughly 40% of US GDP. That is why EliseAI goes deep rather than broad. The team is now focused on reducing operational costs for subsidized housing—a domain of extreme complexity, highest impact on the most vulnerable, and almost zero automation. Cracking it would directly improve residents' lives.
— Mario MartoneProduction AI proved harder than fifteen years of black-hole physics
Mario reflected at the show's close: despite fifteen years researching black holes and string theory, production AI's problem complexity exceeded theoretical physics. He attributes this to AI's pace of change and the requirement that production systems answer for real consequences—a compound difficulty theoretical research never faced.
— Mario MartoneIn their own words · checked verbatim
An AI system that works 90% of the time might look impressive in a research paper, but when you're handling millions of conversations about people's housing and healthcare, that remaining 10% becomes an enormous problem.
Mario Martone0:00
when they're talking about reliability, they talk, okay, if the LLM can take it up to 90%. And then we say 90%, that means for like 60 million of phone calls per month, per year, it means like an enormous amount of failures. That is totally unbearable.
Mario Martone8:30
At small scale, like five or six conversations, all look quite good. But at scale, when you get to the millions, then you see that different methods diverge significantly.
Mario Martone9:36
I don't really have to believe it. I don't even look at the benchmark.
Mario Martone12:42
We do have some Kubernetes loop, but we're working on animating it actually fully.
Mario Martone12:42
Health care and housing. It's like 40% of the GDP in the U.S.
Mario Martone18:59
I would say that the complex, the complexity of the problem of working now is probably higher now than it was before.
Mario Martone23:11
Figures
| Monthly call volume | ~5 million (~60 million annually) | 10:39 |
| Company size | ~600 employees | 16:56 |
| Open-source model fine-tuning time | ~8 months | 13:43 |
| Mario's theoretical physics research years | 15 | 2:06 |
| Healthcare and housing share of US GDP | ~40% | 18:59 |
Glossary
- fine-tuning
- Continued training of an open-source base model on proprietary data to adapt it to a specific domain.
- closed-loop experimentation
- Immediately testing new models against internal production data rather than relying on published external benchmarks.
How to listen
Engineers and product leads building enterprise or government AI systems, choosing between LLM and classical machine learning, and assessing production-environment reliability.
0:00–2:06, guest credentials and opening remarks.