Ready-made data connectors fail over 40% of the time, forcing OpenAI to build its own pipeline
Testing revealed error rates exceeding 40% on well-known MCP data connectors, pushing OpenAI to abandon third-party integration and build proprietary data pipelines as a core competitive advantage for its financial products.
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
Financial products work as consumer ChatGPT plus enterprise layer
ChatGPT for Financial Services is not a new model. It layers industry-specific capabilities—templates, data sources like S&P—on top of the same ChatGPT, co-designed with Morgan Stanley and Evercore. Joseph emphasizes this layering pattern repeatedly: real product value lives in the enterprise customization, not in advancing the base model.
— Joseph KimCodex adoption in finance jumped from 0 to 90%
Internal metrics show Codex adoption among software engineers rose steadily, but adoption in finance, legal, and sales stayed near zero—until Codex shipped a desktop app. Then adoption jumped to 90%. The bottleneck was never capability; it was making capability usable through a product interface.
— Joseph KimThird-party data connectors have over 40% error rates; build them yourself
The team wanted to use existing MCP connectors to ingest financial data. On the surface, it's cheaper—customers already own licenses and MCP subscriptions, so asking for a separate data-integration fee feels wrong. Testing proved otherwise: some well-known MCP connectors had error rates above 40%. The team pulled that work in-house, absorbing the cost. The payoff came in customer feedback, where data quality was the most-mentioned strength.
— Joseph KimCodex gives you 80%; the last 20% is the whole game
Joseph describes his team as operating like a startup built on OpenAI technology. Codex ships with memory, skills, and persistence—enough to get a financial product 80% usable. But wrapping Codex directly as a financial product would fail. That missing 20%—data, UI, integration—is the deciding factor. He sees this pattern (ChatGPT infrastructure plus proprietary secret sauce: data, UI, integration) spawning billion-dollar businesses.
— Joseph KimDesign model permissions like onboarding a new employee
How should next-gen agents be governed? Forget new safety frameworks. Reuse what human organizations already do: graduated trust. Like Google's readability credential—new engineers can't submit code in certain languages until they prove they know the codebase—models should have read/write tiers. Go further: borrow the investment-banking hierarchy (analyst, associate, VP, managing director) where every level reviews the next one's work. This mechanism for handling human error scales directly to models.
— Joseph KimThe killer isn't factual errors anymore—it's bad taste
Joseph separates two error classes: factual error (cite the wrong number, your whole conclusion collapses, you lose your job) and taste error (ugly slide colors, sloppy layout—non-fatal but damaging to how your work lands). He observes hallucination was a frequent word in 2022–23. Almost never mentioned now. Progress on factual errors is real. Subjective judgment—taste—is what comes next.
— Joseph KimTeach aesthetic judgment through pairwise human comparison
How do you train taste? The answer is strikingly simple: show people two slides, ask which is better, feed the pairwise preference signal into a reward model. This is early-stage RLHF—exactly how it started—which the field neglected while chasing verifiable rewards like "does the code compile?" Here's what surprised Joseph: humans size up a slide's aesthetics in seconds. Let models judge instead, and they burn massive reasoning overhead and still get it wrong. For taste, humans still outscore models.
— Joseph KimFinance benchmarks are broken at scale
Knowledge-work evaluations hit saturation with Astra. The parts still unsaturated are the ones that are broken. The team stopped reporting this metric entirely. There's a pattern internally: score goes up, no one asks questions; score doesn't move, dig into it. GPT-6 didn't shift on some finance evals. Investigation revealed the answer keys were wrong, prompts weren't crisp—the problem was the test, not the model. The team is now building new benchmarks that can predict whether a given job can actually be automated.
— Joseph KimIn their own words · checked verbatim
I really do think that there's going to be a billion-dollar business built on top of a ChatGPT plugin.
Joseph Kim0:08
Some of the MCPs that we connected, some big names, 40% plus error rate on their MCPs, right?
Joseph Kim6:08
But 95% of what we built was something that absolutely a startup could do, another company could do.
Joseph Kim7:10
Codex will get you about 80% of the way to a product that would be successful in the market. But the 20% is missing.
Joseph Kim8:12
Their work gets passed off to associates. Associates are reviewing the work. Then the associates are kind of adding their secret sauce, their magic, their reasoning. That gets passed off to the VP.
Joseph Kim12:00
the hallucination used to be a big term that used to come up in 2022, 2023 that doesn't come up nearly as often nowadays.
Joseph Kim14:00
humans are actually really, really good at aesthetic judgment, right? Actually much better than models is something that we discovered
Joseph Kim16:45
Figures
| Error rate on well-known MCP connectors | Over 40% | 6:08 |
| Product completeness delivered by Codex | ~80% | 8:12 |
| General-purpose capabilities in proprietary system | 95% | 7:10 |
Glossary
- MCP
- Model Context Protocol: standard protocol that lets models connect to external data sources and tools.
- RLHF
- Reinforcement Learning from Human Feedback; trains models using pairwise human preference comparisons.
- reward model
- A model trained on human preferences that scores model outputs to guide further training.
- readability
- Google internal credential; engineers must earn it before submitting code in certain languages.
- GDP eval
- OpenAI's internal benchmark set for measuring whether knowledge work can be automated.
How to listen
Product managers and engineers connecting third-party data sources and MCP connectors for enterprise customers, and professionals evaluating AI tools for financial institutions.
The final two minutes (starting 25:40) on recruitment have little relevance for non-job-seekers and can be skipped.