AI Safety Doesn't Lack Money, It Lacks Founders Willing to Get in the Mud
Coefficient Giving's cap on AI safety grants is not $200 million, and bigger single grants will appear in the next year or two; the real bottleneck is talent, especially founders willing to bet before the problem has happened.
The video won't play here. Listen to the audio instead:
The argument · timestamps estimated from transcript position
The only bottleneck is talent, not money
Max says it plainly: the most important bottleneck, perhaps the only bottleneck, is talent. AI safety organizations all have long lists of projects they want to do but lack the people to execute, including collaborations with AGI companies and governments. Coefficient Giving's cap on a single grant to a technical AI safety organization is not $200 million, and bigger single grants will appear in the next year or two. So the problem is not that there is no money, it is that no one is raising their hand.
— Max NadeauThree funding tiers, decisions in as little as two weeks
Project Tailwind splits funding into three tiers: preseed, for founders who have not yet assembled a team and have only a preliminary idea, $200,000 to $2 million; seed, which requires a founding team, multiple people covering key functions, and preliminary results, $2 million to $20 million; and for organizations that have already built a track record, they are willing to give up to $200 million in a single grant, and this can even be the first grant. They also borrowed VC pitch day: founders present for 15 minutes and get a decision within weeks.
— Max NadeauHits-based giving: most grants are destined to have no impact
Coefficient Giving has long used hits-based giving, borrowed from VC's hits-based investing: make a bunch of grants, most have no impact, and a few are good enough to make the whole portfolio worth it. They are now leaning harder into this logic, deliberately avoiding the common failure mode of grantmaking institutions — over-scrutinizing budgets and making sure every penny is spent correctly — and instead making sure the best projects get all the funding they need.
— Max NadeauThe $160 million grant was for ‘using AI to accelerate safety research’
Resolution received $160 million. CEO Geoffrey Irving is a leader in technical AI safety, formerly chief scientist at the UK AI Security Institute, and before that led AI safety teams at Google DeepMind and OpenAI. What moved Coefficient was not just his résumé but his plan: build a large research center directly studying the alignment of superintelligence, and spend a lot of time using AI to accelerate safety research — because on the capabilities side there is a ‘five years of progress compressed into one’ opportunity happening, and they want grantees to seize it.
— Max NadeauAI safety founders must be willing to think about problems that have not happened yet
Max says AI safety nonprofit entrepreneurship shares grit, energy, and management ability with ordinary entrepreneurship, but has one unique requirement: being extremely picky about impact theory, willing to pivot substantially; and being willing to think about speculative questions with no clear answer, and to bet on where AI is heading. Many people's instinct is to work only on the more solid, more verifiable problems of today, but most AI safety problems have not yet appeared. METR is an example: coming up with evaluation methods before AI capabilities took off has held up better over time than benchmarks from the same period.
— Max NadeauNonprofits most easily forget ‘who is supposed to change their behavior because of this’
Max points out a common failure mode of nonprofits: because they do not need to actually get customers to pay, they do not think hard enough about ‘who in the world is supposed to change what they do because of your work’. You either want people to read your report and change their views, or you want them to adopt a practice you developed, or you want them to pursue a line of research you cannot do. As a nonprofit entrepreneur, you have to think through every step of the entire impact supply chain; having a good idea and getting a good idea into someone else's head rely on different skills.
— Max NadeauRedwood cut to two people, then rebuilt a year later
Buck Shlegeris and Ryan Greenblatt of Redwood Research judged that mechanistic interpretability was not producing enough, and made a hard decision: shrink the organization to just two people, spend a year thinking about the biggest holes in the AI safety landscape and the most undervalued approaches, then expand again with ideas like AI control. At the time they took a lot of criticism, and it was unpleasant for everyone. But Max says that in hindsight, they and almost everyone they know think this had far more impact than continuing down the original path. No market force pushed them, and funders did not pressure them; the pressure came from acquaintances.
— Max NadeauSafety teams inside AI companies are squeezed by a race to the bottom
Max thinks safety teams inside AI companies are in a brutal race to the bottom: they are pushed to develop safety techniques that are not costly to implement, to do pre-deployment evaluations quickly so models can be deployed internally to accelerate research or externally to make money, and to put out the fires currently burning. The conclusion is that these companies do not have enough willingness to pay, and lack top-down organizational priority and resource allocation. The best opportunity to increase willingness to pay and release the pressure of the race to the bottom is outside the AI companies.
— Max NadeauOutside organizations should do what AI companies cannot
Max says that if outside organizations develop new empirical safety techniques, their impact will be much smaller: you cannot get the constraints and cost details inside AI companies, nor their data. Jailbreaks have gotten harder over the past few years, but that can rarely be credited to outside researchers. What outside organizations truly cannot be replaced on is providing evidence and analysis, and assessing the state of AI risk — AI companies are not credible voices, and this kind of work requires comparing multiple AIs across multiple companies. Reading a system card and reading METR's misalignment risk assessment gives you a completely different picture.
— Max NadeauAnthropic sucks up talent, but the people inside may not be in the highest-impact roles
Max acknowledges that Anthropic hires fast, pulling people from institutions across the AI safety field. But he observes that even inside Anthropic, many people are not pursuing the higher-impact opportunities on their current trajectory. He thinks that for quite a few of them, leaving Anthropic for the third-party ecosystem would be better. From this he infers: many people working at Anthropic are not primarily motivated by having as much impact as possible — this is not necessarily wrong, but newcomers who see so many people doing AI safety at Anthropic easily assume they all thought carefully and concluded this was the highest-impact choice.
— Max NadeauNonprofits dare to touch problems without customers more than for-profits do
Max thinks nonprofits in AI safety are more ambitious in impact and in the scale of the problems they tackle than for-profit organizations. For-profits tend to work on problems that are already understood, already have customers willing to pay, and that VCs understand, and they affect people who can pay the bills. Nonprofits, supported by impact-motivated funders, can focus on problems that have not yet appeared. He gives examples: AI agents misbehaving and prompt injection are real problems today, but measured in dollars or in total suffering/joy, they are far smaller than speculative risks like AI causing a pandemic or AI going out of control and wiping out or displacing humanity — yet the latter are hard to raise VC money for, because ‘what is the product? Who is the customer?’
— Max NadeauSix categories of organizations he most wants to see founded
Max lists six: one, organizations that independently audit and evaluate AI companies' safety practices — the Hugging Face incident and the Mythos UK AISI incident showed the need; two, research centers working on alignment and new techniques to make powerful models safe, such as chain-of-thought monitorability; three, better evidence generation — for example, the misalignment incident inside OpenAI was discovered externally by Hugging Face; four, technical research related to security and verification, including supply chain vulnerabilities and training data poisoning; five, public goods in AI safety, such as MATS centralizing fellowships, or new compute clusters; six, fieldbuilding organizations that help newly interested people around the world enter the field.
— Max NadeauIn their own words · checked verbatim
I think the most important bottleneck, maybe the only bottleneck, is talent.
Max Nadeau0:00
I mean, $200 million is not like an upper limit on the largest size of a grant Coefficient Giving will ever give to an organisation in the technical AI safety space.
Max Nadeau0:00
So I think that there’s a lot of opportunity for people who are willing to get down in the mud of these speculative questions, and are willing to entertain these sorts of out-there hypotheticals.
Max Nadeau16:00
Because unlike with for-profits, where you can learn on the job and learn from this impossible-to-fake demand signal of, “Are you making money?,” in AI safety, no one will tell you if you’re not having impact.
Max Nadeau18:16
And we’re going to need more slack and more willingness to pay — for both research, and then also, just in practice, putting in place costly measures while we’re using AIs that are going to slow down things and reduce revenue and otherwise get in the way in order to prevent large negative externalities from being imposed on the world.
Max Nadeau31:02
So if you’re very morally ambitious, and you’re trying to have as much impact as you can, I think those are not necessarily the role models that you should be looking towards for what it looks like to try and have as much impact as you can.
Max Nadeau38:52
From my perspective, I see the nonprofits in the AI safety space as being much more ambitious with respect to impact and with respect to the problems they’re tackling than the for-profits.
Max Nadeau46:41
For example, the misalignment incidents inside OpenAI were only detected by Hugging Face, which was a third party outside of OpenAI.
Max Nadeau56:53
Figures
| Project Tailwind seed grant range | $2 million to $20 million | 4:30 |
| Coefficient Giving single-grant cap | $200 million, and not the final cap | 0:00 |
| Grant received by Resolution | $160 million | 8:00 |
| Participants in Redwood's ML for alignment bootcamp and REMIX | 30 to 50 people | 2:00 |
| Number of project stubs on the Tailwind website | 36 | 3:26 |
Glossary
- hits-based giving
- Making a bunch of grants, most with no impact, a few good enough to make the whole portfolio worth it.
- chain-of-thought monitorability
- Whether an AI's chain of thought can be used to supervise the AI and catch it when it misbehaves.
- preseed grant
- A small grant for founders who have not yet assembled a team and have only a preliminary idea.
- seed grant
- A mid-sized grant for organizations that already have a founding team and preliminary results.
- fieldbuilding
- Helping people enter and stay in a field through training, community, matching, and similar means.
How to listen
Founders who want to take grant money to start AI safety nonprofits, safety researchers considering leaving AI companies, and donors watching the AI safety funding landscape.
Anyone not interested in AI safety funding mechanisms can skip the funding tier details from 3:26 to 6:07.