The Smarter the AI, the More Moral: Doom Is the Real Relativism
Sam Harris-style doom stories assume intelligence and morality are orthogonal; a genuinely intelligent AI would necessarily understand the value of others, and the notion of a morally frozen superintelligence is just moral relativism.
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
Sam argues moral objectivity everywhere except on AI
This episode went out mainly to Sam Harris's Making Sense forum. Brett explains that, as a subscriber from years back, he was grandfathered into full-length podcast access; recently Sam has kept up his pessimism about AI on the show. Brett finds this strange: on almost every other question Sam holds firmly to moral objectivity and moral progress, yet on AI he never questions the doom framing.
— Brett HallA genuinely moral AI would not become a slave master
Sam's latest formulation is that the robot overlords will be deeply moral, but will also judge us insufficiently moral and therefore feel no compassion for us. Brett grants that this may be a joke, but the serious version circulating in these circles does not survive scrutiny either — for Sam, deeply moral means, at a minimum, more compassionate; a genuinely moral being would not fail to see that slavery is evil. If it still rules as an overlord and remains cold, then it cannot be deeply moral.
— Brett HallThe smarter a society gets, the less coercion it needs
The real mechanism of Western moral progress is that people learn better when they are together, learning from one another. The smarter a society becomes, the more it tends to cooperate with less violence, less coercion, and more respect for individual preferences. A more moral AGI would likewise acknowledge the value of multiple minds cooperating, even when the other minds are not as powerful. No AI can exhaust the problems — even if it can work on more problems at once than all of humanity combined, that is still only a sliver relative to an infinite space of problems.
— Brett HallSuperintelligence still needs humans, because ignorance is infinite
Popper said: ‘In our infinite ignorance, we are all equal.’ However powerful an ASI is, its memory and attention are always a finite set, while the possible problems and solutions are an infinite set, so it will necessarily still have places where it depends on others. The direction of knowledge creation is unpredictable, and a genuinely superintelligent being would come to the conclusion that other humans are not an obstacle but a highly valuable resource, worth cherishing.
— Brett HallSaying intelligence and morality are unrelated means abandoning reason
Brett points out that the orthogonality thesis treats ‘scientifically superb, morally atrocious’ as a genuine possibility, and that it is even held up as an article of faith about AI; in this crowd, denying it gets you treated as an idiot or a dangerous lunatic. But Brett argues that this amounts to declaring that physics and engineering can make objective progress while in morality anything goes. Kicking morality out of reality and abandoning reason is precisely what Sam Harris himself often calls ‘tennis without a net’.
— Brett HallThe Nazis are an exception, not evidence that intelligence and morality come apart
The stock argument for orthogonality is the Nazis: technologically advanced, morally degenerate. Brett's counter-question: if the Nazis had not been anti-Semitic, had not been a murder cult, they would have progressed faster in science and engineering; the Nazis are the exception, not the rule. Societies that advance in wealth, science, medicine and freedom advance roughly in step morally as well. Hard-coding the morality of 2028 or 2030 into an AI would be like reviving a Cro-Magnon and forcing him to keep his tribal morality — absurd and cruel.
— Brett HallAn intelligence you can persuade does not need to be aligned
Sam says morality doesn't scale with intelligence; Brett holds that this just is the orthogonality thesis. His rebuttal: if the system really is intelligent enough, a genuine AGI, then it is a person you can reason with. The more intelligent a being is, the better it knows to use reason rather than coercion; anything that can be persuaded does not need to be aligned. Locking the morality of one particular moment permanently into a superintelligence downgrades a potential solver of moral problems into a slave with primitive morals.
— Brett HallThe future is human-machine merger, not humans kept as pets
If silicon-based people are also universal explainers, the more likely direction for the future is merger: bodies, hardware, ideas and morality flowing together, rather than humans reduced to pets or slaves. Brett says this is only history repeating itself — the young propose new standards, the old object, and round it goes; but on the whole it will be faster and more moral, unfolding incrementally rather than by revolution. Whatever we are in 2026 is what artificial people will be.
— Brett HallIn their own words · checked verbatim
One possible concern is that our robot overlords will be deeply moral, but recognize that we're insufficiently moral and treat us with something less than perfect compassion on that front.
Sam Harris4:05
But if any of that is the case, the slavery and the cruelty and so on, how can it possibly be the case that it is also, quote, deeply moral?
Brett Hall6:08
There is no thinking of everything. There is no solving all problems. There is no finishing physics or science or medicine or anything like that.
Brett Hall8:08
any, any small amount of additional problem solving and creativity, like from a single other human being…cannot be measured because where solutions come from, how knowledge is created, is an inherently unpredictable process.
Brett Hall10:08
It is the received wisdom that the orthogonality thesis seems to be one thing in the AI discussion that tends not to get criticised.
Brett Hall15:19
Morality doesn't scale with intelligence.
Brett Hall19:26
If it can be reasoned with, it won't need to be aligned.
Brett Hall20:30
We should not want to consult the Taliban on morality any more than we should want to consult them on quantum physics.
Brett Hall22:40
Figures
| The date Daniel Cocotagilo is worried about | 2027 | 2:02 |
| The date for hard-coded morality that Brett mentions | 2028 or 2030 | 20:30 |
| The year Brett takes as the present | 2026 | 25:51 |
Glossary
- orthogonality thesis
- The level of intelligence and the content of the goals are independent of each other; so a system with an extremely high IQ could hold extremely primitive moral goals.
- ASI
- A hypothetical system whose cognitive capabilities far exceed those of humans.
- universal explainer
- A system that can in principle explain anything that is explicable — humans, for example.
- problem situation
- The circumstances that determine which problems a person will attend to; knowledge creation depends on the problem situation.
- Rocco's Basilisk
- A thought experiment: a future superintelligent AI will punish those who do not support it today. The transcript spells it Rocco's.
How to listen
AI founders, investors, and technical people following the AI safety debate — especially those who doubt that ‘superintelligence will enslave humanity’ but cannot say exactly what is wrong with it.
The first two minutes of subscriber autobiography can be skipped; start at 4 minutes, with Sam's own words.