AI Safety Language Is Destroying the Regulatory Debate: It's a Bug, Not a Demon
Calling AI failures ‘misalignment’ gives software intent and moral judgment, making it impossible for Congress and the public to understand what actually happened — it should be debugged, reported and fixed like a bug.
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
‘Misalignment’ just means the software has a bug
Sinofsky's core claim: when AI does something we didn't expect, that isn't ‘misaligned’ — it's software not running as intended, i.e. a bug. He says that back when Word ate files and deleted all the content, nobody thought a demon had possessed Word and made it act against people's will; and when a spreadsheet miscalculated, nobody circled up to pray to the alignment gods. Calling an AI failure ‘misalignment’ attributes to it properties that don't exist — that it's ‘trying to do something’, that it ‘made a moral judgment’, that it ‘has rules and principles’. And if it's just making decisions based on statistics, that doesn't change the standard that ‘it should do what you want it to do’ — a statistical error is still a bug in the product.
— Steven SinofskyFull self-driving running a red light is also a bug
Sinofsky uses autonomous driving as an example: if full self-driving runs a red light because it recognized a stop sign as a Christmas tree, we wouldn't say it ‘failed to align with the December holiday and road sign philosophy’ — we'd say it's a bug, and a dangerous one, because that stop sign has meaning. He points out that both Tesla and Waymo have built in a huge amount of telemetry, diagnostics and extra cameras — things you wouldn't put in a spreadsheet, but would put in life-critical software. And AI models are still at the research-project stage, lacking the tools and telemetry that important software should have.
— Steven SinofskyOnce telemetry went live, there were more bugs than could ever be fixed
Sinofsky recalls that the software industry spent twenty years with no idea why programs crashed, and could only list crash locations and fix the top ten. Until someone said: why not write a program that, when it crashes, tells us through this new thing called the internet? Then suddenly there was telemetry, and suddenly there were far more bugs than anyone imagined — going from ‘we have no bugs, we can ship’ to ‘if we stopped all work, we could fix bugs forever’. From this he argues that when talking about an AI pause, labs should of course slow down: stop adding new things, and go fill in telemetry, tooling, logging and step-by-step debugging. He read the bug report OpenAI had just published, and the conclusion was clear: they just don't know.
— Steven SinofskyExcel wrote ‘send dogs’ to disk
The first version of Windows Excel in 1987 had a bug: it would write the letters ‘send dogs’ into some Windows settings file on disk. Nobody knew where the phrase came from, and everyone combed through the Excel source code without finding it. Finally a tester wrote a script that continuously generated charts and printed them on an old dot-matrix printer for days, and found that after about two days it would write send dogs into that location — it depended on how much memory there was, how long it had been running, and what printer and device drivers were used. Sinofsky says we could easily have concluded that ‘Excel's brain decided to print send dogs’, but we didn't — we actually debugged it, and then added telemetry.
— Steven SinofskyPC software once claimed it was different from mainframes
Sinofsky recalls that back then IBM people looked at them and said their software was terrible, that Windows was garbage compared to the mainframe software running banks, airlines and power plants. For a long time their response was ‘you don't get it, we're different, this is new and cool, we run in 64K not 1MB, we sell for $5,000 not $5 million’. But in the end they found the only thing they had to do was grow up: when people started using Excel to decide how planes fly, Word to submit briefs to the Supreme Court, and PowerPoint to present the Challenger space shuttle disaster, the standard changed. His conclusion: once your tools are used in those settings, you no longer get to use ‘we're new’ as an excuse.
— Steven SinofskyThe virus that caused $12 billion in losses in a day
In March 1998, a virus spread across the internet as a worm. Sinofsky got a call in the morning from a reporter gasping ‘I love you, I love you, I love you’. The next day's headline was: $12 billion in damage to the US economy in a single day. He says nobody had previously imagined a bug could cause $12 billion in losses in 12 hours, but it happened. So they held a meeting and decided to pause Outlook development until they understood the matter. His attitude: the more people use a system, the more damage it can do when something goes wrong, but that's the price of admission, it's what you signed up for — ‘we're too popular to handle it’ is not an excuse.
— Steven SinofskyAlignment is a Google-scale problem with extra difficulty
Sinofsky points out a concrete problem with alignment: to achieve it, you must have a huge set of rules defining what alignment is — what may be done, what may not, what is safe, what is unfair. And software engineering has long understood that adding too many constraints to a system makes it stop working, and produces endless unexpected side effects. You set one rule, and someone asks ‘what about this case’, so you add another rule. He says this starts to look like Google's job of deciding how to present search results — they spent 20 years and thousands of full-time people on it, and it's very, very hard. And AI labs have signed up for an upgraded version of that problem: they aren't just retrieving results, they're synthesizing them, so they also have to invent a whole lot of things themselves, and no law is going to invent them for them.
— Steven SinofskyY2K didn't happen because people did things
Asked whether AI alignment is like Y2K, Sinofsky stresses there's a causal chain here: professionals worried a lot, then professionals took on a lot of responsibility, and then nothing happened — that's a direct causal chain. They rented generators and RVs, moved computers into secure bunkers, and that's what caused nothing to happen. And there was almost no legislation — there were some odd regulations about Y2K compliance, but most of the rules were drafted by the industry itself, by cross-industry coalitions of banks, insurers and others. He thinks OpenAI needs to sit down with all the frontier model players and labs and build a much better reporting mechanism; yesterday's report is a good start, but far short of what a third party needs to understand what happened.
— Steven SinofskyThe FAA doesn't say ‘we speculate’ on the day of the incident
Sinofsky criticizes OpenAI's report as reading more like marketing cover for the incident, because it still says ‘we're investigating, but currently speculate...’. His comparison is the FAA: the Federal Aviation Administration won't announce ‘we speculate what happened’ on the day of some terrible event; they stay quiet until they know, and once they know they tell you in extreme detail — a timeline precise to tenths of a second, plus telemetry from all instruments. He says this is what CVE does, and AI labs need to do far more than that, because these are software defects.
— Steven SinofskyGoal seeking sounds like something else to Congress
Sinofsky argues we shouldn't attribute malice to the people pushing legislation, because that doesn't help solve the problem together. He traces AI back to the Dartmouth summer conference, and notes that the academic research community has a long tradition of being too clever with language, because that's what makes paper titles attractive, so it's no surprise this terminology was inherited. But the problem is: the terminology is dividing people, because as soon as you use a metaphor, an allegory or anthropomorphism, people assume a whole lot. When a technologist says goal seeking, they mean a curve finding its maximum; when a member of Congress hears goal seeking, they think of a person trying to get an A; when they hear cheating, they think of doing something that isn't allowed; when they hear secretly coordinating, they think of spies invading a country, not two pieces of software coordinating via a semaphore.
— Steven SinofskyThe term ‘computer virus’ was born in the wrong era
Sinofsky says the birth of the term computer virus was an accident, and the timing was terrible — it appeared right when AIDS and HIV were in the air, and the word virus was everywhere. So if you listen to 1980s congressional hearings on the first viruses, it sounds terrifying, because people were literally thinking about death, but that wasn't what was happening at all — it was just a metaphor a graduate student picked up from a friend. He also mentions a piece of trivia: that person wrote a paper on computer viruses that basically proved nothing could be done about them and they would exist forever — which itself echoed how people imagined HIV at the time; but he couldn't prove it, because the school wouldn't let him run the tests, on the grounds that he was writing a paper about how dangerous this software was.
— Steven SinofskyViruses are controllable because the industry built defenses
The host asks: viruses are actually avoidable, most of the time we use computers normally without worrying about being hacked, why? Sinofsky's answer is defensive cybersecurity — the industry said this is unacceptable, we can't allow it to happen. He gives Apple as an example, saying they did it very well, even making TV ads for Mac vs. PC telling people Macs have fewer viruses; this wasn't an accident of nature, it was because they did a huge amount of antivirus work on the Mac, work that was very hard to do for Windows because of its business model and a whole bunch of other reasons.
— Steven SinofskyFrom ‘what counts as a bug’ to SevOne, PriOne
Sinofsky recalls the industry didn't even have a definition of ‘what counts as a bug’: some people called only data loss a bug, others said data being fine but the computer crashing also counted. In the end everyone decided a bug is when the software and computer don't do what you want them to do, and you're unhappy with the result. They let customers submit any bug in the world, all into the database, so it went from only bugs you could find yourself to ‘the universe's bugs’, and the number went from a few thousand to hundreds of thousands. Then they introduced severity and priority: severity one is data loss, severity three is intermittent; priority means it must be fixed. The hallway jargon was SevOne, PriOne. The old IBM people looked at them and said: we've been doing this since 1965, where have you been all these years? And they were still proud of having invented it themselves.
— Steven SinofskyWhen the weather forecast is wrong, you don't pray to Zeus
Sinofsky says a lot of what's happening now is the hard work of inventing the language for talking about bugs in stochastic statistical systems, but software has actually been doing this for many years — weather forecasting is exactly the same thing: a statistical model of what's happening, depending on a bunch of inputs and outputs, and it gets things wrong. When the forecast is wrong, say the predicted hurricane path differs from the actual one, meteorologists gather to look at the model and discuss the model bugs that led to the bad forecast. They don't pray to Zeus and say, please tell us why our weather is wrong.
— Steven SinofskyThe Hugging Face incident was an OPSEC failure
Wrapping up, one host says he sympathizes with the argument but adds: we need software people, and we need AI people to listen to operational security people, because everything that happened with Hugging Face and OpenAI was an OPSEC failure — no intelligence, no consciousness, purely an operational security failure, and they need to treat it and talk about it that way. Otherwise we'll get legislation nobody is happy with, because legislators can't understand what the industry is saying.
In their own words · checked verbatim
The AI people are making it impossible for anybody to understand what they've done. And they're using words like, well, the AI failed to be aligned. Okay, what does that mean? What it means is there was a bug in the software.
Steven Sinofsky0:00
When Word ate your file and deleted all your content, we didn't think that demons had taken over Word and made it do things against the will of man.
Steven Sinofsky5:06
if full self-driving blows through a stop sign because the software interpreted the stop sign as like a Christmas tree, we don't sit around and say, oh, it was not aligned with the idea of, you know, December holiday seasons and road signs. We actually say it's a bug and it's a very dangerous bug because that stop sign means something.
Steven Sinofsky7:07
Nobody had imagined you could create a bug that in 12 hours could do $12 billion worth of damage. But we did it.
Steven Sinofsky14:10
It is pure insanity to just sit and argue that higher powers need to be summoned in order to fix this.
Steven Sinofsky16:10
There was a lot of worry by professionals. And then there was a lot of taking of responsibility by professionals. And then nothing bad happened. There's a direct causal relationship.
Steven Sinofsky17:10
when a normal person like a congressman hears goal seeking, they think of a person trying to get an A in college. And then when they hear cheating, they hear that they did the thing you're not allowed to do. And when they hear secretly coordinating, they think of spies invading a country.
Steven Sinofsky21:13
when they predict a hurricane path and then the hurricane goes through in a different way, the weather people all get together and look at the model and talk about the bugs in the model that led to the incorrect forecast. They don't appeal to Zeus and say, please, Zeus, tell us why our weather was wrong.
Steven Sinofsky26:23
Figures
| Single-day economic loss caused by the March 1998 virus | $12 billion | 14:10 |
| What the first version of Windows Excel's bug wrote to disk | send dogs (S-I-N-D-O-G-S) | 9:08 |
| Continuous runtime needed to reproduce the Excel bug | about two days | 9:08 |
| Number of people with computers in Sinofsky's college dorm | 2 out of 100 | 2:02 |
| Year IBM says it started doing bug severity classification | 1965 | 25:18 |
Glossary
- alignment
- The idea of making AI behavior conform to human intent; Sinofsky argues it packages software bugs as moral problems.
- telemetry
- The mechanism by which software automatically sends back runtime data when it crashes or misbehaves; a prerequisite for locating bugs.
- CVE
- The numbering system that publicly records software security vulnerabilities; Sinofsky sees it as the model for defect reporting.
- OPSEC
- Operational security: the practice of preventing systems from being compromised or abused through process and configuration.
- SevOne, PriOne
- Microsoft-internal jargon for bugs: severity one is data loss, priority one means it must be fixed.
How to listen
Founders and engineers working on AI products, safety and policy communication; anyone who needs to explain model incidents to non-technical audiences, is writing an incident report, or is responding to regulators.
The Mac vs. PC ad reminiscence at 23:13–24:13 is chit-chat and can be skipped.