AI Writing Has Stalled: Long-Form Nonfiction Is the Touchstone of Model Capability
Models are improving slowly at long-form nonfiction writing, exposing a defect in their ability to organize knowledge. This is more alarming than progress in code and math, because it bears on whether AI can solve open scientific problems.
The video won't play here. Listen to the audio instead:
The argument · timestamps estimated from transcript position
If it can't explain mature science, it can't solve open problems
Criticism of AI writing mostly centers on creative writing, holding that good writing requires a distinctive voice, a point of view, and deep human expression. But nonfiction writing, such as filler text, was once one of the things LLMs were genuinely useful for. The author points out that the stagnation of models in long-form nonfiction writing is an alarming signal for anyone relying on AI to autonomously solve open scientific problems. Models struggle to organize and present the most mature science in their own domain, and that should be a prerequisite for solving open problems.
— NathanielInsight comes from compression, and models are adding entropy to knowledge
The author argues that current AI progress in science looks more like picking low-hanging fruit and making cross-domain connections than revolutionary insight. Organizing knowledge is a form of compression, and a necessary condition for producing insight. But LLMs add entropy in long-form nonfiction writing, and that cannot be stacked indefinitely. Models will depend on humans for guidance. Even so, the author remains optimistic about the transfer of progress in narrow scientific domains like math to broader fields.
— NathanielCoding went from mediocre to superhuman, writing hasn't moved
The author just finished a textbook on RLHF, using LLMs heavily along the way for typesetting, proofreading, and figures. He had expected a nonfiction book published in 2026 to look obsolete, but the reality is that model progress in writing is far below expectations. Some models known for writing ability, such as GPT-4.5 and Kimi K2, are already old. Models have gone from mediocre to superhuman in coding and math, but writing ability has stalled, and it feels orthogonal to other skills.
— NathanielWriting isn't neglected, inference-time scaling just isn't unlocked
The author argues that writing isn't neglected; it's just a big challenge with a lack of targeted training data. There are some low-hanging fruits, such as dedicated toolchains and prompts, but they are unlikely to multiply capability. Writing is a very hard task, and unfortunately we haven't yet unlocked inference-time scaling for writing. Current models perform terribly at long-form technical writing: sentences are passable, but whole chapters are disorganized, vaguely worded, and often contain conceptual errors.
— NathanielModels can check every sentence, but can't string a chapter together
The author observes that GPT models excel at finding typos and detail problems, while Claude models have better taste as editors, better understand the mental model of the task, and offer more interesting suggestions. These examples show that models are good at checking the content of each unit, but bad at connecting components, which leads to compounding errors. In math and code, RLVR (rule-based verification reinforcement learning) magically reduces these errors, but there is no counterpart in writing yet.
— NathanielWith an expert gatekeeper it isn't cheating, but it's a slippery slope
The author shares that while writing the book, AI models contributed less than 1% of the sentences explaining technical concepts, because the author, as an expert, endorsed those sentences. During editing, he used Claude Code to navigate editorial comments, distinguish simple typos from complex problems, and request suggestions. The author believes that if an expert can gatekeep strictly, using AI output isn't cheating, but it is a slippery slope, and he ultimately accepted some AI suggestions during a full review of the second draft.
— NathanielThe effort writing saves is future intuition traded away
The author stresses that using AI for nonfiction writing weakens the growth of understanding, and that understanding (intuition, taste, instinct) is more valuable in the future. If you aren't an expert, you can't spot AI's flaws. The author wrote the book to give back to the community, so even with AI, finishing is a win. He admits that with more human effort there would be more learning and a better product, but the deciding factor was the fear that the book would be obsolete by publication.
— NathanielEarly organizing needs insight, and AI is far off
The author argues that AI models are good tools for converting knowledge into different forms, suited to producing filler material or background content. But the early organizing stage of nonfiction writing requires insight, which LLMs are far from replacing. The author hopes more experts will use AI to write books and share more knowledge, but for now AI only saves 10-20% of the effort, and won't become the majority in the short term. Social pressure makes some people think writing books is meaningless, but high-quality educational content is always scarce.
— NathanielBad prose means the data-center genius story doesn't hold up
The author concludes that models perform well in two situations: verifiable domains, and small edits with lots of context, such as finding bugs or solving specific math problems. But they perform poorly at open-ended prose generation. Long-form writing will be cracked before creative writing, but this is a strong signal that models cannot express all their knowledge in underspecified problems. This limits their ability to become a "data-center genius" solving major scientific problems.
— NathanielIn their own words · checked verbatim
Models being stagnant in long-form, non-fiction writing should be alarming to those reliant on models autonomously solving grand, open science problems in the near future.
Nathaniel0:00
Today’s LLMs increase entropy in long-form non-fiction writing, and I don’t see how that can be stacked on top of itself endlessly.
Nathaniel1:30
Writing well is a very hard task! It’s a shame that we haven’t unlocked inference-time scaling for one of the great intellectual pursuits.
Nathaniel3:30
The models know how to check every unit of content, in this case usually a sentence or equation or figure, or make one, specific section where you are caught.
Nathaniel4:30
Using AI for non-fiction writing takes away from that progression. Doubly, if you weren’t already an expert you won’t be able to catch its flaws.
Nathaniel7:30
The problem is that you can only use AI models to save 10-20% of the effort today, and I don’t see that percentage becoming the majority anytime soon.
Nathaniel9:00
Long-form writing will definitely fall before creative writing, but it’s a strong tell that the models are not able to express the full extent of their knowledge in underspecified problems.
Nathaniel10:30
Figures
| Share of sentences in the book contributed by AI models | less than 1% | 5:42 |
| Share of writing effort saved by AI models | 10-20% | 9:00 |
| Book length | 200-300 pages | 4:30 |
| Time needed to sync Markdown and LaTeX versions | dozens of hours, five times faster with AI agents | 7:00 |
Glossary
- RLVR
- A reinforcement learning method that verifies rewards through rules, reducing compounding errors.
- RLHF
- A training method that uses human feedback to optimize model behavior.
- inference-time scaling
- A technique that increases computation at inference time to improve model performance.
How to listen
Researchers tracking the boundaries of AI capability, technical writers, and founders who depend on AI for knowledge work.
The opening overview of criticism of AI writing can be skimmed; the focus is the analysis of model capability in the middle and later sections.