AI writing has stalled: long-form nonfiction is the real test of model ability
Models are improving only slowly at long-form nonfiction, which exposes a defect in how they organize knowledge — a more worrying signal than their progress in code and math, because it bears on whether AI can solve open scientific problems.
The video won't play here. Listen to the audio instead:
The argument · timestamps estimated from transcript position
A model that can't explain settled science won't solve open problems
Criticism of AI writing clusters around creative writing, on the argument that good writing demands a highly distinctive voice, a point of view, and human depth of expression. But nonfiction writing — filler text, for instance — was one of the things LLMs were genuinely useful for. The author's point is that the stall in long-form nonfiction is an alarming signal for anyone relying on AI to solve open scientific problems on its own. Models struggle to organize and present the most settled science within their own domains, and that ought to be the precondition for solving the open questions.
— NathanielInsight comes from compression, and models are adding entropy instead
The author argues that current AI progress in science looks more like picking low-hanging fruit and making cross-domain connections than like revolutionary insight. Organizing knowledge is a form of compression, and compression is the necessary condition for insight. In long-form nonfiction, LLMs add entropy instead, and that cannot be stacked indefinitely. Models will depend on humans to steer them. Even so, the author remains optimistic that scientific progress in narrow domains such as mathematics will translate into broader ones.
— NathanielCoding went from mediocre to superhuman; writing has not moved
The author has just finished a textbook on RLHF, and used LLMs heavily along the way for typesetting, proofreading, and figures. He had expected that publishing a nonfiction book in 2026 would look dated, but the reality is that progress in writing fell far short of what he expected. Some of the models known for their writing, such as GPT-4.5 and Kimi K2, already count as old. Models have gone from mediocre to superhuman at coding and math, yet writing ability has stalled — it feels orthogonal to the other skills.
— NathanielWriting isn't neglected; inference-time scaling hasn't been unlocked for it
The author's view is that writing has not been neglected — the challenge is simply large and the targeted training data is scarce. There is some low-hanging fruit, such as dedicated toolchains and prompting, but it is unlikely to multiply capability. Writing is a very hard task, and unfortunately we have not yet unlocked inference-time scaling for it. Today's models are bad at long-form technical writing: individual sentences are acceptable, but a whole chapter comes out disorganized and vaguely worded, and it often contains conceptual errors.
— NathanielModels can check every sentence but cannot hold a chapter together
The author has observed that GPT models are excellent at catching typos and detail-level problems, while Claude models have more taste as editors, grasp the mental model of the task better, and offer interesting suggestions. What these cases show is that models are good at checking the content of each individual unit but bad at stringing the components together, which produces compounding errors. In math and code, RLVR (reinforcement learning from rule-based verification) magically reduced errors of this kind; in writing there is no equivalent yet.
— NathanielUsing AI output isn't cheating if an expert vets it, but it's a slope
The author reports that over the course of writing the book, AI models contributed under 1% of the sentences of technical explanation, because he, as the expert, signed off on those sentences. During editing he used Claude Code to navigate editorial comments, separating simple typos from complex problems and asking for suggestions. His position is that using AI output is not cheating if the expert vets it strictly — but it is a slippery slope, and he did end up accepting some AI suggestions during the full second-draft review.
— NathanielEffort saved in writing costs you the intuition you will need later
The author stresses that using AI for nonfiction writing erodes the gain in understanding, and that understanding — intuition, taste, instinct — is what will be worth more in the future. If you are not an expert, you cannot spot where the AI is deficient. He wrote the book to give back to the community, so even with AI in the loop, finishing it at all was a win. He grants that more human effort would have meant more learning and a better product, but the deciding factor was the worry that the book would already be dated by the time it was published.
— NathanielThe early organizing stage needs insight, and AI is nowhere close
The author sees AI models as good tools for turning knowledge into other forms, well suited to producing filler material or background content. But the early organizing stage of nonfiction writing requires insight, and there LLMs are nowhere near being a substitute. He would like more experts to use AI to write books, so that more knowledge gets shared; but at present AI saves only 10-20% of the effort, and it will not be the majority any time soon. Social pressure leads some people to conclude that writing a book is pointless, yet high-quality educational content has always been in short supply.
— NathanielBad prose means the datacenter-genius story does not yet hold
The author's summary is that models perform excellently in two situations: verifiable domains, and small edits with a lot of context, such as finding a bug or solving a specific math problem. They perform poorly at open-ended prose generation. Long-form writing will be cracked before creative writing, but this is a strong signal that models cannot express the full extent of their knowledge on underspecified problems. That caps their ability to become the "datacenter geniuses" who solve major scientific problems.
— NathanielIn their own words · checked verbatim
Models being stagnant in long-form, non-fiction writing should be alarming to those reliant on models autonomously solving grand, open science problems in the near future.
Nathaniel0:00
Today’s LLMs increase entropy in long-form non-fiction writing, and I don’t see how that can be stacked on top of itself endlessly.
Nathaniel1:30
Writing well is a very hard task! It’s a shame that we haven’t unlocked inference-time scaling for one of the great intellectual pursuits.
Nathaniel3:30
The models know how to check every unit of content, in this case usually a sentence or equation or figure, or make one, specific section where you are caught.
Nathaniel4:30
Using AI for non-fiction writing takes away from that progression. Doubly, if you weren’t already an expert you won’t be able to catch its flaws.
Nathaniel7:30
The problem is that you can only use AI models to save 10-20% of the effort today, and I don’t see that percentage becoming the majority anytime soon.
Nathaniel9:00
Long-form writing will definitely fall before creative writing, but it’s a strong tell that the models are not able to express the full extent of their knowledge in underspecified problems.
Nathaniel10:30
Figures
| Share of the book's sentences contributed by AI models | under 1% | 5:42 |
| Share of writing effort AI models save | 10-20% | 9:00 |
| Length of the book | 200-300 pages | 4:30 |
| Time needed to keep the Markdown and LaTeX versions in sync | tens of hours; five times faster once an AI agent was used | 7:00 |
Glossary
- RLVR
- A reinforcement learning method in which rewards are verified against rules, which reduces compounding errors.
- RLHF
- A training method that uses human feedback to shape model behavior.
- inference-time scaling
- The technique of spending more compute at the inference stage to raise a model's performance.
How to listen
Researchers who track the boundaries of AI capability, technical writers, and founders whose knowledge work depends on AI.
The opening survey of criticisms of AI writing can be skimmed; the substance is in the middle and later analysis of model capability.