Chapter 7. The Flattening

I was looking online for a quick, failsafe bobotie recipe one afternoon, and one promising page that came up had everything a recipe page is supposed to have. Handy tips that didn’t so much contradict each other as provide nothing more than vague ambiguity and a neat little FAQ section; there was a neatness and rhythm about it that felt like its author had never been near a kitchen. Finally, the sentence that gave it away called South Africa’s traditional comfort dish ‘a delightful culinary journey’, and I knew it. AI, caught red-handed because I’m certain nobody who has ever stood over a pan of curried mince at six in the evening has said something like that in the history of the dish. I closed the tab feeling like I’d shaken hands with a mannequin and decided to forgo the shortcut, honour tradition and follow my grandmother’s typewritten recipe with the good apricots and the quantities led by instinct rather than measurement.

I recently found out the correct term for what happens when a model trains on its own output: model collapse. But before I had that official definition, I had already coined my own term for it, which is far less polite. AI cannibalism. I’d been noticing the quality of AI output slipping and getting steadily flatter, and when I mentioned it to a colleague almost in passing, she agreed immediately with the nod of someone who had already been thinking it, commenting it was the snake eating its own tail.

Our copywriter, Luke, put his finger on it best. “With each step forward in AI, I find it gets better and worse at the same time,” he said. “It’s like it’s not getting anywhere fast.” It’s learning from its own poor iterations that people publish all over the internet, so the averaging feeds itself. He’ll grant that AI can replace a writer, then stop you with the real question: can it replace a good one? “Not yet,” he says, because it is learning from the average, and a good writer is the opposite of the average.

I read an article about a background-removal tool that had started botching curly dark hair against light backgrounds. Where it used to cut clean, it now left halos and bit too deep in places, dropping strands it had caught without effort a year earlier. It had stealthily become worse at the thing its training data held least of, which bothered me more than a spectacular failure would have because people can still use a dull tool and not notice the decay unless they’re paying attention.

AI models in training find it harder to see the rare things as they start to get diluted with more and more samples that are levelled towards the average. The textures start to blur; sentences that refuse the usual rhythm begin to disappear; the image composition that doesn’t fit the default aesthetic becomes less likely to survive the next round. The surface stays clean while the strands go, snipped off a few at a time where nobody’s looking. Each generation trained on the last one’s output learns a slightly narrower world, and because every step is tiny, nothing really looks wrong. Nobody logs a ticket and nothing crashes because the tool isn’t broken.

LLMs started out learning from human-generated text. Imperfect, yes. The internet was never a clean dataset, but it was at least anchored to something outside AI itself. That’s eroding now, and fast. An alarming share of what gets published online is AI-generated. I notice the ‘AI tells’ constantly. The flatness of it, or the way a piece will use three synonyms where one word would do, or how it will open with a rhetorical question no real person would ask.

And honestly? That part matters. In this fast-paced digital landscape, we must strive to navigate this intricate tapestry of artificial intelligence—because at the end of the day, the future is not about replacing the human touch, but empowering us to embrace the journey—quietly.

Bajillions of these paragraphs are piling up in the feed, each one grammatically sound and spiritually vacant, and the next model will train on all of them as if they were language itself.

Imagine what happens to the feed in two years: Someone posts an article on Medium in 2026. It is generated by AI, not written by a human, so it would be grammatically sound; structurally familiar; the averaged product of everything the previous model learned. It does not get labelled as AI generated. In 2028, a new model trains on public text and finds that article in the sample alongside a well-thought-out, well-researched article a human spent a week writing, also published on Medium. There is nothing in the tokens that distinguishes the two and both arrive as equal evidence of how language works. The AI-generated article gets included in the training along with human writing, because there is nothing to mark it as the imposter to distrust in the sample. The system has no correction mechanism so it can’t reject what it cannot identify, and what it can’t identify is growing.

The model learns that this is how language works, because this is what language looks like now in the places it’s told to look online.

Nico Goosen, one of the last people I interviewed for this book, holds a PhD in artificial intelligence, and he told me where the loop goes next, and it goes there cheerfully. His nephews build AI agents to help with their schoolwork and arrive at breakfast announcing they are ready for tomorrow’s exam because they have passed four mock versions on GPT overnight. They also run the machine’s output through a second app whose entire job is to make the work read as though a child wrote it, because teachers have learned to spot the machine’s fingerprints. AI text, deliberately disguised as human, gets handed in for grading, and some of it will inevitably end up published somewhere a future model will find it. The loop doesn’t only contain unlabelled machine writing anymore. It contains machine writing wearing lipstick and a wig.

When I first understood what was happening, it felt unsettling. Not that I think all AI writing is inherently bad (some of it is fine, some of it is better than fine. I’m not going to do the romantic-about-human-output thing here). But because the feedback loop has no obvious correction mechanism, there’s no immune response. The system doesn’t know what it’s losing because it can only measure against itself.

There’s a concept in ecology called extinction debt, the idea that when you destroy a habitat, some species don’t disappear immediately; they persist for a while before they die out entirely. I think about that sometimes when I’m reading something that feels almost right but not quite. Perhaps we’re in that window. The original signal is still present enough that we can feel when something deviates from it, but we’re training the next generation on the deviation.

I’ve noticed that people using AI regularly, usually about three months in, will often echo my cannibalism suspicions. “It’s not as good as it was before.” They can’t quite articulate why. There’s something missing… and in its place your intuition signals vague disappointment in everything you once marvelled at.

An AI model learns by absorbing the whole distribution of human writing, our weirdness and our mistakes included, right down to the bits only three people on the internet have ever phrased in exactly that strange way. Those bits are where the juicy stuff lives, the joy of finding an approach that works because nobody’s tried it that way before. Train the model on its own polished averages instead, and that’s the first thing to go. It forgets the strange, uneven, specific human material that made it worth reading in the first place, and what’s left is all polished, averaged, and smoothed into an unremarkable parlance that reads as acceptable, but monotonous and frankly ‘mid’.

I put some of this to Richard Starkey (not Ringo Starr, obviously. IYKYK), a film editor who began his master’s thesis on AI and the creative economy in 2019, back when his university backed the subject precisely because nobody could yet see where it led. He’d been reading about studios feeding movie trailers to a machine to gauge which films would succeed, and he saw where it ended. A world, he wrote, where ‘any deviation from already-accepted formulae will be seen as an enormous risk.’ Nobody had a word for it yet. “Everyone can make their own superhero movie now, featuring themselves as the star, and nobody watches,” he told me. The tools got good enough to grant the wish, and the wish turned out to be boring.

When researchers deliberately trained models on the output of earlier models, generation after generation the later ones got measurably worse, narrower and more averaged until they collapsed into a kind of mush. They gave the effect its name, ‘model collapse’.

There’s a delicious little epilogue to the naming contest. While I was congratulating myself on cannibalism and the researchers were settling on model collapse, the author and journalist Cory Doctorow had already found the word the whole mess deserved. His version describes the platforms, everything getting worse on purpose in the service of one more cent of profit, but it stretches to cover the models, the feeds, the recipe blogs and the mannequin handshakes without straining a seam. The American Dialect Society made it word of the year. The dictionaries opened the door and let it in. My grandmother would not have. Enshittification.