Skip to content
Skjld Labs

Millions of books scanned and discarded: the hunt for human-written training data

AI companies are buying up huge volumes of used books, cutting off the spines, scanning the pages and discarding the remains. Behind the practice lies a growing scarcity of authentic, human-written text. We look at what is driving the development, how the courts have responded, and what the hunt for training data tells Norwegian businesses about the value of their own data.

3 min read Laget med AI-hjelp
Books on a conveyor belt going into a scanner and turning into glowing particles

Norwegian outlet Nettavisen recently covered how AI companies are buying up enormous quantities of used books to source training data for their models. In many cases the books are cut apart, scanned at high speed and then discarded. The practice has sparked debate about both copyright and the preservation of literature, but it also tells us something more fundamental: human-written text is becoming a scarce resource.

Why old books have become gold

The reason lies in the quality of the data the models are trained on. The internet is increasingly filled with AI-generated content, and if language models are trained on text written by other language models, quality can gradually erode, a phenomenon researchers call model collapse.

Books written before the AI wave took off in 2022 have therefore become unusually valuable. They are guaranteed to be human-written, they are edited and quality-assured, and they cover subject areas with a depth web articles rarely match. For model developers, the second-hand bookshop has in practice become a data warehouse.

"Project Panama": scanning at industrial scale

Court documents covered by the Washington Post show how the AI company Anthropic built an enormous digital book archive under the code name "Project Panama". The company brought in the former head of Google's book-scanning project and spent tens of millions of dollars buying millions of physical books from large used-book retailers.

The process was industrial: the spines were cut off with hydraulic machines, the pages were fed through high-speed scanners, and the paper was then recycled. In an internal document the company described the ambition as "destructively scanning all the books in the world", adding that it did not want the work to become known.

A partial win in court, and a billion-dollar settlement

The courts have given the industry a split answer. A federal judge in the US ruled that it was fair use to digitise and train on books the company had bought legally, even though the paper copies were destroyed in the process. The judge compared the training to an author reading books in order to learn to write.

The downloading of millions of pirated books from so-called shadow libraries, however, did not receive the same protection. That part of the case ended in a billion-dollar settlement with rights holders, covering several hundred thousand works. OpenAI, Google and Meta also face similar lawsuits over how training data has been obtained.

The fear that rare books will be lost

The practice worries antiquarian and second-hand booksellers. In the Netherlands, Germany and Switzerland, used-book dealers have received bulk orders for several thousand titles at a time, requests so unusual that several dismissed them as scams. Meanwhile, dedicated services are now being marketed offering AI companies anything from a thousand to a million books per order.

The fear is that out-of-print and rare titles will be bought up, scanned and destroyed, leaving few physical copies behind. There are as yet no documented cases of rare books actually being destroyed this way, but the concern has reached the very top of the technology world: Elon Musk has said his own AI teams will preserve rare books and scan them "the hard way", without destroying them.

What this tells Norwegian businesses

The story of the book scanning is above all a story about data quality becoming the bottleneck in AI development. When the world's largest AI companies spend tens of millions of dollars on physical books, it is because curated, human-made content has a value generic web text cannot replace.

That insight applies at smaller scales too. A company's own data, such as documentation, customer history, internal notes and reports, is precisely this kind of curated, domain-specific material, and often what determines whether an AI solution hits or misses. At the same time, the lawsuits show that where training data comes from has consequences: provenance and licensing are becoming a real compliance question, including under the EU's AI Act.

Our recommendation is therefore twofold. Treat your own data as the strategic resource it is, and ask your AI vendors the question the book trade has now forced into the open: where does your data come from?


From article to practice

Want to put this to work?

AI advisory. We find out where AI makes sense for you, build what is needed and train the people who will use it.

Read about AI advisory Book a free call
Industry and market

Read on.

Newsletter

AI in practice, one email a month.

What we've tested, one tip you can use, and what's coming next.

You'll get the Skjld Labs newsletter once a month, and can unsubscribe at any time. Privacy

First step

Thirty minutes. Free.

A no-obligation conversation about what you are facing. We will say so if AI is not the right answer. We are happy to meet you at Langes gate 8 in Sandefjord, or on video.