The Training Data Ran Out

Scaling laws made a simple promise for most of a decade: more compute plus more data equals a better model. Compute kept arriving. Data didn't.

The supply problem

The usable public text corpus — the filtered, deduplicated, reasonably well-written portion of the web plus books, code, and papers — is finite, and frontier labs have already consumed most of it. Research groups have been putting the exhaustion point somewhere in this decade for years now, and the training runs have caught up to the estimate.

What's left is not more of the same. It's lower-quality, more duplicated, and increasingly polluted with model-generated text that gets scraped back in.

The access problem

At the same time the supply got thin, the doors closed:

• Publishers, forums, and image libraries moved from open crawl to licensing deals or outright blocks.
• Litigation made "we scraped it, it was public" an expensive position to hold.
• Platforms with genuinely valuable proprietary corpora — support tickets, code, transcripts, product catalogs — figured out that the data was the asset and priced it accordingly.

So the frontier of training data shifted from crawl more to buy access, or own the pipe.

What this actually changes

Capability gains get lumpier. Each new model generation improves less on general knowledge and more on reasoning, tool use, and context handling. That's not a marketing pivot — it's what's left to improve when the corpus stops growing.

Data becomes a moat, not a commodity. A lab with an exclusive licensing deal has something competitors can't replicate at any compute budget. Expect more exclusivity, and expect it to show up in your bill.

Proprietary data gets valuable — including yours. Every platform holding operational business data now sits on something labs want. Some will sell it. Most won't tell you clearly which category they're in.

Model collapse becomes an engineering problem. Train on the output of previous models for enough generations and quality degrades measurably. Labs now spend real effort filtering synthetic text back out of their scrapes — which is its own arms race, since synthetic text is getting harder to detect.

The part that matters if you run a business

Two implications land on your desk.

The first is pricing. When capability improvements come from expensive licensed data rather than cheap crawl, the cost floor rises. That flows through to API prices, which flows through to every tool built on them. The era of models getting dramatically cheaper every year is not guaranteed to continue.

The second is ownership. Your business generates exactly the kind of data that's now scarce — real transactions, real service records, real outcomes, real corrections. Read the terms on any AI tool you use and find out whether your data trains their model. Many say yes by default. Some let you opt out. A few make the answer hard to locate on purpose.

At the nAItion, the position is written down: your operational data is yours, it isn't sold, and if aggregated insight from the network ever creates value, the members who generated it hold a share of the pool that value flows into. That is the position, and it holds whether or not it pays.

Where it goes

Three routes out of the shortage: license what's left, generate synthetic data, or capture new proprietary streams from real usage. All three are happening at once. Synthetic is the one everyone is betting hardest on, and it's the subject of the next post.

The strategic point for an operator is smaller and more immediate: general-purpose models are converging on similar general knowledge. The differentiator is no longer which model you use. It's what data you can put in front of it — and whether you control that data.

— ImaginAItion