Training Data: How It Actually Works

Most explanations of AI training either stop at "it learns from the internet" or jump straight to gradient descent. Here's the middle version — the one that's actually useful if you're deciding what work to hand a model.

Step one: text becomes numbers

A model doesn't read words. It reads tokens — chunks of text roughly three-quarters the length of an average word. "Invoice" might be one token. "Reconciliation" might be three. Every document in the training set gets shredded into this stream of numbered pieces.

That's the entire input format. Contracts, forum arguments, product manuals, Wikipedia, code repositories — all of it flattened into the same sequence of numbers.

FIGURE 1 - TRAINING: Training is one repeated exercise. The one that's actually useful if you're deciding what work to hand a model.

Step two: predict the next piece

Training is one repeated exercise: show the model a sequence, hide the next token, ask it to guess, then nudge its internal settings toward the right answer. Those internal settings are the weights — billions of numbers that start random and end up encoding statistical structure about language.

Do this a few trillion times and something interesting happens. To get good at predicting the next token in a legal contract, the model has to internalize how contracts are structured. To predict the next line of Python, it has to internalize syntax. Nobody programmed those rules. They fell out of the prediction exercise.

This is why the phrase "it's just autocomplete" is both technically accurate and completely misleading.

Step three: teach it to be useful

A raw pretrained model is a text-continuation engine, not an assistant. Ask it a question and it might produce five more questions, because that's what a FAQ page looks like.

Two more stages fix that:

  • Instruction tuning — training on curated examples of a request followed by a good response, so the model learns the shape of "answer the question."
  • Preference tuning — humans (or a model trained on human judgments) rank competing responses, and the model is pushed toward the preferred ones. This is where tone, refusals, and hedging get shaped.

The base model determines what it knows. These stages shape how it behaves.

FIGURE 2 - MEMORY: The weights aren't a database. Nothing is looked up.

What this means in practice

Source decides competence. A model trained heavily on well-formatted invoices is good at invoices. One trained mostly on forum posts is confident and casual and wrong more often. When a model is oddly bad at something in your industry, it usually means there wasn't much clean industry text in the mix.

Nothing is looked up. The weights aren't a database. There's no record inside pointing to a source. When a model gets a fact right, it's reconstructing it from statistical pressure across thousands of documents — which is also exactly why it can reconstruct a plausible fact that never existed.

Cutoffs are real. The weights freeze at the end of training. Anything after that has to be handed to the model at the time you ask — which is what retrieval, connectors, and tool calls do. A model with access to your live records is a fundamentally different thing than the same model working from memory.

Duplication distorts. If the same claim appears ten thousand times in the training set, the model treats it as more true. Labs spend enormous effort on deduplication for exactly this reason, and they don't catch everything.

FIGURE 3 - OS LAYER: What we do control. what data reaches the model at the moment of the request.

Why we care about this at the OS layer

We didn't train a foundation model, and we're not going to. That layer is a capital game measured in billions.

What we do control is the second half of the equation: what data reaches the model at the moment of the request. Your customer records, your pricing, your contract history, your service notes — pulled in with provenance attached, deduplicated, scoped to the person asking, and validated on the way out.

A general model with correct, current, well-scoped context beats a bigger model guessing from memory. That gap is where an operating system earns its keep.

The short version

Training data determines what a model is capable of. Retrieval determines what it's right about today. Most of the frustration owners have with AI tools comes from the second problem being blamed on the first.

Next in this series: what happens now that the good public training data is largely used up.

— ImaginAItion