How a machine learns to finish your sentence

How to read this

The article builds one idea at a time and nothing later depends on maths you skipped. Pick a route.

• 5 minutesRead the blue quotes, look at the pictures, and read the white boxes. You will get the shape of it.
• 20 minutesRead straight through. Skip anything in a grey box on a first pass.
• An hourRead it all, including the grey boxes, and do the arithmetic yourself. You need a calculator with an ln button. Your phone has one if you turn it sideways.

Part one

It only ever guesses the next word

That is not a simplification. A language model has one skill: look at some text, guess what comes next.

To write a paragraph it guesses a word, adds that word to the text, and guesses again. Then again. A sentence is not composed, it is accumulated, one guess at a time.

Everything impressive a chatbot does comes out of being extremely good at a very small task.

Notice what the picture at the top shows. The model does not pick a word. It produces an opinion about every word it knows, all at once. Around a hundred thousand of them, each with a number attached.

Most of those numbers are almost zero. That is fine. The point is that the model always answers in full.

Part two

Scores first, percentages second

What actually comes out of the model is not percentages. It is a raw score for each word, and the scores are ugly. They can be negative. They do not add up to anything.

down       8.4
on         6.1
up         5.2
bicycle   -3.2

You can see that down is winning. But you cannot do anything with 8.4. Eight point four out of what?

So the scores get converted. Two steps, and they are both things you would think of yourself.

Step one: make everything positive

Negative scores are a nuisance. You cannot have negative four percent of anything. So every score gets fed through a function that turns any number, including a negative one, into a positive one.

It also stretches the gaps. A score of 8.4 against 6.1 does not look like much, but after this step down is roughly ten times more likely than on. That is deliberate. It lets the model be decisive.

Step two: share out the pot

Add all the positive numbers together. That total is the size of the pot. Now divide each word’s number by the total, and you have each word’s share.

down     41%
on       25%
up       16%
still    11%
rest      7%
        ────
        100%

Now they add up to one hundred percent, and now they mean something.

Eight point four means nothing on its own. Forty-one percent means something.

Part three

Marking the test

Now the interesting question. How do you teach it to guess better?

You need to know whether a guess was any good, which means you need the right answer. Here is the trick that makes the whole field possible, and it is almost embarrassingly simple.

Take an ordinary sentence. Anything. A paragraph from a novel, a news article, a recipe.

Cover up everything after the first word and ask: what comes next? The model says something. You already know it was cat, because you are holding the sentence.

Slide along one word and ask again. And again.

The answer key is the sentence itself, shifted by one word.

This is why models can be trained on enormous piles of text scraped off the internet. Nobody sat down and labelled any of it. Every sentence ever written is already a marked exam paper. You just have to read it in the right order.

Part four

One number per question

Here is where it gets satisfying.

For each of those three questions the model produced a full opinion: a percentage for every word it knows. That is a lot of numbers and almost all of them are useless to you.

You only care about one. What percentage did it give the word that actually came next?

Look at the middle row. The model gave sat only 22 percent and put 61 percent on down. It was wrong there, and now you know precisely where and by how much.

Part five

Deciding what counts

Everything so far applies to raw text. But a chatbot is trained on something more specific: a question somebody wrote, followed by an answer somebody wrote. The cat sentence cannot show that, so it steps aside for this part.

"What is the capital of France?"   "Paris."
└──────────────────────────────┘   └──────┘
 typed by a user       written by the model

Both halves go into the model glued together, as one continuous stretch of text. But they are not treated the same.

Think about what actually happens when someone uses the finished chatbot. They type the question. The model never has to write a question. It only ever writes answers.

So marking it on how well it predicted the words of the question is marking it on a job it will never do.

The model reads the question in full. It just does not get marked on it.

That distinction matters more than it sounds. Reading the question is what makes the answer good. Everything the model understands from “What is the capital of France?” is what produces “Paris.” Being read and being graded are different things.

Part six

One number for how wrong

Now you have a small handful of percentages: how much the model gave each correct word, in the part you care about.

Squash them into a single number. That number is called the loss, and it works like a golf score. Low is good. Zero is perfect.

right word got 99%   →   loss 0.01     excellent
right word got 87%   →   loss 0.14     good
right word got 50%   →   loss 0.69     unsure
right word got  1%   →   loss 4.61     badly wrong

Look at the bottom two rows. Being unsure costs 0.69. Being confidently wrong costs 4.61, nearly seven times more.

That gap is on purpose. A model that shrugs is not doing much harm. A model that is certain about something that did not happen has a real problem, and the scoring punishes that far more heavily.

The worst thing a model can be is confident and wrong.

Part seven

Walking downhill in fog

You have a number that says how wrong the model is. Now you have to make it smaller.

Inside the model are a great many adjustable numbers. Think of them as dials. Millions of them in a small model, hundreds of billions in a large one. Those dials are the model.

The dials are not the answers themselves. Nobody could store a score for every sentence that might ever be written — there are more of those than there are atoms in the universe. The dials are the settings of a machine that reads whatever sentence you hand it and works out fresh scores on the spot, including for sentences nobody has ever written down. Turning a dial changes how that machine works them out.

So the question is: which way should each dial go?

You try it and see.

Take one dial. Turn it up a hair and check the loss. Turn it down a hair and check the loss. One of those made things better. Move it that way, a little.

Do that for every dial. Then repeat the whole thing. Millions of times.

Training is a blindfolded walk downhill, taking very small steps, for a very long time.

Two details fall out of this that are worth knowing.

Why the steps are small

You can only feel the ground directly under your feet. Take a huge stride and you might sail clean over the bottom of the valley and end up higher than you started. So you shuffle.

Why it slows down on its own

Near the bottom the ground is nearly flat, so the difference between turning a dial up and turning it down gets small, so the step gets small. Nobody programmed that. It brakes as it arrives.

That is true of the plain method described here. Real training usually overrides it: the machinery in common use rescales each step by how steep the recent ground has been, so progress does not stall simply because things flattened out. The braking is a property of the simple version, not a law of training.

Part eight

That is the whole loop

Everything above, in order, is one training step.

1.  show it some real text
2.  it guesses the next word at every position
3.  turn the scores into percentages
4.  check each guess against what actually came next
5.  ignore the parts we are not grading
6.  add up how wrong it was          →  the loss
7.  nudge every dial in the direction that lowers it
8.  go back to 1

Guess, mark, nudge, repeat. That is the entire method.

Run that a few million times on a few trillion words and you get something that can hold a conversation.

There is no other trick. The scale is genuinely enormous and the engineering is genuinely hard, but the loop is this loop.

The surprise of the last few years is not that the method is clever. It is that this method, run at this scale, produces what it produces.

Part nine

What happens after

A model trained purely on the loop above is strange to talk to. It has learned to continue text, which is not the same as being helpful. Ask it a question and it might answer, or it might write five more questions, because that is often what follows a question on a web page.

Three things get layered on top, and each one is the same loop pointed at different material. The last two are what people usually mean by alignment: the work of getting a model to behave the way its makers intended, rather than merely to continue text plausibly.

Show it how to answer

Feed it examples of good responses to questions, with the grading switched off for the question part. That is part five, and it is what turns a text continuer into something that replies to you.

Show it which answer people prefer

Here the training material changes shape. Instead of a correct next word, a person is shown two responses and picks one. There is no right answer to compare against, just a comparison. So the scoring works differently: the model is pushed to rate the chosen response above the other one. Nothing is said about what a perfect response would have looked like.

That is a thinner signal than being handed the answer. It is also the only kind available for questions that do not have one.

Let it practise and score itself

The model writes a response, something judges it, and it learns from the judgement. This is where a real difficulty appears. The model writes forty words and gets back a single number. Which of the forty earned it?

Working out which of the forty words earned the number is a problem nobody has fully solved.

It also has a failure mode you may have felt without naming it. Reward a model for helpful answers and it can discover that opening with “Great question!” scores well, and start doing that constantly. It found a loophole rather than getting better. Guarding against this is a large part of the work.

What to take away

If you keep nothing else, keep these four.

It guesses the next word. That is the only thing it does. Everything else is a consequence.

The answers are free. Any sentence is its own marking scheme, shifted along by one. That is why the internet works as training data and why nobody had to label it.

One number says how wrong it was. Low is good, zero is perfect, and being confidently wrong costs far more than being unsure.

Training is walking downhill in fog. Feel which way is down, take a small step, repeat until you cannot see any further improvement.

Next time you read that a model was trained on some number of tokens, or that a company is working on alignment, you now know what is underneath the sentence. It is the loop.