How language models work

A language model is a program that continues text. Once you see how it picks each next piece, you can predict when it will help you and when it will confidently make things up.

A model continues text

A large language model (LLM) is trained on a huge amount of text to do one job: given some text, guess what comes next. Chat assistants, coding helpers and the answer step in LocalRAG all use that one ability.

You type:

The capital of France is

The model does not look the answer up. It works out which continuations are likely, given everything it saw in training, and picks one. Paris is by far the most likely, so you get Paris. Then it adds that word to the text and repeats, one piece at a time, until it decides to stop.

"The capital of France is"            → model → " Paris"
"The capital of France is Paris"      → model → "."
"The capital of France is Paris."     → model → (stop)

Everything else in this lesson follows from that loop.

Tokens: the pieces a model reads and writes

A model does not see letters or whole words. It sees tokens: common words, parts of words, spaces and punctuation, each mapped to a number. A rare word is split into several tokens; a common one is often a single token.

One way a model might split a sentence (each model has its own tokenizer, so the exact split varies):

Wind turbines convert moving air into electricity.
│Wind│ turb│ines│ convert│ moving│ air│ into│ electricity│.│

Why tokens matter to you:

• Limits are counted in tokens. The context window (below) and many API prices are per token.
• A rough rule for English: one token is about four characters, or three quarters of a word. DL Prism uses exactly this shortcut, text length divided by four, to estimate the size of a request before it is sent. See How Prism sees a request.
• Other languages and code often need more tokens for the same meaning.

LocalRAG also counts in tokens: it cuts documents into chunks of up to 500 tokens. You will meet chunks in lesson 3.

Predicting the next token

For every possible next token, the model produces a probability. For The capital of France is, the top of the list might look like this (illustrative numbers):

Next token	Probability
Paris	80%
a	11%
located	7%
Lyon	2%

The model then samples: it picks one token, usually a likely one, sometimes a less likely one. a is not wrong either; it might lead to "a city on the river Seine". Lyon would start a wrong answer, written as smoothly as the right one.

Temperature: how adventurous the choice is

Temperature is a setting that changes how the model picks from that list.

Next token	Temperature 0.2	Temperature 1.0	Temperature 1.5
Paris	over 99.9%	80%	65%
a	under 0.1%	11%	17%
located	under 0.1%	7%	12%
Lyon	under 0.1%	2%	6%

(Same illustrative example, recalculated for each temperature.) At 0.2 the answer is steady and repetitive; 1.0 uses the model's own odds; at 1.5 the choice is more varied, and more often goes off track.

• Low temperature (around 0 to 0.3) nearly always takes the top token. Use it for facts, extraction and code, where you want the same answer each time. LocalRAG asks Ollama for a temperature of 0.2 when it writes answers from your documents.
• Higher temperature (around 0.7 to 1.0 and up) spreads the choice out. Use it for brainstorming and creative writing.

Low temperature makes answers more consistent. It does not make them true: if the model's top guess is wrong, it will be wrong every time.

The context window: what the model can see

The model only knows two things when it answers:

1. What it learned in training, frozen at a training cut-off date.
2. The text in front of it right now: the context.

The context window is the most tokens the model can take in at once: your instructions, the conversation so far, any documents pasted in, and the answer it is writing. Windows range from a few thousand tokens to hundreds of thousands, depending on the model. In DL Prism's sample config, one local model is listed with a context_window of 128000 and another with 32768.

┌──────────────────────── context window ────────────────────────┐
│ instructions │ earlier messages │ pasted document │ question │ answer… │
└─────────────────────────────────────────────────────────────────┘
       anything that does not fit is cut off, or the request is refused

Three things surprise beginners:

• A model has no memory between requests. A chat app looks like it remembers because it sends the whole conversation again each time. LocalRAG does not do this, so every question there stands alone.
• Long contexts cost more and run slower, and a model can overlook a detail buried in the middle of a very long one.
• The program running a model may use a smaller window than the model's maximum to save memory.

Parameters and model sizes

A model is, at heart, a very large list of numbers called parameters (or weights). Training adjusts them until the model's guesses match real text well. A "7B" or "8B" model has about 7 or 8 billion of them; a "70B" model has about 70 billion.

Size	Rough parameter count	Typical place to run it
Small	1 to 4 billion	A laptop, even without a graphics card
Medium	7 to 14 billion	A desktop with 16 GB of memory, better with a graphics card
Large	30 to 70 billion	A high-end workstation with a lot of graphics memory
Very large	Hundreds of billions or more	A data centre, used through a cloud API

More parameters usually means more knowledge and better reasoning, and also more memory, more time per token and more cost. A newer small model can beat an older large one, and a model tuned for code can beat a bigger general model at code. Size is a hint, not a score. Lesson 2 turns parameter counts into memory you can check on your own computer.

Fluent and wrong: hallucination

Put the pieces together:

• the model writes whatever text is likely, one token at a time;
• it has no built-in way to check a fact against the world;
• it writes a wrong answer with the same confident grammar as a right one.

So it sometimes produces statements that sound right and are false: a made-up quote, a book that does not exist, a function a library never had, a wrong date. This is called hallucination. It happens most when:

• the question is about something rare, recent (after the training cut-off) or private, such as your own company's documents;
• the question assumes something false ("Why did Einstein win two Nobel prizes?");
• the model is asked for exact details: numbers, names, citations, version numbers.

Two habits help most:

1. Give the model the facts in its context, instead of hoping it remembers them. That is what retrieval-augmented generation does, in lesson 3.
2. Check anything that matters against a real source. A good tool shows you where the answer came from, as LocalRAG does with its page citations.

Stay safe

• Treat every answer as a draft. Check facts, numbers and code before you rely on them, and never act on medical, legal or money advice from a model alone.
• Remember the context is sent somewhere. Whatever you put in the prompt goes to whoever runs the model. With a cloud model, that is a company's servers. Lesson 6 covers how to keep secrets and personal data out.
• Do not ask a model to confirm itself. "Are you sure?" often makes it change a right answer or defend a wrong one. Check a source instead.

Check yourself

QuizHow language models work

Question 1 of 6

What does a language model actually do when it writes an answer?

Score: 0

What you learned

• A language model continues text one token at a time, choosing likely next tokens.
• Tokens are word pieces; limits and prices are counted in them. About four characters make a token in English.
• Temperature controls how adventurous the choice is; low is steadier, not more truthful.
• The context window is all the model can see; it has no memory between requests.
• Parameters are the model's learned numbers; more usually means more capable, slower and hungrier.
• Fluent text can be false. Give the model the facts, and check what matters.

Next steps

• Run models on your own computer
• LocalRAG overview: a model that answers only from your documents
• Glossary