A large language model, or LLM, is the kind of software behind chatbots such as ChatGPT, Gemini and Claude. It can draft an email, summarise a report or write computer code. Underneath, it does one thing: it looks at some text and works out what is likely to come next. The rest of this article builds up from that idea.

It predicts the next piece of text

Google’s machine learning course describes LLMs as “essentially autocomplete mechanisms”. Given the start of a passage, the model gives a probability to each possible continuation, picks one, adds it to the text and repeats. A question from a user is treated the same way. The model produces the text most likely to follow it, which is usually an answer.

The difference from the autocomplete on a phone is scale. A phone suggests a word or two. An LLM can continue for thousands of words and stay on topic while it does.

Text is broken into tokens

Models do not read text as whole words or single letters. They read tokens. Google’s introduction to language models says a token can be a word, part of a word or a single character, and that most modern models work mainly with parts of words.

Google’s developer guide says that for its Gemini models a token is about four characters, and 100 tokens come to roughly 60 to 80 English words. Other companies’ models split text a little differently, so these figures are rules of thumb. Tokens matter to users because the limits and prices of AI services are counted in them.

Training means adjusting billions of numbers

A model starts out knowing nothing. It is then shown an enormous amount of text, with some tokens hidden, and asked to guess what is missing. Each time it guesses, the error is measured and the model’s internal numbers are nudged so that the right answer becomes slightly more likely next time. Those internal numbers are called parameters. Nobody writes rules of grammar or lists of facts into the model. Whatever it knows is stored in the parameters as patterns picked up from the text.

The word “large” refers mainly to the number of parameters, and that number has grown very quickly.

ModelYear describedParameters
Original transformer, base version201765 million
Original transformer, big version2017213 million
GPT-32020175 billion
How parameter counts have grown

The first two rows come from the 2017 paper that introduced the transformer. The third comes from OpenAI’s GPT-3 paper, which called it ten times larger than any comparable earlier model. Google’s course says transformer models can now contain hundreds of billions or even trillions of parameters. Most companies do not publish the sizes of their current models, so recent figures are hard to confirm.

Training at this scale is costly. The same course says it can take months and very large amounts of computing power and electricity.

How the size of language models grewParameters, in millions
How the size of language models grew
ItemParameters, in millions
Original transformer, base, 201765 million
Original transformer, big, 2017213 million
GPT-3, 2020175 billion

Most companies no longer publish the size of their models.Source: Vaswani and others (2017); Brown and others (2020). Chart: The Sidelight.

The transformer lets every word look at every other word

Almost all current LLMs are built on a design called the transformer. It was set out in June 2017 in a paper by eight researchers titled “Attention Is All You Need”. The paper was about machine translation. Its largest model was trained in three and a half days on eight graphics processors and set a new best score for a single model at translating English into French.

Earlier language software read a sentence one token at a time, which limited how much of the surrounding text it could take into account. The transformer dropped that approach and relied only on a technique called attention. For each token, the model weighs how much every other token in the passage should affect its meaning.

Google’s course gives an example. In the sentence “The animal didn’t cross the street because it was too tired”, the word “it” could refer to the animal or the street. Attention lets the model link “it” to “animal”. Change “tired” to “wide” and the link should shift to “street”. Stacking many layers of this process is what allows a model to follow long and complicated passages.

Nobody writes rules of grammar or lists of facts into the model. Whatever it knows is stored in the parameters as patterns picked up from the text.”

The context window is its working memory

A model can only consider a limited amount of text at once. This limit is the context window. Anthropic, which makes Claude, describes it as the model’s “working memory”. It covers everything in the conversation so far, any documents supplied and the reply being written. It is separate from the text the model was trained on.

The GPT-3 figure comes from OpenAI’s paper, and the other two from Google’s and Anthropic’s developer pages. A million tokens is several hundred thousand words, though the exact number depends on the model. A bigger window is not always better. Anthropic notes that accuracy and recall decline as the amount of text in the window grows.

Why large language models make mistakes

An LLM produces text that is probable, which is not the same as text that is true. When a model states something false in a fluent, confident way, the error is known as a hallucination. Google’s glossary defines it as “plausible-seeming but factually incorrect output”.

There are several reasons for these errors, and researchers do not fully agree on how far they can be removed.

  • The training text contains errors. The UK’s National Cyber Security Centre points out that there is too much training data to filter completely, so inaccurate and offensive material gets in. It also warns that models can be biased and can be led by the way a question is asked.
  • Its knowledge stops at a date. The same NCSC article notes that what a model holds is typically fixed once training ends. Unless it is connected to a search tool or given fresh documents, it knows nothing about later events.
  • It is rewarded for guessing. A 2025 research paper argues that training and testing methods give models credit for a confident guess and none for admitting uncertainty, so they learn to guess. The authors propose changing how tests are scored.
  • Appearing to reason is not reliable reasoning. Google’s course says a model that seems to do sophisticated mathematics is still completing text. It adds that researchers dispute how well models cope with problems unlike those in their training.

What this means in practice

An LLM is a powerful tool for working with language. It is not a database of checked facts. It is at its most dependable when it is given the source material and asked to summarise, rewrite or explain it. It is least dependable on niche subjects, recent events and exact figures. For anything that matters, check its claims against an original source.