Contents
A large language model, or LLM, is the kind of software behind chatbots such as ChatGPT, Gemini and Claude. It can draft an email, summarise a report or write computer code. Underneath, it does one thing: it looks at some text and works out what is likely to come next. The rest of this article builds up from that idea.
It predicts the next piece of text
Google’s machine learning course describes LLMs as “essentially autocomplete mechanisms”. Given the start of a passage, the model gives a probability to each possible continuation, picks one, adds it to the text and repeats. A question from a user is treated the same way. The model produces the text most likely to follow it, which is usually an answer.
The difference from the autocomplete on a phone is scale. A phone suggests a word or two. An LLM can continue for thousands of words and stay on topic while it does.
Text is broken into tokens
Models do not read text as whole words or single letters. They read tokens. Google’s introduction to language models says a token can be a word, part of a word or a single character, and that most modern models work mainly with parts of words.
Google’s developer guide says that for its Gemini models a token is about four characters, and 100 tokens come to roughly 60 to 80 English words. Other companies’ models split text a little differently, so these figures are rules of thumb. Tokens matter to users because the limits and prices of AI services are counted in them.
Training means adjusting billions of numbers
A model starts out knowing nothing. It is then shown an enormous amount of text, with some tokens hidden, and asked to guess what is missing. Each time it guesses, the error is measured and the model’s internal numbers are nudged so that the right answer becomes slightly more likely next time. Those internal numbers are called parameters. Nobody writes rules of grammar or lists of facts into the model. Whatever it knows is stored in the parameters as patterns picked up from the text.
The word “large” refers mainly to the number of parameters, and that number has grown very quickly.
| Model | Year described | Parameters |
|---|---|---|
| Original transformer, base version | 2017 | 65 million |
| Original transformer, big version | 2017 | 213 million |
| GPT-3 | 2020 | 175 billion |
The first two rows come from the 2017 paper that introduced the transformer. The third comes from OpenAI’s GPT-3 paper, which called it ten times larger than any comparable earlier model. Google’s course says transformer models can now contain hundreds of billions or even trillions of parameters. Most companies do not publish the sizes of their current models, so recent figures are hard to confirm.
Training at this scale is costly. The same course says it can take months and very large amounts of computing power and electricity.
| Item | Parameters, in millions |
|---|---|
| Original transformer, base, 2017 | 65 million |
| Original transformer, big, 2017 | 213 million |
| GPT-3, 2020 | 175 billion |
Most companies no longer publish the size of their models.Source: Vaswani and others (2017); Brown and others (2020). Chart: The Sidelight.
The transformer lets every word look at every other word
Almost all current LLMs are built on a design called the transformer. It was set out in June 2017 in a paper by eight researchers titled “Attention Is All You Need”. The paper was about machine translation. Its largest model was trained in three and a half days on eight graphics processors and set a new best score for a single model at translating English into French.
Earlier language software read a sentence one token at a time, which limited how much of the surrounding text it could take into account. The transformer dropped that approach and relied only on a technique called attention. For each token, the model weighs how much every other token in the passage should affect its meaning.
Google’s course gives an example. In the sentence “The animal didn’t cross the street because it was too tired”, the word “it” could refer to the animal or the street. Attention lets the model link “it” to “animal”. Change “tired” to “wide” and the link should shift to “street”. Stacking many layers of this process is what allows a model to follow long and complicated passages.
Nobody writes rules of grammar or lists of facts into the model. Whatever it knows is stored in the parameters as patterns picked up from the text.”
The context window is its working memory
A model can only consider a limited amount of text at once. This limit is the context window. Anthropic, which makes Claude, describes it as the model’s “working memory”. It covers everything in the conversation so far, any documents supplied and the reply being written. It is separate from the text the model was trained on.
The GPT-3 figure comes from OpenAI’s paper, and the other two from Google’s and Anthropic’s developer pages. A million tokens is several hundred thousand words, though the exact number depends on the model. A bigger window is not always better. Anthropic notes that accuracy and recall decline as the amount of text in the window grows.
Why large language models make mistakes
An LLM produces text that is probable, which is not the same as text that is true. When a model states something false in a fluent, confident way, the error is known as a hallucination. Google’s glossary defines it as “plausible-seeming but factually incorrect output”.
There are several reasons for these errors, and researchers do not fully agree on how far they can be removed.
- The training text contains errors. The UK’s National Cyber Security Centre points out that there is too much training data to filter completely, so inaccurate and offensive material gets in. It also warns that models can be biased and can be led by the way a question is asked.
- Its knowledge stops at a date. The same NCSC article notes that what a model holds is typically fixed once training ends. Unless it is connected to a search tool or given fresh documents, it knows nothing about later events.
- It is rewarded for guessing. A 2025 research paper argues that training and testing methods give models credit for a confident guess and none for admitting uncertainty, so they learn to guess. The authors propose changing how tests are scored.
- Appearing to reason is not reliable reasoning. Google’s course says a model that seems to do sophisticated mathematics is still completing text. It adds that researchers dispute how well models cope with problems unlike those in their training.
What this means in practice
An LLM is a powerful tool for working with language. It is not a database of checked facts. It is at its most dependable when it is given the source material and asked to summarise, rewrite or explain it. It is least dependable on niche subjects, recent events and exact figures. For anything that matters, check its claims against an original source.
References and sources
- 01Attention Is All You Need · Vaswani et al., arXiv · 12 June 2017
- 02LLMs: What’s a large language model? · Google Machine Learning Crash Course · Accessed 12 November 2026
- 03Introduction to large language models · Google Machine Learning Crash Course · Accessed 12 November 2026
- 04Understand and count tokens · Google AI for Developers · Accessed 12 November 2026
- 05Language Models are Few-Shot Learners · Brown et al., arXiv · 28 May 2020
- 06Context windows · Anthropic · Accessed 12 November 2026
- 07Machine Learning Glossary: Generative AI · Google for Developers · Accessed 12 November 2026
- 08ChatGPT and large language models: what’s the risk? · National Cyber Security Centre · 14 March 2023
- 09Why Language Models Hallucinate · Kalai, Nachum, Vempala and Zhang, arXiv · 4 September 2025
- 10Lead image: Data center infrastructure in the United States.jpg · US Department of Energy / National Renewable Energy Laboratory, via Wikimedia Commons (public domain) · Accessed 13 November 2026
House byline
Articles under this byline are researched and drafted with AI assistance, with every figure checked against its source before publication. The editor, Pankaj Guleria, reviews them and corrects them where needed.
Edited by Pankaj Guleria, Founder and Editor



