Types of models
What is a large language model as a type, and what is it good and bad at?
A large language model is trained to predict the next token of text, then generates by doing that over and over, feeding each new token back in. The approach makes it strong at writing, summarising, and pattern-following across language, and weak at exact arithmetic, fresh facts, and knowing when it is wrong.
A large language model is the type most people mean when they say AI. It reads text and writes text, and underneath it is doing one thing: predicting the next token. A token is a chunk of text, often a word or part of a word. The model outputs one token at a time, then adds that token to its input and predicts the next, a loop called autoregression[1]. Run the loop enough times and you get a sentence, a paragraph, or a page.
That single objective, next-token prediction, is the whole reason an LLM behaves the way it does. It has read an enormous amount of text and learned the statistical shape of language: how arguments are structured, how code is written, how a polite email reads. So it is strong wherever the task is really a language-patterning task. Drafting, rewriting, summarising, translating, explaining, turning messy notes into clean prose, following a format you show it. These play to its training.
The architecture under almost every modern LLM is the transformer, introduced in 2017, which lets the model weigh every part of the input against every other part as it decides what comes next[2]. That is why it can keep track of context across a long passage rather than just the last few words.
The weaknesses come from the same place. The model is trained to produce likely text, not true text, so it can write a fluent, confident answer that is simply wrong, and it has no built-in sense of when that has happened. It is unreliable at exact arithmetic and counting, because those are not really language patterns. It only knows what was in its training data, so it is blind to anything that happened afterwards unless you give it the information. Knowing this shapes how you use one: lean on it for language work, and supply facts, tools, or a way to check whenever being exactly right matters.
That last weakness is the one that causes the most damage in practice, because it doesn’t announce itself. A model that is bad at arithmetic at least produces an answer you know to double-check. A model that is confidently, fluently wrong about a fact looks identical to one that is confidently, fluently right, which is why grounding it in something you can check matters more than raw capability once the cost of being wrong goes up.
References
- The Illustrated GPT-2 (Visualizing Transformer Language Models) — Jay Alammar
- Attention Is All You Need — Vaswani et al. (arXiv)
Common questions
- What does autoregressive mean?
- It means the model generates one token at a time, and each token it produces is added back to the input before it predicts the next one. The output is built left to right, one step feeding the next.
- Why do language models make things up?
- An LLM is trained to produce fluent, likely-looking text, not to check facts. When it has no reliable pattern to draw on it still produces something plausible, which is why fluent wrong answers are a normal failure mode rather than a bug.