PhyseaWiki How AI actually works physea.ai →

The transformer moment

Why did the Transformer change the course of AI?

Because the Transformer trained in parallel rather than step by step, researchers could build much larger models on much more text. That ability to scale, more than any single trick, set off the modern era of language models.

Last updated 2026-07-25 · Physea Labs

A faster way to train translation models does not sound like a turning point. The reason it became one is subtle. Older designs read text one step at a time, which meant the computer mostly waited on itself: step two could not start until step one finished. Even the recurrent models that added attention on top “still suffered from the same issue,” unable to run in parallel over a whole sequence and unable to take full advantage of GPUs as a result.[2] The Transformer’s attention looks at the whole input together, so the math spreads across many processors at once.[1] That single property removed the main bottleneck on size.

When a design trains in parallel, you can throw far more computing power and far more text at it without the training time becoming impossible. Researchers could now build models with many more internal settings, trained on a far larger slice of the written web. The lever stopped being a clever new architecture and started being scale.

The shift Before the Transformer, progress often meant a smarter model design. After it, much of the progress came from making the same design bigger and feeding it more data. The papers that followed, including BERT and GPT, were built on this foundation.

The original paper was about translation, and its authors could not have spelled out everything that would come next. But the architecture they described turned out to be general. Nearly every well-known language model since has been a Transformer of one kind or another. By 2026 the paper had been cited more than 250,000 times, placing it among the top ten most-cited papers of the 21st century.[2] That’s an unusual amount of influence for a paper whose stated subject was machine translation.

Older designs read step by step step two waits on step one Attention reads the whole input at once the math spreads across many processors Training parallelizes, the size ceiling lifts The lever becomes scale, not cleverness
The paper was about translation. What it actually removed was the ceiling on model size, and scale is what the field spent the next several years exploiting.

References

  1. Attention Is All You Need — arXiv (Vaswani et al., 2017)
  2. Attention Is All You Need — Wikipedia

Related