What is Attention Mechanism?
Introduction to the Attention Mechanism
Before 2017, translating a paragraph from English to French with AI relied mainly on sequential models that were slow to train and struggled with long sentences. The standard approach used Recurrent Neural Networks (RNNs), including LSTMs. RNNs read sentences sequentially, one word at a time, from left to right.
In a long sentence, information about early words tends to fade by the time the network reaches later words. If an early word was “bank” and a much later word was “river,” the model could fail to connect them and miss that “bank” meant the side of a river rather than a financial institution.
Attention mechanisms were first added to RNN-based translation models around 2014 to help with this problem. The 2017 Google paper “Attention Is All You Need” went further: it introduced the Transformer, an architecture built on Self-Attention without recurrence. Self-attention is the core component that makes Transformers (and Large Language Models like GPT-4) possible.
How Self-Attention Works
Self-Attention does not process words one after another. It processes all the tokens in a sequence together.
Its goal is to work out how relevant each token is to every other token in the same sequence, producing a set of weighted connections between them.
Consider the sentence: “The animal didn’t cross the street because it was too tired.” What does the word “it” refer to? The animal, or the street?
When the Transformer processes the word “it”, the Self-Attention mechanism compares it with every other word in the sentence. It does this using learned projections called Query, Key, and Value. The result is a set of attention weights over the other words. In an illustrative example:
- Most of the attention goes to “animal”.
- A small share goes to “street”.
- Some goes to “tired”.
From these weights, the model’s representation of “it” draws mostly on “animal,” which is how it resolves the reference.
The Power of Parallelization
Because self-attention processes all tokens at once during training (rather than waiting for word #1 before handling word #2), it is well suited to parallel computation.
This is a major reason the AI boom is closely tied to GPUs. GPUs are designed to run many matrix multiplications in parallel, and self-attention turns much of language modeling into large matrix multiplications. This made it practical to train neural networks with hundreds of billions of parameters.
Multi-Head Attention
Transformers do not use a single attention calculation; they use Multi-Head Attention. Each layer runs several attention “heads” in parallel, each with its own learned projections. The largest GPT-3 model, for example, used 96 heads per layer.
- One head might learn to track syntactic relationships, such as which verb goes with which subject.
- Another might learn to follow references, such as which noun a pronoun points to.
- Others capture patterns that are harder for people to label.
By combining the outputs of these heads, the network builds a richer representation of the text than a single attention calculation could.
Conclusion
The Attention Mechanism is one of the most important developments in modern AI. By letting a model weigh the most relevant parts of its input while giving less weight to the rest, it addressed the long-range memory problems of earlier sequential networks. It is the central component of the Transformer architecture and a key reason for the language fluency of modern Generative AI.
Learn More
To learn more about the Data Lakehouse, read The Lakehouse Built for Everyone by Alex Merced. You can find this and other books by Alex Merced at books.alexmerced.com.