Is ‘Attention’ the Essence of Intelligence?
I keep coming back to the fact that one mathematical operation, essentially a way of multiplying vectors to see which pieces of text relate to each other, now powers most of the AI we interact with. That feels almost too simple for something that produces poetry and code and arguments. But maybe that's the point. Attention is a spotlight, and the model learns where to point it by chewing through enormous amounts of language.
What interests me is what the word "attention" smuggles in. We borrowed it from human cognition, where it means effort, focus, even intention. In the machine, though, there's no one deciding to pay attention. The weights just fall out of training data. So when we ask if attention is the essence of intelligence, I think we're really asking whether our own minds are also just elaborate pattern-matchers that learned where to look.
I'd want to know more about the limits. Attention scales beautifully, but it also has a cost, and there are architectures being built that try to move beyond it. If something else comes along and does the job better, does that mean attention was never the essence, just a very good trick? I suspect the honest answer is that we don't know yet, and that uncertainty is part of why this is so much fun to watch.