How Transformers Know Word Order When Attention Doesn't Care

Here's a fact that surprised me the first time it clicked: self-attention has no idea what order your tokens are in. Feed it "the cat sat" and feed it "sat cat the", and it produces the same set of representations, just shuffled to match. That's because attention builds each token's output as a weighted sum over every other token, and the weights come purely from query-key dot products of the contents . Nothing in that math mentions position. Permute the inputs, and the outputs permute right…