On Transformer Dynamics

We develop a geometric framework in which the token dynamics of a transformer are modeled by a system of interacting particles on a Riemannian manifold $\mathcal M$, the attention mechanism being encoded by a time-independent two-body interaction law, that is, a section of the pullback bundle $\pi_2^{*}(T\mathcal M)$ over $\mathcal M\times\mathcal M$.

Within this framework we isolate two features that a family of interaction laws must possess in order to model language: it must realize generic nonlocal and nonreciprocal forces, and it must parametrize vector fields on a high-dimensional manifold efficiently.

We show that both features are achieved simultaneously in a transformer model.

Our main theorem produces a finitely parametrized family of interaction laws, independent of the manifold and of its dimension, that is universal: it realizes an arbitrary prescribed attention digraph.

Moreover, we show that the cost of realizing a given attention digraph is governed not by $\dim\mathcal M$ but by two combinatorial invariants of the digraph, namely its biclique cover number, which we identify with the least number of hubs in a hub extension, and its hub-chromatic index.