Attention Residuals: Rethinking Information Flow in LLMs

Standard residual connections with PreNorm accumulate every layer’s output with fixed unit weights, letting the hidden state grow uncontrollably with depth and diluting each layer’s unique contribution — what if each layer could instead learn which earlier representations to read from?

This paper introduces Attention Residuals (AttnRes), a novel architectural mechanism by the Kimi Team designed to rethink how information flows in modern Large Language Models. The central challenge is that standard residual connections with PreNorm accumulate all layer outputs using fixed unit weights; this uniform aggregation causes uncontrolled hidden-state growth with depth, progressively diluting each layer’s unique contribution.

To overcome this limitation, the authors replace the fixed accumulation with a softmax attention mechanism over preceding layer outputs, allowing each layer to selectively aggregate earlier representations using learned, input-dependent weights.

Join us at CASI for discussion at 8 pm, and (optional) quiet reading from 7 pm.