Manifold-Constrained Hyper-Connections

How does constraining residual connections to a geometric manifold restore identity mappings and enable stable scaling in large language models? Can richer cross-layer connectivity improve reasoning and representation quality without sacrificing optimization stability?

DeepSeek’s Manifold-Constrained Hyper-Connections (mHC) introduce a principled fix to recent attempts at widening residual streams. While Hyper-Connections improve performance by allowing richer cross-layer mixing, their unconstrained residual matrices break the identity-mapping property that makes deep networks trainable, leading to severe instability at scale. mHC resolves this by projecting residual mixing matrices onto the manifold of doubly stochastic matrices, ensuring residual updates remain convex combinations of features and preserving signal magnitude across depth.

Empirically, this constraint dramatically stabilizes training in large language models, eliminating gradient explosions observed in unconstrained HC while retaining its performance gains. With careful systems co-design—kernel fusion, recomputation, and pipeline overlap—mHC incurs only ~6–7% overhead at 27B scale, while consistently improving downstream reasoning benchmarks. The result reframes architectural scaling as a geometric constraint problem, showing that richer connectivity can be safely exploited when paired with mathematically grounded structure rather than ad-hoc regularization.

Join us at CASI for discussion at 8 pm, and (optional) quiet reading from 7 pm.