On December 31, 2025, while much of the world was preparing for the new year,
the DeepSeek-AI team published a new paper on arXiv.
mHC
(Manifold-ConstrainedHyper-Connections)
Paper: https://www.arxiv.org/abs/2512.24880
It proposes a method for optimizing the macro-architecture of large-scale neural networks,particularly Transformer-based models.The work focuses on structural design.
Its objective is to addresstraining-stability and scalability problems in existing Hyper-Connections, or HC.。
After the New Year holiday, we reviewed the paper and a range of supporting material from beginning to end.
It is technically demanding work.After studying it in detail, one conclusion stood out:
rather than pursuing another increase in parameters or benchmark scores, the paper is concerned with a different question.
“How can a model continue operating reliably as it scales?”。
What problem does the paper address?
In one sentence, the paper is not primarily askingwhether：
- a model can become slightly more intelligent,
- whether inference can become slightly faster,
- or whether a benchmark position can improve by one place.
It concentrates on an engineering problem that every large-scale model eventually encounters:
“as the architecture expands and connections multiply, can information still move through the system in a stable and reliable way?？”
In existing Hyper-Connection architectures,
complex connectivity can cause signal intensity to become uncontrolled, creating visible training instability and reducing scalability.
The problem may not appear in smaller models,
but at large training scalesit becomes almost unavoidable.，
A precise instrument becomes harder to stabilize as the number of components and interactions grows.
The evolution of three network connection structures begins with standard residual connections in (a), moves to HC in (b), which introduces learnable mappings to expand information paths,and arrives at mHC in (c), which constrains the mapping space through manifold projection and substantially improves training stability.
The solution proposed by mHC is conceptually clear:
introduce manifold constraints and system-level optimization to govern connection strength and structure in a unified way,，
allowing HC to remain stable as it scales
and providing an efficient, stable, and scalable macro-connection architecture for large-model training.。
This is the kind of risk that is easy to underestimate
but difficult to avoid when building a real system.
01
A repeatedly underestimated question
Large language model capabilities have advanced rapidly over the past several years.
Models are larger, deeper, and built around more complex information flows.
At the same time, that growth amplifies a longstanding problem:
does information remain reliable as it travels through the model?
The problem can be summarized simply:
'As informationmovesthrough the network, it can become distorted, amplified, or disappear entirely.'
Neural-network engineering has several names for these failures:
vanishing gradients, exploding gradients, and unstable training.……
All are fundamentally failures in the information-transfer mechanism.
The industry has developed many ways to reduce these risks. One of the most important breakthroughs was
the residual connection.。
Its intuition is straightforward:
allow the original information tobypassintermediate layers and contribute directly to later computation.
That idea became a foundation of modern deep networks.
Since ResNet, residual connections have become standard in both convolutional networks and Transformers.
Their central advantage is the identity mapping,（identity mapping）
which gives information a dependable fallback path and prevents it from being completely distorted during transmission.
02
More Information, More Complexity
As models become larger and more complex, one direct information path is no longer sufficient.
That led to a more aggressive design:
Hyper-Connections.。
The concept can be understood asreplacing one direct route with several parallel paths for information exchange.。
The benefits are evident:
- greater information throughput,
- stronger representational capacity,
- and short-term improvements in model performance.
The design also introduces a distinctly practical engineering problem.
With multiple paths, two extreme behaviors can emerge:
- some signals are amplified repeatedly as they accumulate across paths,
- while responsibility for other signals becomes diluted until the information disappears.
Training then becomes highly unstable and may collapse at a particular stage.The problem is not whether the model is intelligent. The system is losing control as its structure becomes more complex.。
Training-stability comparison between HC and mHC
HC shows more pronounced loss shifts and gradient fluctuation during training,
while mHC maintains a smoother and more stable optimization trajectory throughout the training process.
One observation in the paper is particularly striking.：
In a standard HC architecture, loss unexpectedly spikes at roughly 12,000 training steps and the gradient norm begins fluctuating sharply.
This instability has a structural signature and results from a systematic weakness in the connection design itself.
03
What mHC doesis add constraints to freedom.
The core idea in DeepSeek's paper is disciplined rather than complicated.
The researchers do not reject Hyper-Connections,
and they do not retreat to a more conservative architecture.
They take a third approach:introduce strict mathematical constraints for multi-path information flow.。
The paper gives this mechanism a specific name:
Manifold-ConstrainedHyper-Connections
without focusing on the equations, its intuition can be described as follows.
- Information can continue moving through multiple paths,
- but every movement must satisfy conservation and traceability constraints.
- The system cannot amplify a signal without limit,
- and responsibility cannot be dispersed indefinitely.
Mathematically, this is expressed througha doubly stochastic matrixconstraint.
The paper uses the Sinkhorn-Knopp algorithm to project the connection matrix onto a specific manifold, the Birkhoff polytope.
Every row sums to one, and every column also sums to one.
This establishes traffic rules for information flow:
the total capacity of each route remains fixed, preventing one route from becoming overloaded while another is left unused.
When HC composes these transformations across a deep network,
the amplification of backward gradients can accumulate to as much as thousands of times the original magnitude.
That is one reason large-scale HC models are difficult to train reliably.
In engineering terms, mHC creates several direct improvements:
- signals are not amplified abnormally,
- information does not disappear silently between paths,
- and training fluctuations are substantially reduced.
The paper provides a dramatic but intuitive comparison.
In the worst case, unconstrained HC can amplify signal strength bynearly 3,000 times.；
With manifold constraints, mHC
limits maximum signal amplification toapproximately 1.6 times.。
This is not a parameter-level optimization. It is risk containment.
Like a precise power-delivery system, mHC keeps voltage within a safe operating range.
04
System-level insurance at a 6.7% performance cost
The engineering implementation of mHC is one of the paper's most impressive aspects.
The researchers provide not only a theoretical design but also a complete set of system-level optimizations.
1. Kernel fusion
combines several small operations into larger ones to reduce memory access.
For RMSNorm and high-dimensional hidden states in particular, reordering mathematically equivalent operations substantially improves efficiency.
2. Selective recomputation
addresses the additional intermediate activations introduced by mHC, which would otherwise increase GPU memory use significantly.
The team designed a block-based recomputation strategy:
store only the first-layer input for each computation block and recalculate other values during backpropagation as needed. A theoretical analysis identifies the optimal block size to minimize total memory use.
3. Overlapping communication and computation
is needed because mHC introduces additional cross-stage communication in distributed training.
The team extended the DualPipe scheduler so mHC computation overlaps with pipeline communication instead of leaving accelerators idle.
The additional computation path does not become a pipeline burden.
By extending DualPipe, operations such as F_post_res run in parallel with communication and primary computation,
keeping the overall increase in training time to approximately 6.7%.
At an expansion rate of n=4, these optimizations add only about 6.7% to training time.
That is a notable result.。
For a team training a frontier-scale model,
- one failed run can erase weeks of work and tens of millions in investment.
- Stability is itself an exceptionally valuable capability.
In essence, mHC exchanges a controlled additional cost, 6.7% more training time, for an order-of-magnitude reduction in systemic risk.
In any serious engineering system, that is cost-effective insurance.。
05
Experimental results
The paper evaluates models at 3B, 9B, and 27B parameter scales.
The results are persuasive.
Part1
Training stability improves substantially
mHC entirely avoids the loss spikes observed in HC, converges smoothly throughout training, and keeps gradient norms within a stable range.
Part2
Downstream performance improves broadly
Across eight established evaluations, including BBH, DROP, and MMLU, mHC outperforms the baseline in every case and exceeds the original HC on most tasks.
The improvement is particularly visible in reasoning, with BBH up 2.1% and DROP up 2.3%.
Part3
Scalability is validated
Across experiments from 3B to 27B parameters, mHC retains its performance advantage without losing effectiveness at the larger scale.
Comparison of downstream task performance
At 27B parameters, mHC outperforms the baseline on most tasks
and exceeds HC in overall performance.
06
The broader signal
The most important part of DeepSeek's paper
is not only mHC itself. It also indicates a clear direction:
competition among large models is moving away fromwhich model appears most intelligent，
and toward which model is more stable, controllable, and capable of long-running operation.。
As models take on real assignments, operate continuously, and participate in decisions,
- stability matters more than peak performance,
- effective constraints matter more than unconstrained freedom,
- and system consistency matters more than an occasional breakthrough.
That closely matches what we observe in real business systems.
Reliability is the first requirement for any system expected to operate around the clock.
AI models are moving from demonstrations into production tools.
That transition creates a new set of system-design requirements.
Final thoughts
Looking back, one change is already clear.
- Early large models were primarily demonstrations of capability.
- Current models increasingly resemble systems that must operate for long periods.
Systems engineering has never succeeded through inspiration alone. It depends on
- constraints:explicit boundaries and rules,
- stability:predictable behavior and output,
- and dependable operation:consistency even in complex environments.
This paper addresses a critical part of the system that has long received too little attention.
In our view,work of this kind is what determines whether AI can move from usable to sustainable.。
Technology will continue to evolve, but the systems that endure
will be those designed with respect for engineering constraints and real complexity.
DeepSeek begins 2026 with a paper about systems engineering capability.
While much of the market asks what a model can do, this work asks how the model can continue doing it reliably.
That may be one of the most important questions for AI in 2026.。
