DeepSeek V4 Technical Report: First Impressions

DeepSeek V4’s technical report had just been released, and I opened it immediately. The amount and breadth of information in it, and the technical depth, are vast.

My feeling is that after several hours I had probably read only 20% of it; of that 20%, I may have understood only another 20%. Fully understanding it will take a long time.

But the value of news is that it is new. I really could not wait until tomorrow to write this up, so I am writing down all my current impressions first.

Everything below is a hot take. The value of a hot take is not that it is very accurate. In fact, I think only 5% to 10% of what I say here may be right. But because it is non-consensus, its value has not yet been discovered, so it may still be interesting to hear.

1. DeepSeek may not be great at alchemy

The first hot take I want to make is this: DeepSeek may not be as strong as Kimi at “alchemy.”

Here are a few choices in the technical report that I found very strange.

The first puzzling move is the “m” in mHC. The mHC paper has been out for a few months. The idea behind HC is actually quite simple: widen the residual stream, so that layers can pass information forward in arbitrary forms, thereby greatly increasing the expressive power of the model. This part itself is fine.

The problem is the manifold treatment. DeepSeek found that because HC does not constrain changes in scale during the forward pass, it can lead to exploding or vanishing gradients, so some method is needed to constrain the scale. In the end they used a method called manifold, which introduces Sinkhorn.

But this Sinkhorn procedure has quite a few issues. First, it is not very “beautiful.” As a later paper, MHC Lite, points out, on ill-conditioned matrices, even after 20 iterations, Sinkhorn may not converge to a good probability distribution close to the ground truth. That alone introduces a nontrivial error.

At the same time, the algorithm itself is relatively slow. The original MHC paper says it adds about 4% compute time, which is not a small cost.

More importantly, because “alchemy” itself is basically mysticism, any extra complexity makes debugging extremely difficult. On the other hand, neural networks have a very strong ability to discover patterns automatically. In many cases, you do not need to manually give them overly strong or overly complex expressive structures. Instead, as long as you give them enough capacity, they will naturally learn all sorts of representations through combinations of different matrices.

Later, someone on Zhihu proposed a cleaner architecture called Identity HyperConnection. The core idea is: even if the hidden state becomes wider, the forward pass still preserves identity. This looks much more natural, and much closer to Kaiming He’s original design philosophy for residual networks.

In other words, we only need to give the model a larger space for forward information flow. We do not need to manually prescribe the semantic transformations between these forward paths. Those semantics do not need to be hand-designed. You only need to give it a wider “writing space,” and then let each layer decide what to write at each position. That is enough.

From this perspective, introducing MHC adds a lot of unnecessary complexity to the whole system.

The second strange point is the tuning of Muon.

In Muon, the principle behind tuning the Newton-Schulz iteration has already been explained very clearly in the PolarExpress paper. There is a relatively clear, even almost “optimal,” way to tune it.

So unless you believe low-precision training has a large effect on this kind of iteration, using two different sets of coefficients for the first two terms seems somewhat unjustified.

I also do not know what experiments led to this setting, or whether V4’s training actually happened at a relatively early stage, so that many later techniques that had already been verified as better were not used.

The third strange thing is its treatment of MoE.

First, it says that when people normally train MoE, the router uses either softmax or sigmoid, while here they chose a different path: the square root of softplus.

I have long had a conjecture about the optimal activation function for MoE: perhaps the best one should be soft top-k. This is also recorded somewhere on kexue.fm.

But square root of softplus is really hard to understand. What exactly is the intuition behind it?

And if you only change this one local design, the gain looks limited, while the added complexity to the whole system is not small. Whether this trade-off is meaningful is worth doubting.

Let us keep going.

The report mentions that during pretraining, training would collapse. They searched for the cause for a long time and eventually attributed it to MoE, saying that the interplay between MoE and the router would lead to certain extremely large values. Then they proposed a fix. Just hearing it sounds a little incredible: if a loss spike appears, roll back and continue training using the routing decisions from a previous checkpoint.

The problem is that to obtain these routing decisions, you need to do one extra prefill pass, and this extra cost is around 20%.

First, by 2026, it is already a little strange to still have MoE training collapse, especially with obvious loss spikes. Second, after training collapses, fixing it this way is not cheap, especially with this 20% overhead. The whole report is describing almost god-tier systems design. Can you really claw back more than 10% efficiency from somewhere else? I am honestly a bit skeptical.

Of course, this mechanism only triggers when something goes wrong, so the average cost may be controllable. But the bigger issue is that the loss spike likely points to a more fundamental training instability, rather than a local problem that can be simply patched.

One could even speculate one step further: could the timing of V4’s release have been delayed partly because of this problem?

Here is my conjecture: is it possible that the MoE training collapse is precisely because of the square root of softplus mentioned above?

Softplus is a continuous version of ReLU, specifically log(1 + e^x). Even if you wrap a square root around it, its value is still unbounded. That means if my logit increases, the gate value also increases, without an upper bound.

This is different from traditional sigmoid and softmax, and also different from my conjectured soft top-k.

What kind of problem might this create? When the gate is unbounded, if a certain expert really “likes” this hidden state, then even late in training, positive gradient will keep flowing in to further raise this logit. The gradient may therefore keep accumulating until the router blows up.

Of course, all of this is just conjecture.

These are the strange choices I noticed.

Of course, since this is mysticism, I cannot claim to understand the principles behind all of it. But looking at the overall style of work, dealing with mysticism should be completely different from dealing with systems, which is what DeepSeek is good at.

What is the difference?

For systems problems, complexity is often additive and linear. It is controllable. If I change a kernel in the front, it will not affect some later operation. At the same time, the improvements from most optimizations are predictable. For example, if I write a mega-kernel, before writing it I can roughly judge how much benefit I will get from pushing communication and computation overlap to the limit.

But since alchemy is mysticism:

First, its risk is uncontrollable and unpredictable. Every change may be good, or it may be bad. Only by running a large number of experiments can you slowly develop a bit of sense for how to handle it.

At the same time, the risk is not isolated. If I change attention in the front, it may affect the MoE later. If I change the hidden state, or some router, it may affect Muon’s computation. Everything is interplay.

And once you cannot isolate risk, the risk grows exponentially with the number of optimizations.

How do you handle these possible changes? The only method is ablation. But ablation quickly becomes a combinatorial problem. Fundamentally, the only way to guarantee a good method is grid search, and the number of experiments required by grid search also grows exponentially with the number of optimizations.

Therefore, if you have enough respect for mysticism, each generation of model should use at most three new techniques. With three new techniques, grid search is cubic in n, which is still controllable. Any more than that, I think you cannot afford it no matter how many GPUs you have.

So does this suggest that DeepSeek’s engineering background somehow determines that it does not respect mysticism enough?

This reminds me of something. Earlier, when I was writing the Expert Threshold article on MoE, both the writing and the experiments drew heavily from DeepSeek’s very famous Loss-Free Load Balancing paper. In my view, that paper was the most important MoE paper of 2024-2025. You could even say it was the only meaningful architecture-change paper in that period.

My thought at the time was: since this is the best one, I can just follow it. For example, it only reported perplexity and did not run all kinds of benchmarks like normal pretraining papers do. I thought, if DeepSeek does it this way, then I can be lazy too; I even tested one more thing than they did, DataComp.

The figures in Loss-Free were basically drawn casually. I still put a little care into mine. Of course, they were not as polished as figures in other accepted papers, but I thought mine were at least a bit better than DeepSeek’s. That should be fine.

Then the reviews came back, and the scores were extremely low. I was confused and showed them to my advisor. My advisor asked which paper I had used as reference. I said DeepSeek.

He told me to find the OpenReview page for Loss-Free. I opened it, and it had three rejects, even lower than mine.

My advisor looked at the paper and gave the following evaluation:

No matter how important the technique is, from the perspective of academic writing, the paper is simply unqualified.

First, there are even grammar mistakes in the paper. Second, the figures are very casual. Third, the experiments that should have been done were not done.

So even if it is a very important technique, it deserved to be rejected. Its influence mainly comes from DeepSeek. If V3 had not used it, it would not later have become a famous paper.

That was how I understood the matter at the time.

Looking back, this may precisely mark a certain lack of respect DeepSeek has for “mysticism.”

Why do the top ML conferences need so many “stories”? Why so many rules, ablation studies, and extra experiments? Precisely because when facing mysticism, you must be extremely rigorous and extremely cautious. You need to do many extra things before you can establish even a little confidence in whether a technique is effective.

But DeepSeek’s paper did not write these things. Perhaps that is because it does not care.

The Loss-Free paper may have “hit the target” because its intuition was too good. But if the same attitude is used for later mysticism research, many mistakes may follow.

So, about this alchemy thing: even though we mock it every day, as a large model company you still need people inside who have that god-given touch.

For example, if Kaiming He were at DeepSeek, I do not think he would allow the “m” in mHC to be published as the final solution. He would definitely run a huge number of ablations here and finally reduce this m to the simplest possible form before being satisfied.

Only when you reduce it to the simplest form can you be sure that its risk is controllable: that it will not significantly increase system complexity, and that it will not collapse during large-scale model training.

Yang Zhilin wrote papers with more than ten thousand citations during his PhD, so he clearly received systematic training in alchemy. Never mind whether the “storytelling” and all the theories in those papers are actually reasonable. In practice, there really is a difference between judgments made by people with experience and people without that touch.

In even more extreme cases, people like Alec Radford or Noam Shazeer have an even stronger touch, and that itself is a talent. Sometimes they may not be able to fully explain why their architecture works, but the result is that it works.

So I arrive at the hot take: DeepSeek may be missing such people. At the same time, should its organizational style make more conservative engineering choices for this kind of “mysticism,” rather than being as aggressive as it is in systems optimization?

2. The missing product side

On the product side, before DeepSeek V4 was released, I predicted that it would have a lot of black magic in infrastructure and algorithms, but as a model, as a product, it would not perform extremely well, or at least would not be better than Kimi, Zhipu, and MiniMax.

Here is how I understand it: products need user feedback. They are the result of continuous iteration.

For example, why is Anthropic doing well? Because it first built Claude Code. The agent itself is nothing magical, right? I think all agent harnesses are nothing magical. The agent wrapper itself does not bring any magical experience.

But in the process of using this model, Anthropic obtained a lot of user feedback, a lot of data, and a lot of practical insight. Then it could bring these insights back to the training team and say: we need to run post-training specifically for this, run RL, do human labeling, collect more data. Only after this kind of training can the model truly learn how to use this agent system.

In other words, this is an iterative process. Start with 3.7, build Claude Code on top of 3.7, then use the experience from Claude Code to train 4. After 4 is trained, use its usage experience to train 4.5 Sonnet. Then Sonnet trains 4.5 Opus, then 4.6, 4.7, and so on. Even Anthropic went through at least four or five rounds of model iteration and agent-product co-design and co-optimization before it finally became a useful product.

Kimi, MiniMax, and Zhipu all also care a lot about coding. Internally, they are iterating on all kinds of products every day.

Just take MiniMax. From 2.0 to 2.1, 2.2, 2.5, and then today’s 2.7, that is already many rounds of iteration.

If DeepSeek really cared about this, then we would not only see a 3.2. We should at least have seen 3.25, 3.28, 3.3, right?

3. Learning Curve

Ever since ChatGPT appeared, I have been thinking about one question: how should we understand the dynamics of commercial competition in large models? What standard should we use to judge whether a company will succeed or fail?

For example, the previous era was the internet era. In the internet era, a company’s success fundamentally came from scale effects, platform effects, or network effects.

In other words, if your product is good, it first spreads virally. After you have a sufficiently large user base, the value of the whole platform increases as the number of users increases.

It is like WeChat: if everyone uses WeChat, then even if you do not like WeChat, you still have to use it.

This dynamic determines the division of labor in that system: why lean startup was needed, why product managers were needed, why iteration cycles were fast, and where those high margins came from.

However, so far, large models have not shown obvious platform effects. The most intuitive sign is that user loyalty is basically zero.

For example, if Anthropic releases a new model, I may go use Anthropic. If OpenAI releases a new model, I switch to Codex. Users have no stickiness.

Therefore, the evaluation standards of the internet era do not apply in the LLM era.

So what standard should we use to think about the essence of commercial competition?

I increasingly feel that the learning curve can explain it.

I first saw the idea of the learning curve from TSMC founder Morris Chang. He attached great importance to the learning curve and applied it to every aspect of running his business.

Specifically, the learning curve itself is simple: the more you do something, the better you get at it.

But many people have not thought through the consequences behind it.

Suppose I have a product. The more I sell, the better I become at making it. The better I become, the lower my cost and the higher my quality. In semiconductors, quality here usually means yield. If my quality is high and my cost is low, customers naturally like me and place more orders. Those additional orders then give me more opportunities to practice, and my cost continues to fall.

When enough time passes, this flywheel enters a positive cycle, and eventually my advantage becomes something other competitors cannot match.

The most obvious example is TSMC. When it began preparations in 1985 and was formally founded in 1987, TSMC’s technology was not leading. At the time there were at least dozens of competitors, and many semiconductor companies did not use fabs but manufactured chips themselves.

By the 1990s or 2000s, perhaps only about five competitors were left. By the 2010s, basically only Samsung and Intel remained as competitors. Recently, entering the 2 nm era, Samsung has gradually fallen behind too. So TSMC has become almost the only choice for all chip design companies, and naturally its gross margin can be very high.

However, the cycle required to build this advantage is far longer than the monopoly produced by scale effects in the internet era. As I said above, it took TSMC a full 40 years to reach today’s dominant position. Nvidia’s position in graphics cards also took roughly 30 years.

So the essence is really endurance: endure continuously for decades, outlast all competitors, and naturally become the winner.

My guess is that the development of large language models will also follow the law of the learning curve.

Concretely, we can view training as R&D and inference as production. I invest a certain amount in R&D, then I go produce, and this brings me profit and revenue. I then take part of that revenue and put it back into R&D.

In that case, one very important key determining a company’s eventual development is revenue.

For example, if, like Anthropic and OpenAI, my annual revenue this year can reach tens or hundreds of billions of dollars, then I can take, say, 10% of that for R&D, which is already billions or tens of billions of dollars in R&D.

If you are a small company with only a few billion dollars in revenue, and you also take out 10%, you are left with only a few hundred million dollars. It becomes very hard to compete with larger firms, even if your gross margin is similar.

So when is victory decided?

When your revenue is large enough to squeeze out all competitors, while the R&D cost of each generation of model keeps rising until other competitors and smaller firms cannot bear it. Then only you are left. It is like TSMC today: developing a new process generation costs tens of billions of dollars, and even Samsung and Intel cannot easily put up that money. Or even if they do put up the money, because their sales volume is not enough, the investment is bound to lose money.

Only under these conditions can you win normal competition.

OpenAI’s R&D spending last year was nine billion dollars. I do not know about this year; it may keep rising. When it rises to the point that other companies cannot bear it, perhaps the winner will be decided.

Of course, several factors make the matter less simple.

The first is the existence of external capital.

Today, no company funds R&D purely from profits out of revenue. Everyone uses venture capital money.

And venture capital money is somewhat arbitrary. Suppose my company has no accumulated advantage at all. Through enormous investment, external investment, can I close part of the gap?

Of course, maybe this is not exactly venture capital. It is more like the behavior of large companies, such as Xiaomi, Meituan, and xAI, though xAI now seems to have other problems.

Second, so far, when we observe Chinese companies, they can still use only a few hundred million dollars of R&D cost per year to reach something close to what OpenAI achieves with ten billion dollars of R&D cost.

Of course there is a lag, roughly three to six months.

When competitors spend a lot of money training a new model, I can distill it.

Since this gap can save dozens of times the money, are there still some subtleties here that need to be considered?

If my hypothesis is true, then from the perspective of the learning curve, we can look back at DeepSeek.

Perhaps we can make this conjecture: DeepSeek not raising capital last year may have been a huge mistake.

Without financing, you cannot expand capacity. Without capacity expansion, you do not have users, and therefore you do not have revenue. Without revenue, sure, for now Liang Wenfeng is paying out of his own pocket, but if in the future you have no revenue, you will not have enough money for R&D.

No matter how good your culture is or how strong your team is, if competitors use ten times the money to poach your people, they will eventually poach quite a few of them, right?

Recently there have also been rumors that several core collaborators have already gone to ByteDance, while others have gone to Xiaomi and Tencent.

Of course, Liang Wenfeng’s problem is that, at least in 2025, financing could not be converted into compute.

DeepSeek also could not smuggle in a large number of Nvidia GPUs the way ByteDance could.

At that time, Huawei cards were probably also very hard to use. Buying them would not have helped much.

At the same time, there is another difference in large model R&D: if your headcount crosses the threshold of three to five hundred people, the efficiency of the whole organization drops sharply, because communication costs rise. The organization must also have structure, which means all kinds of bureaucracy and similar things will appear.

In other words, the ideal team size, even if you have infinite money, may still be three to five hundred people.

Therefore, hiring more people is not very meaningful. But if you raise more money and give current researchers higher salaries, does that help? It probably still helps.

The learning curve also determines the product, or rather the customer relationship. Serving customers is the key indicator that determines whether even a pure technology company can develop in the future.

For example, TSMC’s culture strongly emphasizes customer priority. Customer priority comes before technology leadership. That is, it first satisfied the requirement of customer priority. Only then did it have enough revenue, enough gross profit, and enough money to spend on R&D.

Then, when your R&D budget is large enough, you can achieve technology leadership.

So TSMC achieved technology leadership only around 7 nm in 2018, 31 years after it was founded.

DeepSeek seems to focus purely on technology and not pay enough attention to demands such as coding agents, various strange agents, or user requests for multimodality and things like that.

What worries me more is whether it has failed to establish this kind of culture, and whether the whole organization does not value this matter.

Then can it really win this marathon?

DeepSeek V4’s own official account ended with this sentence:

“Not tempted by praise, not afraid of criticism; follow the way and hold oneself upright. We will always uphold the principle and philosophy of long-termism, move forward steadily through attempts and reflection, and keep working toward the goal of achieving AGI.”

What I want to emphasize here is this counterintuitive judgment: is persisting in technological exploration truly long-termism?

If you believe in the learning curve, then serving customers, expanding revenue as much as possible while maintaining and pursuing technology leadership, is the correct approach. Serving customers comes before technology leadership.

What kind of company does DeepSeek want to become? Does it want to become Bell Labs? Did Bell Labs survive in the end? Or does it want to become Fairchild, or one of those companies in semiconductor history that were technologically leading but ultimately disappeared?

Everything above is purely personal speculation. Please just treat it as me talking nonsense.