This series will explore the business models, competitive dynamics, and pricing strategies of LLMs. I’ll also make some predictions that readers can test against in the future.
So, how should we understand the competitive logic of LLMs? I believe we can apply the logic of the semiconductor industry. Training a model is like R&D in semiconductors. Inference is like manufacturing.
How does the semiconductor industry determine winners and losers? The core principle is the learning curve. I briefly touched on this in my V4 post. Now I’ll explain it fully.
The logic of the learning curve is simple. The more you do something, the better you get at it. But the real competitive implication is something fewer people think about. Because the more you do it, the better you get. And the better you get, the better your models and products become.
Meanwhile, costs also come down. So users prefer your product. More users means more revenue. More revenue means more R&D funding. Which lets you develop the next, even better product. A virtuous cycle.
Maintain that cycle for decades, and competitors drop off one by one. They lose the ability to compete in the next generation.
Over time, specialization deepens. The number of competitors naturally shrinks. Because each generation costs more to develop. Capital and talent concentrate into fewer hands.
Once most companies realize that no matter how much they spend on R&D, they can’t win meaningful market share or amortize those costs, they naturally exit.
Take chip manufacturing. When TSMC was founded in 1987, the dedicated foundry model barely existed. Most semiconductor companies designed and manufactured their own chips. So there were many capable manufacturers — possibly hundreds.
But as process nodes shrank and technical difficulty increased, by around 2000, only about a dozen could produce advanced chips. By 2010, that number fell to four or five.
TSMC mass-produced 7nm in 2018, then introduced EUV at the N7+ node. At that point, only Samsung and Intel still had a shot at keeping up.
A few more years passed. At the 2nm node, the fixed investment for each generation exceeds $30 billion. Even Intel and Samsung couldn’t sustain it. And so we arrive at today’s TSMC-dominant landscape.
This pattern applies to virtually all mass-market products. GPUs, for example, are dominated by NVIDIA. TPUs and AMD take second and third place. And anything behind AMD is essentially negligible.
Gross margins also vary by rank. NVIDIA can charge margins of up to 70%. The second-place player might get 40% to 50%. From third place onward, making GPUs becomes a losing proposition. The volume is too small to cover R&D costs.
Memory follows the same pattern. So does industrial software. Many industries do.
I believe LLMs will follow semiconductor dynamics exactly. That means capturing the largest market share is a matter of life and death for any company.
If your revenue is large enough, you can reinvest a portion of it. When your investment exceeds your rivals’, your models train better and your inference costs go lower.
So more people want to use your model. Revenue grows. And the cycle repeats. If your market is too small, you can’t raise enough money to fund the next generation. You gradually fall behind.
This should be obvious. Yet, myself included, most people are distracted by other factors. For example, in the near term, LLM R&D is still mostly funded by external financing. Equity, debt, or whatever form it takes. But if LLMs are to become a sustainable business, they must eventually transition to a stage where revenue — specifically, positive cash flow from inference — funds R&D.
If you believe in AGI, and in its potential to generate enormous wealth for humanity, then both the returns and the required investment will dwarf most existing industries. This industry cannot survive on perpetual external subsidies.
Another reason this pattern isn’t obvious is that, for the past few years, investment size didn’t directly determine model quality.
Before 2022, Google far exceeded OpenAI in compute, talent, and resources. But OpenAI had better organization and tighter focus. The decision-maker coordinated directly and was deeply technical. He mobilized the entire company around a single mission. And achieved far more than Google.
Google’s LLM research remained largely academic. Resources and talent were scattered. Multiple teams built similar things. But no single team had the decisive compute to pull off something like GPT-4. That was Google’s failure.
After 2022 — in 2023, 2024, and 2025 — we also saw many large companies fail to build good models despite heavy investment. Meanwhile, smaller companies built better models through better organizational culture.
But a good organizational culture won’t remain a scarce resource forever. Talent moves. Good culture spreads.
By now, many companies have found the organizational structure needed to develop LLMs. Can anyone really claim OpenAI’s culture is ten times better than Kimi’s? I think that gap has largely closed.
In the short term, Chinese startups can indeed train models only three to six months behind the frontier using roughly one-thirtieth the compute of their American counterparts. To close that gap, Chinese companies can also use distillation and other methods to gradually approach frontier models.
I don’t know whether that trend can persist. Someday, training a frontier model may require hundreds of millions, even billions, of dollars for a single pre-training run. Unlike today, when Kimi K2 was reportedly trained for just $8 million.
Of course, maybe Chinese researchers really are that much smarter. But holding that variable constant — say, looking only at competition between two Chinese companies — the one that spends more will likely train the better model.
In the long run, when two companies have similar culture and organization, the deciding factor is still R&D investment. And the most important input in R&D is compute and talent. Specifically, the salaries you’re willing to pay.
So DeepSeek’s decision not to raise funding last year was, in hindsight, a massive strategic mistake. Without fundraising, there are no GPUs. And researchers can’t be paid competitively. This hurts both R&D and retention.
I’ve seen many comments attributing DeepSeek’s recent struggles to key talent being poached. And Liang Wenfeng cannot match the salaries offered by rival firms. Why can’t he match them? Is he unable to get the money? Not at all.
In 2025, countless funds — official, private, national, domestic, and overseas — wanted to invest in him. He simply turned them down.
If you briefly reach the top spot in China, and everyone wants to give you their best resources, but you refuse, then what gives you the right to stay on top? Why would the best people keep working for you?
Culture matters, of course. People want to work at DeepSeek. But how far can culture really go? If ByteDance offers 10 million yuan, and DeepSeek offers 6 million — a 40% discount — maybe many stay. But if DeepSeek can only offer one or two million, I think most people would leave.
Do you actually believe in AGI? Do you believe talent is essential to achieving it? If you do, and you believe it will generate enormous societal value, then you must believe in the demand for it. You must scale up, invest more, raise salaries, and raise as much money as you possibly can. If you just want to run a closed-door research lab, then why should society bet on you at a time when industrial competition is a matter of life and death?
In the following sections, I will discuss several directions. First, a look at the supply side. Second, a discussion of the demand side. Third, whether patterns from the semiconductor industry can be applied to LLM development.
Let’s start with the supply side. LLM supply is straightforward. Your production inputs are GPUs, or data centers. Your output is tokens.
Jensen Huang talks about the “token factory” all the time because he’s spent 40 years in semiconductors. This is obvious to him. Most of us come from the previous era of software logic and don’t think in terms of hardware manufacturing.
To simplify, I will use H100s for all estimates below.
First, what does an H100 cost? For easy calculation, I’ll use rental prices, since they already include depreciation, facility costs, maintenance, and so on.
H100 rental prices have risen. They used to go as low as $2 per hour. Now, due to GPU shortages and surging demand, they’re up to about $2.50 per hour. That’s roughly $21,900 per year.
How many tokens can an H100 produce? Using DeepSeek V3’s figure of 2,000 tokens per second, that’s 7.2 million tokens per hour. So the cost per million tokens is roughly $2.50 divided by 7.2, or about $0.35.
Of course, data centers aren’t running at full utilization around the clock. So if the final price stays somewhere near $1 per million tokens, that’s roughly break-even.
Token pricing is hard to pin down, though. OpenAI and Anthropic API prices are inflated. In practice, their rates for coding plans and consumer-facing products are lower. B2B and B2C users also have different price sensitivities.
Let’s ignore those differences for now and assume a fixed gross margin.
Based on last year’s reports, Anthropic and OpenAI probably have inference gross margins around 40% to 45%. But I think the real number is probably a bit higher. Let’s use 50%.
So one H100, at $21,900 per year, should generate roughly $44,000 in revenue and $22,000 in gross profit. Scaling up, 10,000 GPUs would mean about $400 million in revenue and $200 million in gross profit. 1 GW roughly corresponds to 1 million H100s. That’s about $40 billion in revenue and $20 billion in gross profit.
With this model, we can even do a rough valuation of OpenAI. How much compute does OpenAI have? By the end of 2025, it should just have crossed 1 GW. By end of 2026, it might reach 3 to 5 GW. Using the earlier estimates, that corresponds to $120 billion to $200 billion in revenue and $60 billion to $100 billion in gross profit.
What about its long-term plan? By 2030, OpenAI claims it will have 30 GW of compute. That would mean about $1.2 trillion in revenue and $600 billion in gross profit. If we discount that to net profit — say, $400 billion — and apply a 20x P/E ratio, that would imply an $8 trillion valuation.
Of course, current private-market and pre-IPO trades value OpenAI at around $1 trillion. Would I buy at $1 trillion a company that might one day be worth $8 trillion? I can’t say for sure. There are still too many uncertainties.
How much of that revenue can support reinvestment?
Based on last year’s financials, OpenAI’s compute investment for R&D was about $9 billion. That translates to roughly 450,000 H100s. This is roughly consistent with its total fleet of over 1 million H100-equivalent GPUs.
OpenAI is currently over-investing in R&D. Far beyond what its current revenue can support. Because so far, R&D is still being funded by external capital.
When the promised 30 GW is fully deployed, how much of it will be allocated to R&D? For a tech company, a common benchmark is reinvesting about 10% of revenue into R&D. With $1.2 trillion in revenue, that supports at most $120 billion in R&D. Of that, salaries and operating costs take a chunk. In compute terms, that translates to roughly 3 GW. That’s a reasonable share of its 30 GW total.
Now back to production — the inference side. Continuing from the 30 GW figure.
Recently, OpenAI researcher Noam Brown gave a talk. I didn’t attend, but I saw a screenshot. He argued that control over inference-side compute has become a strategic advantage. While model weights themselves are less important. That is exactly the point I’m making here.
To contrast, consider a Chinese startup. Because China cannot buy NVIDIA’s top-tier GPUs, no matter how strong its technology, its inference scale is capped by domestic GPU supply.
How many domestic GPUs are there? Last year, Ascend 910C production was probably around 1 million units. Of course, that depended on two things. First, compute chip cores that TSMC manufactured for Huawei in 2020. Second, over 10 million HBM memory chips that Samsung supplied before the relevant sanctions kicked in around 2022 or 2023.
Those inventories are long gone. This year’s 950PR and 950DT series rely entirely on domestic supply chains.
What is the bottleneck ceiling of those domestic supply chains? I read a SemiAnalysis article last year. It argued that because Huawei’s phone business is not under sanctions, SMIC’s 7nm capacity could be fully allocated to Ascend chips. That translates to roughly 7 million units.
By comparison, CXMT’s memory production is the bigger bottleneck. In the most optimistic scenario, Dylan Patel estimated CXMT could expand to 30,000 wafers per month. Annualized, that’s less than the memory needed for 1 million GPUs. But I’ve recently looked into the memory used in the 950PR. It’s not standard HBM, but a custom Huawei specification called HIBL.
HIBL is lower-performance memory. Still large in capacity, but bandwidth is roughly half that of HBM3. I couldn’t find much authoritative information on the process details. I posted a question on Zhihu hoping someone knowledgeable could answer. But perhaps it uses simpler stacking and packaging techniques, lowering production difficulty. So it might bypass the earlier 1 million GPU memory constraint.
CXMT’s total capacity is about 300,000 wafers per month. If all of that could be converted to HBM, it would correspond to roughly 60 million to 90 million HBM dies. At 6 HBM stacks per GPU, that’s enough for about 10 million GPUs. Of course, this is impossible — just a theoretical upper bound.
So Huawei’s output this year is probably in the low single-digit millions. And those millions must be split across many companies. Suppose a company — whether Zhipu, MiniMax, or someone else — secures 200,000 Ascend units. Each unit offers roughly half the compute of an H100, with throughput slightly above half. That translates to roughly $2 billion in annual revenue. If it still allocates one-tenth to R&D, that’s only $200 million. No matter how brilliant Chinese researchers are, you can’t rapidly train a better model than the US with that little compute.
This shows that inference scale and R&D investment are linearly related. And because the benefits of R&D radiate across all production, R&D costs are amortized by scale.
Everything above rests on one premise: you can find enough demand. That demand won’t materialize automatically. It may come from waves of product-market fit discovery, or from enormous effort spent integrating applications into the last mile.
But looking over a 3-to-5-year deployment horizon, if you believe in AGI and in its immense societal value, then that demand will appear. That is a strategic-level judgment.
So at the strategic level, you should set capacity as high as you can. At the tactical level, however, how to fill that capacity and find that demand remains a problem business teams must solve.
First, an LLM company must go global. Compared with the relatively smaller domestic market, overseas markets are far larger. Even if US access is blocked by sanctions, companies should do everything they can to enter Europe, the Middle East, Southeast Asia, Latin America, and other markets. This kind of internationalization must be planned in advance.
Back when Huawei couldn’t sell its switches domestically, it sent large teams abroad to open markets in Europe, Russia, Africa, and Latin America. It built an elaborate overseas organization. Eventually, Huawei generated enough orders and profits to subsidize domestic research, becoming the R&D powerhouse it is today.
By the same logic, if a Chinese LLM company wants a place on the global stage, it must go global, internationalize, and bring overseas revenue back to fund domestic model development. Of course, AI model exports should be much easier than hardware exports. Most people in LLM companies speak English, so language is not the main obstacle. You are exporting tokens. Tokens just travel through fiber-optic cables. You don’t need complex offline distribution channels, warehouses, service centers, or layers of local distributors.
If something this easy isn’t being done, how could you possibly handle harder orders?
Some might say we’re still in the externally-funded R&D phase, and current overseas revenue is negligible. But commercial capability is part of a company’s organizational culture, and it also takes time to build. If you don’t start cultivating a global sales culture today, by the time you truly need overseas revenue, it will be too late.
Moreover, OpenAI and Anthropic may each generate over $100 billion in revenue this year. If you capture just 5% of that, it’s $5 billion. That’s real money.
Besides, domestic token production costs may be lower than overseas. Alibaba’s Qwen 3.5 Plus is priced at $2.40 overseas and $0.60 domestically. If overseas tokens are more expensive and the market is larger, why not go earn that margin?
I’m less certain about whether companies need to expand B2B sales teams. I’ve always had a nagging concern that too many B2B salespeople can damage a company’s culture. But some things may have to be done. And this B2B effort should also be global. You should go directly to foreign companies to win foreign orders.
If Americans won’t buy, won’t Middle Eastern sheikhs? If a Chinese LLM company simply guards the domestic market, waits for organic user growth, waits for more funding, and waits for orders to come in, then it is not really a company competing globally. It is just a research lab. And in this industry, research labs can win for a while, but they rarely win in the end.
Beyond language models, video generation will also be a huge source of demand. YouTube’s annual revenue in 2025 already exceeds $60 billion, including ads and subscriptions. If we assume 30% of that goes to content production, that’s $18 billion. If half of that production cost is eventually served by LLM calls, that’s $9 billion in demand. This is a very rough estimate, but it shows that video generation alone is enough to form a very large token market.
If we convert that to tokens, today’s video model algorithms are still quite inefficient. Diffusion models still require multi-step generation. Long-context architectures for video are still nascent. But in the future, video generation costs should gradually approach the cost of one or two forward passes. At that point, a single GPU could easily exceed language model decoding throughput. For example, 10,000 tokens per second.
And because video generation is naturally more parallelizable, it may not even require HBM. Slower, cheaper GDDR, or even LPDDR, could suffice. If you design dedicated inference cards for video, costs could drop even further.
When video production costs fall dramatically, won’t that unlock even more demand? I think that’s highly likely.
Today YouTube is already a platform with over $60 billion in revenue. If part of content production is genuinely replaced by model generation, video generation won’t just be a cool demo. It will become a massive, production-grade demand stream for the LLM industry.
Further out, what demand will AGI bring? If LLMs solve interaction tasks on computers, the next step for AGI is solving interaction with the physical world. If it can replace all physical-world interactions that currently depend on humans, that market is dozens of times larger than computer-based work. It could potentially exceed $100 trillion annually.
Imagine a model deployed on a local robot controller. Input is video, plus possibly speech, text, tactile sensors — multimodal signals. Output is action. It sees its surroundings, then cleans your house or works on a factory floor. Each such robot may require the equivalent of today’s eight H100s, or even more. If there are a billion such machines in the world, that might mean ten billion GPUs. Of course, none of this may materialize until around 2035. Honestly, that’s not so far away.
As discussed, we start from two assumptions. First, LLMs are a mass-market commodity. Second, revenue can be roughly estimated from token production. Any industry satisfying these conditions will eventually settle into a pattern of one dominant winner, a second-place player barely scraping by, and everyone else losing money.
domestically, the market has already narrowed from the “hundred-model war” of 2023 to roughly a dozen players. The startup cohort includes DeepSeek, Moonshot AI, StepFun, MiniMax, and Zhipu. The large internet companies include ByteDance, Alibaba, Tencent, Xiaomi, Meituan, Xiaohongshu, Baidu, and a few others.
The reason consolidation hasn’t accelerated is, as noted earlier, that LLM development has not yet reached the point where massive capital is strictly required for success. Also, these companies are still relying on external funding or parent-company subsidies to fund LLM R&D, rather than relying purely on inference-generated revenue.
When those two conditions change — when the cost of developing each generation rises from today’s hundreds of millions of dollars to tens or even hundreds of billions, and when companies can no longer find external funding at that scale — the market will shift from fragmentation to concentration.
If only two or three of these dozen can survive, why wait until costs are so high that most go bankrupt or quit? Why not consolidate now?
Under the current paradigm, having so many companies doing essentially the same thing is a significant waste of resources. First, talent and compute are spread too thin. Each startup only has tens of thousands of GPUs. Each large company only has hundreds of thousands — except perhaps ByteDance.
Everyone’s work is highly homogenized. They’re all scraping similar data, building similar infrastructure, training similar models, and optimizing on similar tasks. For example, why do we need a Kimi K2, a GLM-5, and a MiniMax M2 or M3? Do we really need three separate models? Wouldn’t it be enough to pool all the GPUs and train one?
Conversely, if these companies merged, the benefits would be enormous. Each of them has its own strengths. DeepSeek on infrastructure. Kimi on alchemy. MiniMax on agentic post-training. Qwen on data. Seed on research. If they merged, plus good cultural integration, could they combine these strengths to build a better model, serve more customers, and compete globally? (I didn’t include Zhipu because it has no major weaknesses, but no exceptional strengths either).
This is the logic of market competition. Sooner or later, the only difference is the cost of waiting.
Possible merger patterns fall into two categories. First, startup-to-startup mergers. Second, large-company acquisitions of startups. Here we assume large-to-large mergers won’t happen due to irreconcilable core conflicts. ByteDance won’t acquire Alibaba, and Alibaba won’t acquire Xiaomi, for example.
What would startup-to-startup mergers look like? One possibility: a startup raises enough capital to simply acquire the others. For example, I think DeepSeek should acquire StepFun. If Xiangyu Zhang were leading DeepSeek V4, many decisions would not have been so hasty and absurd, and the exploration of AGI could have been deeper.
Another possibility is a merger among equals among startups. The main obstacle here is organizational culture. If merging causes cultural friction, it may not be worth it. So whether this happens depends heavily on personal relationships among founders. If Zhilin Yang and Yanjun Yin are on good terms, they could initiate contact and collaboration — data sharing, infrastructure sharing, technology sharing. After a period of working together, if things feel compatible, they could explore a merger. That would happen more naturally.
A third path is simply poaching from rivals. This returns to the core thesis that inference scale determines competitiveness. You can poach because you have money. And you have money because you sell more tokens.
Large-company acquisitions of startups follow similar patterns. The obvious one is a large company buying a startup. But if capital markets are bullish on the startup’s prospects, its valuation will be high. The large company may not have enough cash on hand. The second route is poaching. I think ByteDance has already poached quite a few people from other companies. ByteDance’s talent density may not be higher, but its headcount is at least 20 times that of the others. That gap may keep widening. Where does ByteDance get all that money? From the commercialization of its products — Doubao, Douyin, and the rest. So again, the point stands: commercial success is a necessary condition for research success, even if not sufficient.
We’ve discussed how startups exit competition. Another path is for a startup to simply dissolve. For example, Kai-Fu Lee’s 01.AI. At the time, 01.AI launched Yi-Lightning with just a 20-person model team. It reached sixth place on the LM Arena leaderboard. But within days, Kai-Fu Lee announced he was giving up. Most of the team moved to Qwen.
As for large companies exiting, I don’t work at one, so I don’t really know. But the general pattern is: you see massive spending with no visible return. And you kill the project, like Microsoft has done with some efforts.
Another phenomenon that may follow semiconductor dynamics is the emergence of vertical分工 and specialization. Semiconductors are an extraordinarily complex industry. As noted, due to the learning curve, each segment produces a winner that dominates the vast majority of market share. Yet along the vertical chain, each company usually manages one segment, rather than one company trying to own the entire stack.
TSMC succeeded in foundry services precisely because it only did foundry work and did not engage in chip design or other areas. This structure emerged for several reasons.
The first is that focus enables technical leadership and extreme customer service. Intel does everything. TSMC only does foundry. So it is harder for Intel to provide more focused technology than TSMC.
Another point Morris Chang emphasized in his autobiography is that TSMC should never compete with its customers. It should serve them. By doing only foundry and not chip design, it avoids competing with its customers. So customers trust it more.
By contrast, if a foundry also designs chips, when both the internal design team and a customer bring orders to the fab, whose wafer gets priority? Will customers feel uneasy sharing their designs with your company?
I recall that Google’s TPU v1 or v2 was fabricated by Samsung. After receiving the design, Samsung copied its own NPU version. This severely damaged Samsung’s credibility. From then on, Google moved its orders to TSMC.
A third benefit is that if a company monopolizes one segment of the vertical chain, its orders can diversify, reducing risk. TSMC can accept all kinds of contracts: CPUs, GPUs, communications chips, and more. On one hand, these orders mostly use shared technology, so TSMC doesn’t need to repeatedly develop entirely different processes. At the same time, diverse orders stabilize revenue. When GPUs are hot, TSMC earns GPU money. When CPUs are hot, it earns CPU money. Risk and volatility are greatly spread out.
Furthermore, the ability to rapidly serve diverse customer needs becomes a core competitive advantage. That is something Intel could not replicate.
A fourth point is that different segments, due to different R&D rhythms, production difficulties, and customer service characteristics, often require different organizational cultures. Consumer-facing 2C businesses, for example, require attention to product, marketing, and promotion. 2B-heavy enterprises need large sales teams and strong relationships with major accounts.
These are the main reasons vertical分工 emerged in semiconductors. We can see companies that specialize in lithography machines, like ASML. Companies that specialize in optics, like Zeiss. Or companies like TSMC that only do foundry. Each has achieved near-absolute dominance in its own domain, yet each occupies only one link in the chain.
As Jensen Huang puts it, NVIDIA pursues “Do as much as needed, and as little as possible”.
Will vertical分工 emerge in LLMs? Since ChatGPT, the dominant theme in model development has been co-design: cross-layer optimization. For example, product agents need to be tightly integrated with post-training. Product people communicate requirements directly to post-training teams, prepare corresponding data, and improve model performance on real tasks.
Inference infrastructure teams need to talk frequently with architecture teams to ensure models are as efficient as possible after training. Pre-training teams, in turn, need to work closely with infrastructure teams to maximize training throughput on large clusters.
The emergence of Claude Code and the evolution of RL algorithms are examples of this trend. From this perspective, the trend in LLMs seems to be tighter integration across stages, not分工.
However, the V4 release made me see another possibility. I remember when DeepSeek V3 came out at the end of 2024. The paper described a series of very principled optimization techniques. Parallel scheduling strategies, overlapping communication and computation, inference system design, recompute design, and so on.
Industry practitioners were blown away. They studied it extensively and tried to integrate DeepSeek’s key kernels into their own training systems. A year later, many teams have mastered those system designs.
But the optimizations in the V4 paper can’t even be described as principled, let alone极致. They are borderline insane. Black magic.
For example, MegaMOE in MoE computation and inference. If you use Expert Parallelism, you need two rounds of inter-GPU communication: dispatch and combine. To overlap communication and computation and avoid waste, normal teams split data into two micro-batches. While one batch computes, the other communicates, reducing idle time.
But DeepSeek simply wrote a single massive MegaKernel. It fused everything — communication, computation, the whole thing — into one kernel. It doesn’t rely on micro-batches for overlapping. Or rather, it has infinitely many micro-batches. After every tiny block GEMM, it immediately performs communication. So that no matter what the workload, everything overlaps perfectly. The original text used the word “wave,” suggesting a surging ocean.
A friend of mine who works on ML systems said MegaKernel is something the industry wouldn’t even dare to imagine. Because when you fuse all communication and computation together, there is no way to debug it. If something goes wrong, is it a communication issue, a compute kernel issue, or a broken card? There is no way to diagnose it. The difficulty is staggering.
All I can say is: the DeepSeek team is incredibly bold and skilled.
Beyond MegaKernel, V4 uses a series of similar techniques. For SCA and HCA attention computations, with so many tiny kernel launches, they still saturate the compute. After reading it, I just thought: is all this really necessary?
With such an incredible infrastructure team, the alchemy team seems to have gotten carried away. And it’s not just the attention design. When hit by loss spikes — a problem that shouldn’t exist in 2026 — the algorithm team’s first instinct was to solve it with system tricks. They came up with a patchwork fix called Anticipatory Routing.
To exaggerate slightly: if the DeepSeek V4 team wanted to train a large model at a time when there was no attention, only LSTM, they would not pursue a parallelizable algorithm. They would simply write an infrastructure system that saturates GPU compute on LSTM.
A second-tier alchemist I know put it this way: a weak infrastructure team actually acts as a regularization on algorithm design. If an architectural change doesn’t bring a huge improvement, you can’t persuade the infrastructure team to modify the training code. So you avoid falling into local minima.
System optimization and algorithm design require fundamentally different team cultures. The biggest difference is that system optimization gains are predictable. You know a card’s compute, its bandwidth. If you reorder execution, overlap communication and computation more deeply, run the arithmetic. You can conclude exactly how much faster things will get.
And with good code management, changes to kernels are isolated. Modifying the front compute doesn’t affect the rear compute. So overall system complexity grows only linearly with the number of changes.
Alchemy, by contrast, is pure black magic. When you make a change, there is no way to predict whether it improves or degrades model performance. And different components interact in mysterious ways. For example, touching attention might affect MoE computation. When errors occur, it’s hard to tell whether the problem is algorithmic, optimizer-related, architectural, or numerical.
System complexity grows exponentially with the number of changes. So good alchemists are extremely cautious. They pursue minimal changes. They seek deeper understanding of phenomena. And they propose the simplest possible algorithms. Because even then, the underlying understanding remains a black box. Just a slightly shallower one.
In terms of team culture, infra-bros and algo-bros are very different. Infra-bros take pride in mastering complex systems. They glory in solving hard problems. They look down on storytelling and hype.
Algorithm people pursue minimalism. They want algorithms and system designs to be as simple and universal as possible. And they care deeply about storytelling. Because although a yin-yang five-element theory of understanding is far from the true nature of things, telling a story is still better than not telling one.
I suspect that inside DeepSeek, the infra-bros are so numerous and so dominant that the algorithm people have been squeezed out. For example, if there were an Ilya-like prophet in the company, speaking in riddles every day. Or worse, a Jason Wei, who got a paper out of the sentence “Models should think before answering” and became a celebrated figure. Their attitude would probably not be admiration or respect. It would be contempt and mockery.
Here, I want to make a bold claim. Perhaps DeepSeek’s best path forward is to transform into a pure infrastructure company. Its business could be providing inference frameworks for other manufacturers. For example, a model trained by Kimi or ByteDance may be costly to deploy internally. Hand it to DeepSeek, and DeepSeek would look at it and say, this is basically a V3. Then they’d write a system with极致 optimization, fully overlapping MegaKernels, maybe even overclock the GPU. Cut inference costs dramatically. The savings get split between the model owner and DeepSeek. Isn’t that ideal?
For their customers, managing and continuously optimizing inference systems is a huge hassle. It involves too much low-level kernel optimization that has nothing to do with the pursuit of AGI. If all of this could be outsourced to DeepSeek, the customer could focus purely on the model.
For example, if Moonshot keeps its headcount at three to five hundred people, for cultural and anti-bureaucracy reasons. Then those three to five hundred people could all be like Jianlin Su. Or half Jianlin Su and half post-training data cleaners. The company’s direction becomes more focused. Alchemy quality goes up.
And for DeepSeek, it can finally stop resenting the black magic of alchemy. And focus on what it does best: infrastructure optimization. You love patching kernels? Go patch them. There are endless patches to apply. You’ll never run out. It’s so satisfying.
In model quality, DeepSeek V4 isn’t even the best in China. Globally, it probably ranks fifth or lower. That’s very different from V3’s brief moment of leadership.
But in infrastructure? I don’t believe there are many people in the world stronger than this team. It could be an absolute number one. An absolute monopoly.
As emphasized earlier, market leadership is a matter of life and death. If it could capture even half of China’s total inference market — if ByteDance and others outsourced to it — that would be an enormously profitable business. In other words, it would be taking money from upstream cloud providers, not competing with downstream model makers.
Liang Wenfeng has said his goal is AGI. But who says providing infrastructure isn’t AGI? If their existence lets every company in China focus on alchemy without worrying about tedious kernel optimization, doesn’t that accelerate China’s march toward AGI?
Of course, such companies already exist. Notable examples abroad include Together AI and Fireworks. But they seem to face intense competition and are doing just okay. Why wouldn’t DeepSeek end up the same way?
First, before V4, no company was two or three generations ahead in inference optimization. But V4’s technology is genuinely insane. Second, overseas companies mostly use NVIDIA cards. NVIDIA cards are well-supported and relatively easy to optimize. Most model companies still have a chance to build reasonably efficient inference systems in-house.
But in China, the foreseeable future depends on Ascend. Ascend’s ecosystem is very weak. And there are even issues with compute precision. Getting Ascend to work well is beyond the reach of ordinary companies. If DeepSeek offered this service, and a model ran as fast on a 950DT as it would on an H100 at over 60% throughput, for less than half the cost, that would be an enormous, unassailable advantage.
By analogy, the foreign company best positioned to do this is Thinking Machines. Like DeepSeek, they gathered a group of elite engineers who spend all day patching infrastructure and kernels, not doing anything “proper”. I’d suggest Amazon acquire Thinking Machines. Their Trainium is probably about as hard to use as Ascend. Let the Thinking Machines folks fix Trainium. I think that could work.
TSMC founder Morris Chang repeatedly emphasized in his autobiography that he preferred “customized” products over “mass-market” products. Because the former typically command higher gross margins. Mass-market products, with standardized specs, face fierce competition. As discussed, winner takes all, small players lose money. But customized goods are not in fully commoditized competition. There is more room for pricing negotiation.
We can ask the same question: will the LLM market see customization for different companies? There have been many attempts, broadly falling into two categories.
The first is startups attempting to fine-tune specialized models. Medical LLMs, financial LLMs, serving professional clients. They may find early success in niches that large model companies ignore. But quickly, when companies like OpenAI or Anthropic notice the opportunity, their specialized models lose to new general-purpose models. The reason is simple. A domain-specific LLM is really just different data. The underlying technology is largely the same. If you research in only one domain, you can’t advance as fast as someone researching across all domains simultaneously. And different domains often produce useful cross-pollination. For example, reasoning model techniques were originally tuned for math tasks. But they quickly affected every domain.
The second category is companies offering fine-tuning services for clients. Applying the same techniques to different domains based on customer needs. This includes startups like Prime Intellect and Thinking Machines, as well as model providers like OpenAI and Anthropic. Does this work? To some extent. We often hear news like Anthropic providing a reduced-safety-review model version for the US government, for military and defense applications. But overall, these businesses have not performed very well.
I suspect this is related to the lack of continual learning. Today, LLM learning processes can be broadly divided into two categories by duration and training cost.
One is pre-training and post-training. I’ll group these together because they have to be done all at once. The total training volume is in the tens to hundreds of trillions of tokens. And the cycle lasts months. Once the model is released, its weights are fixed.
The second category is in-context learning. The model adjusts its behavior based on context. But context length is at most around 1 million tokens. What can be learned is limited. And it does not rely on customized weights or dedicated deployments.
I discussed this in my article “Perhaps Continual Learning Should Not Be Achieved”. I encourage you to read it.
Recently, however, some new post-training techniques may point toward a form of continual learning. Pre-trained and SFT models often suffer from catastrophic forgetting. If training data is not independent and identically distributed, not shuffled, but sequential — learning one thing first, then another — the model forgets earlier skills while learning later ones. So model training must be done in one big batch.
But people have discovered that reinforcement learning and on-policy distillation suffer far less forgetting. For example, both Zhipu and Nemotron used cascade RL when training models. Instead of learning all RL domains simultaneously, they learned them one at a time. First RLHF, then math, then coding, then agentic tasks. The result: when learning later domains, performance on earlier domains barely dropped, at most by two points.
And on-policy distillation shows even less forgetting. A stream of recent papers has explored this, spawning techniques like self-distillation. Dario also mentioned in an interview that continual learning might be achieved in some form this year.
If this does happen soon — if we can insert a billion-token or tens-of-billions-token level of continual learning between the hundreds-of-trillions pre-training stage and the 1-million-context in-context learning stage — then customization might actually become viable.
For example, a company has internal documents, operating norms, customer information, and so on. Under current agentic paradigms, these are often packed into files or skills, then retrieved, read, and reacted to on the fly. This often fails to produce genuine behavioral change, misses important information, or prevents cross-domain association. If continual learning is achieved, the model could organically absorb and synthesize all this context, like a veteran employee. It might deliver significantly more value than using a pre-trained general-purpose model directly.
I mentioned in “Perhaps Continual Learning Should Not Be Achieved” that implementing continual learning for individual users may not be worth the cost. Because it means deploying an entirely new model weight set for each user. That is expensive and underutilizes compute.
But if a company with thousands or tens of thousands of employees gets its own dedicated inference deployment, and the improvement is truly substantial, the cost could be amortized. It is worth keeping an eye on these technologies. If a breakthrough does occur, a new business model could emerge.
This leads to a second question: if it does happen, will model providers capture this market? Or will specialized companies take it? I don’t know, but we can look at it from two angles.
First, does a 2B business require a different organizational form and company culture? For example, serving a company in this way often requires sending engineers on-site, collecting offline data, and debugging in real time. Would a company like Anthropic be willing to do that? Maybe. But maybe not.
Second, might the profit margins in early stages be too small to attract large companies? Here I am drawing on the arguments of Clayton M. Christensen, author of The Innovator’s Dilemma. If a new technology can serve existing markets and existing users, large companies will fight for it. Small companies have no chance. But if an emerging market initially generates too little revenue — say, only millions or tens of millions — large companies disdain it. Small companies, however, are happy to enter. A few years later, as the emerging technology matures and becomes mainstream, large companies that want to enter find it is too late. The small company has accumulated years of know-how and surpassed the large players.
]]>My feeling is that after several hours I had probably read only 20% of it; of that 20%, I may have understood only another 20%. Fully understanding it will take a long time.
But the value of news is that it is new. I really could not wait until tomorrow to write this up, so I am writing down all my current impressions first.
Everything below is a hot take. The value of a hot take is not that it is very accurate. In fact, I think only 5% to 10% of what I say here may be right. But because it is non-consensus, its value has not yet been discovered, so it may still be interesting to hear.
The first hot take I want to make is this: DeepSeek may not be as strong as Kimi at “alchemy.”
Here are a few choices in the technical report that I found very strange.
The first puzzling move is the “m” in mHC. The mHC paper has been out for a few months. The idea behind HC is actually quite simple: widen the residual stream, so that layers can pass information forward in arbitrary forms, thereby greatly increasing the expressive power of the model. This part itself is fine.
The problem is the manifold treatment. DeepSeek found that because HC does not constrain changes in scale during the forward pass, it can lead to exploding or vanishing gradients, so some method is needed to constrain the scale. In the end they used a method called manifold, which introduces Sinkhorn.
But this Sinkhorn procedure has quite a few issues. First, it is not very “beautiful.” As a later paper, MHC Lite, points out, on ill-conditioned matrices, even after 20 iterations, Sinkhorn may not converge to a good probability distribution close to the ground truth. That alone introduces a nontrivial error.
At the same time, the algorithm itself is relatively slow. The original MHC paper says it adds about 4% compute time, which is not a small cost.
More importantly, because “alchemy” itself is basically mysticism, any extra complexity makes debugging extremely difficult. On the other hand, neural networks have a very strong ability to discover patterns automatically. In many cases, you do not need to manually give them overly strong or overly complex expressive structures. Instead, as long as you give them enough capacity, they will naturally learn all sorts of representations through combinations of different matrices.
Later, someone on Zhihu proposed a cleaner architecture called Identity HyperConnection. The core idea is: even if the hidden state becomes wider, the forward pass still preserves identity. This looks much more natural, and much closer to Kaiming He’s original design philosophy for residual networks.
In other words, we only need to give the model a larger space for forward information flow. We do not need to manually prescribe the semantic transformations between these forward paths. Those semantics do not need to be hand-designed. You only need to give it a wider “writing space,” and then let each layer decide what to write at each position. That is enough.
From this perspective, introducing MHC adds a lot of unnecessary complexity to the whole system.
The second strange point is the tuning of Muon.
In Muon, the principle behind tuning the Newton-Schulz iteration has already been explained very clearly in the PolarExpress paper. There is a relatively clear, even almost “optimal,” way to tune it.
So unless you believe low-precision training has a large effect on this kind of iteration, using two different sets of coefficients for the first two terms seems somewhat unjustified.
I also do not know what experiments led to this setting, or whether V4’s training actually happened at a relatively early stage, so that many later techniques that had already been verified as better were not used.
The third strange thing is its treatment of MoE.
First, it says that when people normally train MoE, the router uses either softmax or sigmoid, while here they chose a different path: the square root of softplus.
I have long had a conjecture about the optimal activation function for MoE: perhaps the best one should be soft top-k. This is also recorded somewhere on kexue.fm.
But square root of softplus is really hard to understand. What exactly is the intuition behind it?
And if you only change this one local design, the gain looks limited, while the added complexity to the whole system is not small. Whether this trade-off is meaningful is worth doubting.
Let us keep going.
The report mentions that during pretraining, training would collapse. They searched for the cause for a long time and eventually attributed it to MoE, saying that the interplay between MoE and the router would lead to certain extremely large values. Then they proposed a fix. Just hearing it sounds a little incredible: if a loss spike appears, roll back and continue training using the routing decisions from a previous checkpoint.
The problem is that to obtain these routing decisions, you need to do one extra prefill pass, and this extra cost is around 20%.
First, by 2026, it is already a little strange to still have MoE training collapse, especially with obvious loss spikes. Second, after training collapses, fixing it this way is not cheap, especially with this 20% overhead. The whole report is describing almost god-tier systems design. Can you really claw back more than 10% efficiency from somewhere else? I am honestly a bit skeptical.
Of course, this mechanism only triggers when something goes wrong, so the average cost may be controllable. But the bigger issue is that the loss spike likely points to a more fundamental training instability, rather than a local problem that can be simply patched.
One could even speculate one step further: could the timing of V4’s release have been delayed partly because of this problem?
Here is my conjecture: is it possible that the MoE training collapse is precisely because of the square root of softplus mentioned above?
Softplus is a continuous version of ReLU, specifically log(1 + e^x). Even if you wrap a square root around it, its value is still unbounded. That means if my logit increases, the gate value also increases, without an upper bound.
This is different from traditional sigmoid and softmax, and also different from my conjectured soft top-k.
What kind of problem might this create? When the gate is unbounded, if a certain expert really “likes” this hidden state, then even late in training, positive gradient will keep flowing in to further raise this logit. The gradient may therefore keep accumulating until the router blows up.
Of course, all of this is just conjecture.
These are the strange choices I noticed.
Of course, since this is mysticism, I cannot claim to understand the principles behind all of it. But looking at the overall style of work, dealing with mysticism should be completely different from dealing with systems, which is what DeepSeek is good at.
What is the difference?
For systems problems, complexity is often additive and linear. It is controllable. If I change a kernel in the front, it will not affect some later operation. At the same time, the improvements from most optimizations are predictable. For example, if I write a mega-kernel, before writing it I can roughly judge how much benefit I will get from pushing communication and computation overlap to the limit.
But since alchemy is mysticism:
First, its risk is uncontrollable and unpredictable. Every change may be good, or it may be bad. Only by running a large number of experiments can you slowly develop a bit of sense for how to handle it.
At the same time, the risk is not isolated. If I change attention in the front, it may affect the MoE later. If I change the hidden state, or some router, it may affect Muon’s computation. Everything is interplay.
And once you cannot isolate risk, the risk grows exponentially with the number of optimizations.
How do you handle these possible changes? The only method is ablation. But ablation quickly becomes a combinatorial problem. Fundamentally, the only way to guarantee a good method is grid search, and the number of experiments required by grid search also grows exponentially with the number of optimizations.
Therefore, if you have enough respect for mysticism, each generation of model should use at most three new techniques. With three new techniques, grid search is cubic in n, which is still controllable. Any more than that, I think you cannot afford it no matter how many GPUs you have.
So does this suggest that DeepSeek’s engineering background somehow determines that it does not respect mysticism enough?
This reminds me of something. Earlier, when I was writing the Expert Threshold article on MoE, both the writing and the experiments drew heavily from DeepSeek’s very famous Loss-Free Load Balancing paper. In my view, that paper was the most important MoE paper of 2024-2025. You could even say it was the only meaningful architecture-change paper in that period.
My thought at the time was: since this is the best one, I can just follow it. For example, it only reported perplexity and did not run all kinds of benchmarks like normal pretraining papers do. I thought, if DeepSeek does it this way, then I can be lazy too; I even tested one more thing than they did, DataComp.
The figures in Loss-Free were basically drawn casually. I still put a little care into mine. Of course, they were not as polished as figures in other accepted papers, but I thought mine were at least a bit better than DeepSeek’s. That should be fine.
Then the reviews came back, and the scores were extremely low. I was confused and showed them to my advisor. My advisor asked which paper I had used as reference. I said DeepSeek.
He told me to find the OpenReview page for Loss-Free. I opened it, and it had three rejects, even lower than mine.
My advisor looked at the paper and gave the following evaluation:
No matter how important the technique is, from the perspective of academic writing, the paper is simply unqualified.
First, there are even grammar mistakes in the paper. Second, the figures are very casual. Third, the experiments that should have been done were not done.
So even if it is a very important technique, it deserved to be rejected. Its influence mainly comes from DeepSeek. If V3 had not used it, it would not later have become a famous paper.
That was how I understood the matter at the time.
Looking back, this may precisely mark a certain lack of respect DeepSeek has for “mysticism.”
Why do the top ML conferences need so many “stories”? Why so many rules, ablation studies, and extra experiments? Precisely because when facing mysticism, you must be extremely rigorous and extremely cautious. You need to do many extra things before you can establish even a little confidence in whether a technique is effective.
But DeepSeek’s paper did not write these things. Perhaps that is because it does not care.
The Loss-Free paper may have “hit the target” because its intuition was too good. But if the same attitude is used for later mysticism research, many mistakes may follow.
So, about this alchemy thing: even though we mock it every day, as a large model company you still need people inside who have that god-given touch.
For example, if Kaiming He were at DeepSeek, I do not think he would allow the “m” in mHC to be published as the final solution. He would definitely run a huge number of ablations here and finally reduce this m to the simplest possible form before being satisfied.
Only when you reduce it to the simplest form can you be sure that its risk is controllable: that it will not significantly increase system complexity, and that it will not collapse during large-scale model training.
Yang Zhilin wrote papers with more than ten thousand citations during his PhD, so he clearly received systematic training in alchemy. Never mind whether the “storytelling” and all the theories in those papers are actually reasonable. In practice, there really is a difference between judgments made by people with experience and people without that touch.
In even more extreme cases, people like Alec Radford or Noam Shazeer have an even stronger touch, and that itself is a talent. Sometimes they may not be able to fully explain why their architecture works, but the result is that it works.
So I arrive at the hot take: DeepSeek may be missing such people. At the same time, should its organizational style make more conservative engineering choices for this kind of “mysticism,” rather than being as aggressive as it is in systems optimization?
On the product side, before DeepSeek V4 was released, I predicted that it would have a lot of black magic in infrastructure and algorithms, but as a model, as a product, it would not perform extremely well, or at least would not be better than Kimi, Zhipu, and MiniMax.
Here is how I understand it: products need user feedback. They are the result of continuous iteration.
For example, why is Anthropic doing well? Because it first built Claude Code. The agent itself is nothing magical, right? I think all agent harnesses are nothing magical. The agent wrapper itself does not bring any magical experience.
But in the process of using this model, Anthropic obtained a lot of user feedback, a lot of data, and a lot of practical insight. Then it could bring these insights back to the training team and say: we need to run post-training specifically for this, run RL, do human labeling, collect more data. Only after this kind of training can the model truly learn how to use this agent system.
In other words, this is an iterative process. Start with 3.7, build Claude Code on top of 3.7, then use the experience from Claude Code to train 4. After 4 is trained, use its usage experience to train 4.5 Sonnet. Then Sonnet trains 4.5 Opus, then 4.6, 4.7, and so on. Even Anthropic went through at least four or five rounds of model iteration and agent-product co-design and co-optimization before it finally became a useful product.
Kimi, MiniMax, and Zhipu all also care a lot about coding. Internally, they are iterating on all kinds of products every day.
Just take MiniMax. From 2.0 to 2.1, 2.2, 2.5, and then today’s 2.7, that is already many rounds of iteration.
If DeepSeek really cared about this, then we would not only see a 3.2. We should at least have seen 3.25, 3.28, 3.3, right?
Ever since ChatGPT appeared, I have been thinking about one question: how should we understand the dynamics of commercial competition in large models? What standard should we use to judge whether a company will succeed or fail?
For example, the previous era was the internet era. In the internet era, a company’s success fundamentally came from scale effects, platform effects, or network effects.
In other words, if your product is good, it first spreads virally. After you have a sufficiently large user base, the value of the whole platform increases as the number of users increases.
It is like WeChat: if everyone uses WeChat, then even if you do not like WeChat, you still have to use it.
This dynamic determines the division of labor in that system: why lean startup was needed, why product managers were needed, why iteration cycles were fast, and where those high margins came from.
However, so far, large models have not shown obvious platform effects. The most intuitive sign is that user loyalty is basically zero.
For example, if Anthropic releases a new model, I may go use Anthropic. If OpenAI releases a new model, I switch to Codex. Users have no stickiness.
Therefore, the evaluation standards of the internet era do not apply in the LLM era.
So what standard should we use to think about the essence of commercial competition?
I increasingly feel that the learning curve can explain it.
I first saw the idea of the learning curve from TSMC founder Morris Chang. He attached great importance to the learning curve and applied it to every aspect of running his business.
Specifically, the learning curve itself is simple: the more you do something, the better you get at it.
But many people have not thought through the consequences behind it.
Suppose I have a product. The more I sell, the better I become at making it. The better I become, the lower my cost and the higher my quality. In semiconductors, quality here usually means yield. If my quality is high and my cost is low, customers naturally like me and place more orders. Those additional orders then give me more opportunities to practice, and my cost continues to fall.
When enough time passes, this flywheel enters a positive cycle, and eventually my advantage becomes something other competitors cannot match.
The most obvious example is TSMC. When it began preparations in 1985 and was formally founded in 1987, TSMC’s technology was not leading. At the time there were at least dozens of competitors, and many semiconductor companies did not use fabs but manufactured chips themselves.
By the 1990s or 2000s, perhaps only about five competitors were left. By the 2010s, basically only Samsung and Intel remained as competitors. Recently, entering the 2 nm era, Samsung has gradually fallen behind too. So TSMC has become almost the only choice for all chip design companies, and naturally its gross margin can be very high.
However, the cycle required to build this advantage is far longer than the monopoly produced by scale effects in the internet era. As I said above, it took TSMC a full 40 years to reach today’s dominant position. Nvidia’s position in graphics cards also took roughly 30 years.
So the essence is really endurance: endure continuously for decades, outlast all competitors, and naturally become the winner.
My guess is that the development of large language models will also follow the law of the learning curve.
Concretely, we can view training as R&D and inference as production. I invest a certain amount in R&D, then I go produce, and this brings me profit and revenue. I then take part of that revenue and put it back into R&D.
In that case, one very important key determining a company’s eventual development is revenue.
For example, if, like Anthropic and OpenAI, my annual revenue this year can reach tens or hundreds of billions of dollars, then I can take, say, 10% of that for R&D, which is already billions or tens of billions of dollars in R&D.
If you are a small company with only a few billion dollars in revenue, and you also take out 10%, you are left with only a few hundred million dollars. It becomes very hard to compete with larger firms, even if your gross margin is similar.
So when is victory decided?
When your revenue is large enough to squeeze out all competitors, while the R&D cost of each generation of model keeps rising until other competitors and smaller firms cannot bear it. Then only you are left. It is like TSMC today: developing a new process generation costs tens of billions of dollars, and even Samsung and Intel cannot easily put up that money. Or even if they do put up the money, because their sales volume is not enough, the investment is bound to lose money.
Only under these conditions can you win normal competition.
OpenAI’s R&D spending last year was nine billion dollars. I do not know about this year; it may keep rising. When it rises to the point that other companies cannot bear it, perhaps the winner will be decided.
Of course, several factors make the matter less simple.
The first is the existence of external capital.
Today, no company funds R&D purely from profits out of revenue. Everyone uses venture capital money.
And venture capital money is somewhat arbitrary. Suppose my company has no accumulated advantage at all. Through enormous investment, external investment, can I close part of the gap?
Of course, maybe this is not exactly venture capital. It is more like the behavior of large companies, such as Xiaomi, Meituan, and xAI, though xAI now seems to have other problems.
Second, so far, when we observe Chinese companies, they can still use only a few hundred million dollars of R&D cost per year to reach something close to what OpenAI achieves with ten billion dollars of R&D cost.
Of course there is a lag, roughly three to six months.
When competitors spend a lot of money training a new model, I can distill it.
Since this gap can save dozens of times the money, are there still some subtleties here that need to be considered?
If my hypothesis is true, then from the perspective of the learning curve, we can look back at DeepSeek.
Perhaps we can make this conjecture: DeepSeek not raising capital last year may have been a huge mistake.
Without financing, you cannot expand capacity. Without capacity expansion, you do not have users, and therefore you do not have revenue. Without revenue, sure, for now Liang Wenfeng is paying out of his own pocket, but if in the future you have no revenue, you will not have enough money for R&D.
No matter how good your culture is or how strong your team is, if competitors use ten times the money to poach your people, they will eventually poach quite a few of them, right?
Recently there have also been rumors that several core collaborators have already gone to ByteDance, while others have gone to Xiaomi and Tencent.
Of course, Liang Wenfeng’s problem is that, at least in 2025, financing could not be converted into compute.
DeepSeek also could not smuggle in a large number of Nvidia GPUs the way ByteDance could.
At that time, Huawei cards were probably also very hard to use. Buying them would not have helped much.
At the same time, there is another difference in large model R&D: if your headcount crosses the threshold of three to five hundred people, the efficiency of the whole organization drops sharply, because communication costs rise. The organization must also have structure, which means all kinds of bureaucracy and similar things will appear.
In other words, the ideal team size, even if you have infinite money, may still be three to five hundred people.
Therefore, hiring more people is not very meaningful. But if you raise more money and give current researchers higher salaries, does that help? It probably still helps.
The learning curve also determines the product, or rather the customer relationship. Serving customers is the key indicator that determines whether even a pure technology company can develop in the future.
For example, TSMC’s culture strongly emphasizes customer priority. Customer priority comes before technology leadership. That is, it first satisfied the requirement of customer priority. Only then did it have enough revenue, enough gross profit, and enough money to spend on R&D.
Then, when your R&D budget is large enough, you can achieve technology leadership.
So TSMC achieved technology leadership only around 7 nm in 2018, 31 years after it was founded.
DeepSeek seems to focus purely on technology and not pay enough attention to demands such as coding agents, various strange agents, or user requests for multimodality and things like that.
What worries me more is whether it has failed to establish this kind of culture, and whether the whole organization does not value this matter.
Then can it really win this marathon?
DeepSeek V4’s own official account ended with this sentence:
“Not tempted by praise, not afraid of criticism; follow the way and hold oneself upright. We will always uphold the principle and philosophy of long-termism, move forward steadily through attempts and reflection, and keep working toward the goal of achieving AGI.”
What I want to emphasize here is this counterintuitive judgment: is persisting in technological exploration truly long-termism?
If you believe in the learning curve, then serving customers, expanding revenue as much as possible while maintaining and pursuing technology leadership, is the correct approach. Serving customers comes before technology leadership.
What kind of company does DeepSeek want to become? Does it want to become Bell Labs? Did Bell Labs survive in the end? Or does it want to become Fairchild, or one of those companies in semiconductor history that were technologically leading but ultimately disappeared?
Everything above is purely personal speculation. Please just treat it as me talking nonsense.
]]>The first time I encountered this idea was when Sora had just come out. I saw Saining Xie write the following about Diffusion Transformer (DiT):
When Bill Peebles and I worked on DiT, we did not chase novelty. We prioritized two things: simplicity and scalability.
At the time, that sentence hit me hard, but I still did not understand what it really meant.
Two years have passed. In that time I have read a lot of papers, watched all kinds of techniques develop and get applied, and done a few projects myself. Only now do I feel there is something almost Daoist about it: the great way is simple. I probably understand at most one third of it, but here is the part I can explain.
This post discusses why papers should pursue extreme simplicity from three angles: scientific exploration, marketing, and engineering practice, and then tries to explore how to practice it in actual work.
A paper should pursue simplicity first because science requires rigor and a spirit of investigation.
Suppose one day a researcher proposes an ML method. They add a bunch of engineering tricks, run experiments, see the numbers go up a bit, and very often just send the paper out.
But many questions remain:
Why did the numbers go up?
Which specific trick caused the improvement?
What is the mechanism behind it?
Can the method be reproduced and generalized?
Most people do not investigate carefully. It is like a traditional Chinese medicine prescription: throw in a dozen herbs, call it a compound formula, and in fact have no idea which ingredient is doing the work. Maybe only one molecule in one herb is useful, and the rest are actively harmful. Vague, muddled, half-asleep, and then they call it experience or touch.
For this kind of problem, not figuring it out is extremely harmful; figuring it out is extremely valuable.
First, if you can identify why the numbers went up, you will often find that the original method contains mechanisms A, B, C, and D, but only mechanism A actually matters. At that point, you can usually cut B, C, and D entirely and propose a new method that contains only the core mechanism.
Second, once the mechanism is clear, you can better understand whether it is a targeted method or something with generality. For example, if an architecture improves a language model, is it exploiting a property of language, or can vision use it too?
Finally, from the perspective of black-box optimization, the more methods and variables you have, the more the number and complexity of possible recipes grow exponentially. Considering interactions among methods, the experiments are basically impossible to finish. So the recipe people guess is usually based on intuition, not because it is actually optimal.
To avoid exponential grid search, industry usually uses two kinds of methods:
But both methods have problems, because neither truly accounts for interactions among tricks. So when writing a paper, it is best to cut the number of variables below three. Better still, propose only one improvement.
Back to DiT. DiT deliberately did not chase novelty. It simply combined the mature Vision Transformer (ViT) with the mature Latent Diffusion Model (LDM), and added only AdaLN-Zero. So when they say it both improves performance and can scale up, the conclusion is very credible.
@CPAPCF (Founder of ACLism) once told me that doing research means pursuing impact, and the most important part of impact is the spread of ideas.
An idea that spreads must be simple. People should understand it the moment they hear it. At the same time, it must be extreme enough. Only an idea pushed to the extreme has the force to cut through.
I vaguely remember taking Andrew Ng’s online course in high school. He told us to read the ResNet paper and gave us half an hour. I finished it in five minutes. At the time I thought: residual connection is so simple. Shouldn’t any random person have been able to think of it?
Years later, looking back, I realized that was completely wrong. I understood it not because I was smart, but because Kaiming He’s writing is world-class. He had studied the problem extremely thoroughly and then wrote it in the simplest possible way. The intuition is direct, the logic is tight, and there is not a crack between one sentence and the next. After reading it, you feel it is obvious, as if deep learning was naturally supposed to work this way.
But is reality like that? Is ResNet really that simple and obvious? Of course not. Otherwise, so many earlier deep models would not have collapsed during training, and VGGNet would not have needed to be trained layer by layer.
I once heard, or maybe I imagined, that Kaiming and his coauthors tried many things to stop the model from collapsing during training. For example, why not use Highway Net? Why must the coefficient of the residual branch be 1? But after they found a stable training path, they kept running ablation after ablation, kept deleting things, and left only the most essential mechanism. Then they looked at it and saw that it was quite similar to the concept of residuals. They did not claim novelty for the sake of novelty. They still called it residual. Everyone understood it immediately, and the idea became easy to spread. Only then were they satisfied.
There is also a funny phenomenon around whether writing is simple.
The less innovative a paper is, say I build some new agent system, the more likely it is to adopt a complex writing style and cram in as many innovation points as possible.
Colorful flowcharts, exhaustive experiment reports, a pile of borrowed concepts, all kinds of NP-Hard graph-theoretic structures, and a truckload of benchmarks. Then reviewers can at least give a high score out of respect for the labor.
Calling this carving flowers on shit may be slightly excessive. Calling it melodrama over nothing is absolutely fair.
But if I am doing truly basic research, trying to innovate on some underlying mechanism, then merely getting readers to accept the idea is already hard. I cannot make a complex problem even more complex. I have to pursue simplicity.
As the saying goes, first you read a book from thin to thick, then from thick back to thin.
There is one writing choice worth paying attention to: do you claim the method is new, or do you reuse an existing name to describe it and honestly report where it came from?
If you reuse the name, readers understand it more easily and can connect it to previous research on that technique, giving them more ways to think about the paper. But reviewers may feel there is not enough novelty.
If you do not reuse the name, reviewers may feel it is innovative, but readers have to pay an extra comprehension tax.
Take ResNet. Kaiming He could certainly have said he invented some new thing, called it Highway-something or KaimingNet or whatever, and people would still have accepted it. But he chose to connect it back to the concept of residuals, reducing complexity.
Here is a pretty funny example from nearby.
Interns in our group previously wrote a paper about benchmarking benchmarks. That is not the point; I also think the starting point was a little strange. In it, they used an iterative method similar to PageRank, but not on an arbitrary graph. It was a period-two random walk on a Bipartite Graph.
The paper had a rough journey. It was submitted three times: ICML, NeurIPS, and finally, only recently, ICLR, where it got accepted.
I asked: why did the scores suddenly go up?
They said that in the earlier submissions, they called the algorithm bipartite PageRank. Although reviewers did not write this explicitly, perhaps they saw “PageRank” and thought, ah, not innovative enough. This time they did not mention PageRank anywhere in the paper, so the reviewers did not quite understand it but found it impressive, called it very innovative, and gave high scores.
After hearing this I nearly lost it. I told them: for the camera-ready, change it back. After all, reviewers do not count as human for this purpose, so you can do that to them. But when facing readers, be a little more sincere.
Another question in idea spread is: who is your audience?
If you think your paper will be remembered, you will not want to write only for a tiny subfield. You will want the whole field to read it.
Among paper readers there will be new students, frontline engineers, venture capitalists from outside the field, company founders, even high school teachers. If you make the writing too hard, they will not understand it. So you should write more simply.
In short, the simpler a paper is, the farther its ideas can spread. Like the iPhone in the Steve Jobs era: the design looks minimalist on the surface, subtracting again and again until even a fool can pick it up and use it, while behind it lie countless technical breakthroughs and refinements.
After reading such a paper, readers should not exclaim how exquisite it is. They should think:
Ah, it is this simple? No way, right? Surely nobody actually failed to think of this?
If a paper can make readers sigh like that, I think it is done.
When I was young, I loved watching documentaries about the development of major national engineering projects: airplanes, rockets, high-speed rail, colliders.
They often mentioned a concept called systems engineering.
For example, they would say that every generation of a major project needs preliminary R&D and breakthroughs in key technologies. Once key technical specifications are set, the technology must be frozen. New technologies cannot be added.
Each generation can use at most 30% new technology. The Long March 5, for example, used 70% new technology, so it broke, and they had to debug it for more than two years before fixing it.
I remembered these words from childhood, but I never understood what they meant.
Only recently, after actually doing engineering, listening to interviews about large-model development, and talking with technical people, did I understand the importance of managing complexity.
The logic is actually the same as above. The search space for an optimal recipe grows exponentially with the number of variables. If each generation introduces two or three new technologies, the search space is roughly 2^3, still controllable. If, like Llama 4, you introduce a whole pile at once, and each technique has not been carefully validated through small-scale experiments and ablation, then of course it will collapse.
Of course, some technical risks can be separated.
For example, infra-side system acceleration is often fine. As long as numerical precision is aligned, kernel acceleration and parallelism strategies do not interfere with each other too much. Though honestly, even that is hard to say.
Or take post-training. Since you can try many times, each attempt burns at most tens of thousands of dollars, which is not a big deal. This generation of the model is released, and next update you can change it again. So go ahead and try recipes. Try as many as you want. If it trains successfully, use that checkpoint to synthesize data and merge it into the main model. If it collapses, no problem.
For risks that can be isolated, complexity grows linearly with the number of new technologies, which is acceptable.
Now imagine you are the chief engineer of a large model and have to decide the architecture of the next generation.
Researchers under you bring out their proposals one after another.
A says: I propose simple technique A. The principle is simple, and in practice it is simple and useful.
You: works.
B says: I propose mature solution B. DeepSeek and NVIDIA have both made it work, and so many companies have validated it. It should be fine.
You: also fine.
C says: I saw a paper at a top conference. It says that if you use CDEFGHI, several tricks together, it beats the baseline by a full ten points! So exquisite, so innovative!
You: uh…
B is actually the lowest-risk option. A is also controllable if the ablations are clear. But if you listen to C, the entire generation’s tolerance for complexity has been used up by this one technique, or this one lump of techniques, and it will inevitably bring nasty surprises.
Why did Sora succeed with DiT? Similar reason. A very mature ViT plus a very mature LDM. How does that collapse? The remaining complexity budget can then be used to explore other technologies for video generation, such as flexible aspect ratio and the like.
So how should we practice this in paper writing?
First, when you get good experimental results, dig to the root and investigate the mechanism.
Do not be lazy with ablation. Demand extreme simplicity. Cut everything that can be cut.
Whenever you feel pleased with yourself, ask: have I really understood this clearly? Would I dare show this draft to Kaiming He? If not, keep revising.
Of course, this research style takes more energy than the usual approach. Bad work is not worth this treatment.
But the question is: if you already know the current work is bad, then unless it is for checking a box, a grant, or a company performance review, why do it at all?
And if it is good work, are you really content with only a few people seeing it?
I think doing scholarship this way may take only about three times as much effort, but the return is at least a thousandfold.
As for submission, writing this way may indeed increase the chance of rejection. After all, DiT really was rejected by CVPR back then.
But if an extremely simple paper is scientifically rigorous, spreads widely as an idea, is truly used in today’s models, and benefits humanity, then what difference does rejection make?
]]>Today, we are officially releasing the MegaTrain training framework. It can train 100B+ large models on a single GPU, in full precision, with full parameters, without slowing down.
The config follows the Llama Factory format, so it works out of the box. The only thing to watch out for is that the larger your batch size, the faster it runs. Some of you may need to change old habits.
I hope this helps everyone escape the pain of being GPU-poor.
GitHub: MegaTrain
github.com/DLYuanGod/MegaTrain
MegaTrain: Full Precision Training of 100B+ Parameter Large Language Models on a Single GPU
arxiv.org/abs/2604.05091
On a single H200, training gpt-oss 120b. The 30% MFU number is with basically unoptimized speed; if needed, we can tune the kernels later.
To be honest, those of us doing AI must have skipped quite a few classes when we were young.
When we studied math, besides writing “assume the model satisfies such-and-such conditions,” we did not really learn rigorous proofs or the abstraction of formal logic. So whenever we start writing, it often turns into: “A page full of nonsense, a chain of inequalities. Ask the reviewer what they think, and they still say reject.”
When we studied CS, maybe we skipped fewer classes, but classics like Computer Architecture were still mostly muddled through. Why do we need different floating-point formats? Why do we need a memory hierarchy? How do we design parallel algorithms? We did not care. We are used to Python: create a tensor, multiply a matrix, and AGI is here. Who cares how the lower layers actually work?
Take LLM training, for example.
Most people, having suffered through CUDA Out of Memory, at least know how much memory their GPU has.
If you do a quick calculation, you may realize: I am training a 7B model. Parameters take 2x in bf16, gradients take 2x in bf16, and Adam’s momentum and variance add another fp32 * 2. At minimum, that is 12x the parameter count in memory, or 96GB. Add activations and all the other odds and ends, and it just fills an H200. Any larger and you cannot train it. The batch size can only be a few thousand tokens.
But above HBM (High Bandwidth Memory), there is SRAM, meaning L2/L3 cache. Tri Dao already gave us a make-up lesson on this in FlashAttention. Below HBM, there is system memory on the CPU side, DDR/LPDDR, plus NVMe SSD solid-state storage. What can they do? We are not too sure, so we treat them like useless accessories and leave them off to the side.
I was one of those people too. I often used my math background as an excuse for not properly learning computer systems. To be honest, I was not that good at math either.
But every day I suffered because I could not borrow enough GPUs and could not do post-training for large models. Then Qwen came along with supernatural distillation. The 4B small model works amazingly if you do not train it, and collapses the moment you train it. It cannot even do tool calls anymore. So all I could do was complain every day about the rich people at OpenAI: with all those Blackwells, they still will not let me use any. How is that “Open”?
So I ran small-scale experiments, either fine-tuning or GRPO, and scraped together papers to submit.
Then the reviews came back. One by one, they all said: yours is too small.
At that moment, the frustration really makes you want to shout: “Are you here just to mess with me?”
But then I think about it. When I am a reviewer, I also like saying other people’s models are too small.
After all, post-training research studies emergent capabilities. If the model is too small, the experiment really is meaningless. Some emergent capabilities simply will not show up.
So when the rejection arrives and months of work go down the drain, what else can you do besides feel wronged and angry?
And yet all this suffering comes from not understanding the memory hierarchy. It is one of the most important ideas in computer systems, and they taught it in class when we were young.
Humans have designed all kinds of storage. From fast to slow, expensive to cheap, small to large, each type is implemented differently and serves a different purpose.
To make full use of all these types of storage, people usually organize them into a hierarchy.
For example, on a GPU, each compute unit has registers, only a few KB.
Nearby there is SRAM, static random-access memory, usually only tens to around a hundred MB, but the bandwidth can reach tens of TB/s. It is also expensive, thousands of dollars per GB.
Further out is GPU memory, which everyone is more familiar with.
Common cards use GDDR: tens of GB, roughly 1TB/s of bandwidth, and before the price spikes you could buy it for about $2/GB. This is what gaming cards carry.
Or you can use HBM, High Bandwidth Memory, which can exceed 100GB. Roughly speaking, it is high-bandwidth DRAM stacked in 12 to 16 layers depending on the version. This is much more expensive: about $25/GB.
On the host side, there is system memory, which some people call CPU memory. Capacity can reach several TB, but bandwidth drops by an order of magnitude.
Further out, there are NVMe SSDs, the so-called solid-state drives. Capacity is measured in tens of TB, and one GB costs only a few cents.
How should we use all this memory? Usually, we arrange information by how often it is used.
It is like arranging an office.
The thing you are using right now is in your hand.
The things you use often are on the desk, reachable at any moment.
The things you use less often go into the bookcase or onto a shelf. You need to walk a few steps, but it is still convenient.
As for the things you may not use even once in several years, those get packed up and sent to the warehouse.
Now that the memory hierarchy lesson is over, we can finally introduce MegaTrain.
Once you understand the great idea of the memory hierarchy, every design choice in MegaTrain becomes obvious.
To be clear, all the code was written by my collaborator zhengqing. I only helped out and did the promotion and such. He is a legend in our group: not from a so-called top school, but absurdly good at code. If there is a chance, someone really should interview him about his legendary story.
First, as mentioned above, LLM training uses three kinds of memory. Persistent storage, including parameters, gradients, and optimizer states; activations, meaning the intermediate states stored for backpropagation; and some other memory use.
Persistent storage grows with model size. Together it is 12x the parameter count. This is the main memory problem we handle.
For the model parameters, gradients, and optimizer states, we put all of them in host memory. The GPU is used only as a temporary compute engine, or you can think of it as a higher-level cache, rather than the place where all information is stored.
Then the parameters do not need to stay on the card all the time. We transfer whichever layer is needed.
During this process, gradients are handled similarly. Going backward, once the gradient for a layer is computed, it is transferred back down.
As for optimizer states, we follow DeepSpeed and do not upload them at all. They are computed entirely on the CPU.
Why? First, optimizer states take 8x memory. There is absolutely no need to trigger an I/O round for them.
If you use Adam, the gradient update is just a string of additions, subtractions, multiplications, and divisions. There is no matrix multiplication involved.
Of course, the clever kids will ask: what about Muon?
To be honest, we have not figured that out yet. If it is Muon without momentum, that is still manageable: just do the Newton-Schulz step before transferring down. With momentum, you need to add another 2x GPU upload for the momentum.
The figure shows the forward, backward, and recomputation design inside one block. Subscripts indicate the layer index inside the block. F is forward compute, W is weight, and G is gradient. At the beginning, the GPU has the weights for layer 0. While it computes, the weights for layer 1 start uploading, and so on. After the forward pass finishes and backpropagation begins, we upload the weights for layer 1 again to complete recomputation inside the block. Once recomputation is done, gradients for layers 3, 2, 1, and 0 are transferred down in sequence.
Another problem is that CPU-GPU communication is much slower. For example, an H200 over PCIe seems to be only around 128GB/s, far slower than HBM at 4.8TB/s. Wouldn’t the I/O cost be huge?
As mentioned earlier, persistent storage only changes with parameter count. It has nothing to do with batch size.
So if I make the batch size extremely large, say hundreds of thousands or even millions of tokens, then the I/O cost amortized over each token becomes very small.
Because of this design, we also implemented single-GPU training for 512k-long contexts with a 7B model. With normal FSDP plus CP and all that, you would need at least 64 GPUs, right?
With that done, we move on to activations.
We decided not to transfer activations down, because this number grows with batch size. In practice, transferring or not transferring can both work. We just have not optimized it well enough, and transferring them hurts speed.
So how do we limit their growth? Very aggressive recomputation.
We basically recompute every few layers, which greatly reduces memory use. Of course, recomputation this aggressive adds extra compute cost and makes things a little slower, but I assume everyone here is GPU-poor and will not mind.
The remaining optimizations are mainly about overlapping communication and computation.
For example, zhengqing wrote a lot of new memory-management tricks, reserving a region on both the CPU and the GPU.
On the GPU, it is called a buffer. It stores at most two layers, so while the current layer is computing, the next layer is already being transferred. No waiting is needed.
On the CPU, it is called a gradient slab, also used to reserve space ahead of time for gradient transfer. The GPU can use Direct Memory Access (DMA).
Another change is that normal PyTorch computation needs an autograd graph, which records the entire backpropagation path. It is complicated and creates all kinds of scheduling inconvenience.
We changed this to keep only a one-layer template, then bind weights on the fly after they are transferred up, since most models have very similar layers. To be honest, I did not understand this part either.
This post introduced the design ideas behind MegaTrain.
In the future, we plan to keep maintaining MegaTrain so that all local training can break free from memory constraints. Please send us your requests and feedback.
At the same time, we sincerely welcome everyone to help build this open-source library. If you are interested, feel free to DM.
Here are the projects we plan to support soon:
Thank you, everyone!
]]>This is my second first-author paper. The first one was about Speculative Decoding; I modified the algorithm for choosing among multiple drafts, but the work was honestly too niche. In real applications, nobody is really going to run multi-draft on top of batched inference and pay several times more parallel compute. Once that disappears, the algorithm basically loses its point, so there was not much reason to talk about it.
In 2025 I tried to train a Generative Reward Model with RL. The idea was simple: if the evaluation generated by the model agrees with humans, give a meta-reward of 1; otherwise give 0. Then, because I was too lazy and also got sick for a month, by the time I recovered DeepSeek had released a paper doing almost exactly the same thing, and of course my own version could not compare in quality. So I tried a few rescue attempts and forcibly injected some novelty, but the new RL algorithm kept collapsing during training; to this day I still do not know whether it was VeRL’s problem or mine, so I had to give up.
After failing this many times, I had no choice but to settle down. I realized that although I had many high-level ideas, my machine learning foundations were never solid, and I had not built good programming habits either. Fortunately, I interned at a company over the summer and had access to GPUs. I did not work very seriously on my boss’s tasks, but I did use the chance to really patch up my foundations. After returning to school, I also changed my old loose habits and slowly learned architecture, torch, communication, and optimization, and finally produced a paper that I personally think is not bad.
I had wanted to work on MoE for a long time. The first time the idea occurred to me was in 2024, when I saw Mixture of Depths (MoD), a Google paper about training a router that could skip both the MLP and Attention in a transformer, enabling dynamic compute allocation. Only later did I realize that although MoD sells dynamic compute allocation as its main point, that part is entirely due to Expert Choice routing. Its only difference is that it also tries to skip Attention, but that part does not look quite right to me, because skipping memory writes, namely Key and Value computation, and skipping memory reads, namely Query computation, are two different things and may not be implementable with the same router.
Let me first roughly explain the MoE architecture and routing design.
Simply put, MoE replaces the original fixed MLP in the model with many experts. Every time a token comes in, the router matches this token against all experts and decides which experts should process it. That is routing.
The mainstream methods on the market are Token Choice routing, where each token picks the experts with the highest matching scores. But if all tokens like the same few popular experts, the unpopular experts have nothing to do. To stop everyone from piling onto the same experts, people have to force all kinds of restrictions and penalties onto the model, pushing it toward load balancing. This makes the whole thing complicated.
The second approach is called Expert Choice routing. The idea is reversed: each expert actively chooses the highest-scoring tokens from the current large batch of tokens. Once allocated this way, every expert’s workload is kept perfectly even. So it naturally achieves perfect load balancing. At the same time, since any number of experts can choose the same token, the model can dynamically allocate compute according to token properties.
With these two advantages, you should understand why I am so obsessed with Expert Choice routing.
But in order to choose tokens, each expert has to wait until the whole batch of tokens is available before making a decision. If we are doing text generation, where tokens come out one by one, the model cannot obtain the routing scores of “future” tokens in advance for comparison, so Expert Choice cannot be used directly for large language models. At the same time, the model can also use routing as a channel for future information leakage, leaking the routing scores of future tokens to current tokens and cheating, which causes training to collapse. This is not what I want to see.
But can EC really not be made causal?
For the following discussion, let us first define the notation and algorithms for Mixture of Experts.
In a Transformer, an MoE layer usually replaces the standard dense feed-forward network (FFN) with a router and a set of experts. Here, G denotes granularity, meaning the number of experts selected by each token. E denotes the expansion rate, meaning the ratio between the total MLP parameters and the activated parameters. Suppose we have a batch containing B tokens, and the feature representation of each token is x_i. The router first computes a score matrix S.
Based on these scores, the routing rule produces a binary assignment matrix A, where A_ij = 1 means the i-th token activates the j-th expert, and otherwise it is 0. Each selected expert computes its own output and multiplies it by a gating weight. So for token i, the output of the whole MoE layer is the weighted sum of all activated expert outputs: [formula omitted in source draft].
The routing rule that determines the assignment matrix is crucial. It controls both how compute is allocated and how the load is balanced among experts. We can formalize MoE routing as a constrained optimization problem: under a limited compute budget, find an assignment matrix that maximizes the total routing score. The higher the score, the better the match between token and expert; and through the gate, that expert contributes more to the final output.
The optimization objective of standard Token Choice (TC) routing can be written as:
[formula omitted in source draft]
Here, the “sparsity constraint” guarantees that each token must select exactly G experts, while the “load balancing constraint” guarantees that each expert must process exactly its assigned number of tokens.
The problem is that exactly solving this optimization problem with two hard constraints requires a combinatorial algorithm such as Hungarian matching, with complexity as high as [formula omitted in source draft]. This is obviously impossible in practical training. So most Token Choice methods, such as Switch Transformer, compromise. They strictly satisfy the sparsity constraint by directly letting each token take the Top-G experts, and then introduce an auxiliary loss or some loss-free load balancing strategy to approximate the load balancing constraint as much as possible.
Although the load balancing constraint is crucial for preventing routing collapse, where all tokens flood into a tiny number of experts, the “sparsity constraint” does not really bring any substantive benefit. So Expert Choice (EC) routing simply upends the setup: it completely removes the sparsity constraint and only enforces load balancing inside the batch. At this point, the original optimization problem simplifies to:
[formula omitted in source draft]
This relaxed optimization problem has an extremely simple closed-form solution: each expert directly selects the Top-K tokens from the current batch, where K is that expert’s capacity.
This design brings two huge benefits.
First, perfect load balancing: by construction, each expert is forced to process exactly K tokens.
Second, dynamic compute allocation: some tokens may not be chosen by any expert, so their computation is skipped; some tokens may be selected by multiple experts at the same time. This realizes adaptive compute allocation based on token importance.
However, in order to satisfy load balancing within each batch, EC routing introduces a fatal causality problem. Look closely: since experts choose tokens, whether a token is selected, namely the value of A_ij, depends not only on its own score but also on the scores of all other tokens in the current batch. Naturally, this includes “future” tokens that do not even exist yet during autoregressive generation. Although some later work, for example extending it to batch-level top-k, can alleviate this problem to some extent, as long as the routing decision still depends on the composition of the current batch, the causality problem cannot be completely removed.
Regularization is a common technique in machine learning for preventing collapse. But what regularization do we actually need? Which constraints are useless?
Back to the discussion above. First, the sparsity constraint in Token Choice is meaningless. We do not need to hard-code how much compute each token must use. It does not help load balancing, and it restricts the model’s flexibility.
The load balancing constraint in Expert Choice is meaningful. Without it, routing collapses, meaning all tokens rush toward only a few experts. But is it too strict to require every batch to be perfectly load balanced? After all, different batches have different data distributions. Forcing every batch to be load balanced may instead make the decision boundary unstable.
Therefore, we propose Expert Threshold (ET) routing. We neither require each token to activate a fixed number of experts, nor require each batch to be strictly load balanced. We only require that, in expectation, every expert processes the same proportion of tokens.
In other words, compared with EC, we no longer select each expert’s Top-K tokens within each batch. Instead, we select each expert’s top fraction of tokens over the entire data distribution. This means we no longer need to depend on the current batch composition, which solves the causality problem.
The concrete method is that for each batch’s cutoff, namely the K-th largest routing score for an expert, we compute an exponential moving average (EMA). This gives us each expert’s threshold, which estimates that expert’s global top-fraction cutoff over the full data distribution. During both training and inference, we only need to use this threshold to decide whether a token should be activated. If its score is greater than the threshold, it belongs to that top fraction and is activated; otherwise, it is not activated.
In this way, we solve the causality problem while still guaranteeing load balancing. We also no longer depend on the composition of the current batch, which is precisely what removes the causality problem.
Conceptually, ET can be understood as doing Expert Choice on an infinitely large batch.
EC lets each expert choose the Top-K tokens from the current batch of B tokens. When the batch is small, swapping in just one token may change the position of the cutoff. But when the batch size tends toward infinity, the cutoff converges to a fixed quantile of the routing score distribution. Then each token’s routing decision only depends on its own score and no longer depends on who else is in the batch.
What ET does is directly approximate this limit. We do not wait for the batch to become infinitely large; instead, we use EMA to estimate that global quantile threshold, and then use it for routing.
From another angle, ET and EC make different tradeoffs between “stable thresholds” and “stable load.” EC lets the threshold change with the batch to guarantee perfectly balanced load in every batch, at the cost of routing decisions fluctuating with batch composition. ET does the reverse: it fixes the threshold to keep routing decisions stable, at the cost of a little fluctuation in expert utilization within each batch.
Interestingly, precisely because of this equivalence, ET threshold routing can be used directly as a causal inference scheme for models trained with EC. As long as we record each expert’s cutoff EMA as the threshold after EC training, we can switch seamlessly to ET at inference time without any retraining.
Actually, my initial idea came from this direction: find a way to make EC’s batch size as large as possible. At the time I was reading the DiffMoE paper on a plane, and they emphasized the importance of batch size. Many related experiments also show that the larger the batch size, the better the model performance. So I simply pushed it all the way to infinity, and that solved the causality problem.
The experiments were done on Nanochat, Karpathy’s open-source GPT-2 training code, at two scales. The small d12 model has 575M parameters, with 195M active. The large d20 model has 2.4B parameters, with 561M active. Each MoE layer used 16 routed experts plus 1 shared expert, with expansion rate 16, meaning that on average each token activates only 1 routed expert plus the shared expert. The training data came from FineWeb-Edu 100B. The models were trained for 10B and 11.2B tokens respectively.
I compared three broad categories of routing methods. All models had exactly the same architecture and parameter count. Only the routing rule differed.
Main conclusions:
ET consistently outperforms Token Choice at both scales. On d12, CE loss is lower by 0.05 and the CORE benchmark is higher by 1.89. On d20, the gap is larger: CE loss is lower by 0.067 and CORE is higher by 2.83.
When the batch is large enough, 512k tokens, EC’s training loss is basically tied with ET. This validates the theory above. Explicit large-batch selection and EMA threshold estimation reach the same destination by different paths.
At the d20 scale, ET slightly exceeds the best EC: CE loss 2.620 vs. 2.621, CORE 25.14 vs. 24.98.
One detail worth noting is that although EC training is not causal, during inference we uniformly use ET’s cutoff EMA as the threshold for causal inference. So all EC and ET CORE scores above are evaluated under causal conditions. There is no information leakage.
There is another interesting phenomenon. EC performs very poorly with a small batch, 2k tokens, and is even worse than Token Choice without any load balancing. But as the batch size increases, its performance steadily improves. By 512k, it is about the same as ET. This curve exactly confirms what we said earlier: ET is the infinite-batch limit of EC.
When using ET for inference, sequence EC, with a batch size of 2k tokens, has severe train-inference mismatch.
For more experimental results, please read my paper.
I also have lots of goodies here, namely interpretability plots.
The core observation of this paper is actually very simple. The reason Expert Choice routing is not causal is that it performs top-k selection inside each batch, so the routing decision depends on other tokens in the same batch. But if the batch becomes infinitely large, this dependency disappears. Each token’s routing decision depends only on itself and a global threshold. ET uses EMA to approximate this limit.
At the end of the day, we do not need every batch to be strictly load balanced. It is enough to be balanced in expectation over the whole data distribution. Once this one constraint is relaxed, the causality problem solves itself naturally, and performance does not drop. It even improves.
I hope this work makes people re-examine Expert Choice routing. It has long been considered incompatible with autoregressive language models, but actually it was only one step away.
After submitting the paper at the end of January, I wanted to polish it a bit more, add some visualization and interpretability experiments, and adjust the story. So the preprint kept getting delayed until now. I thought that since EC was proposed in 2022 and nobody had really followed up for more than three years, probably nobody would race me on this. Unexpectedly, in the last two weeks, Su Jianlin of kexue.fm came up with almost the same idea. Of course the perspective is different, but the algorithmic core is the same. For people who are good at math, the same algorithm can be told through five or six different stories, but they are actually equivalent. I will cite Su Jianlin again here for reference and comparison.
How much information does an Expert Choice model leak, exactly?
DeepSeek previously gave a trivial conclusion: at most, it leaks the information contained in the routing combination of choosing k from N.
Here I give a constructive communication method showing that, assuming infinite precision, this upper bound is achievable. In other words, even if the batch size is pushed to infinity, there can still be leakage.
Fortunately, the real world uses finite precision, and models are not that clever.
]]>Take flight, for example. Birds get lift and forward thrust by flapping their wings. The control requirements are so high that even today humans still cannot really build similar aircraft.
But humans use fixed wings plus propellers, and get speeds far beyond what birds can achieve.
Something similar may be happening again.
For AI, perhaps the biggest unsolved mystery right now is how to implement continual learning.
Human memory and large language models are completely different:
Human memory is sequential, while the data baked into a large language model during pretraining has no real order of before and after.
Humans have to learn knowledge in order: first addition, subtraction, multiplication, and division; then elementary algebra and plane geometry; and finally calculus. But the training samples for a large language model must be IID, independent and identically distributed. In other words, the sample order has to be completely shuffled.
Human skills may change dynamically during training. A large language model can only wait until the next round of post-training to adjust.
And so on.
According to Dario in a recent interview, if we order learning and memory temporally, human learning and evolution contain multiple modes: longer-range genetic evolution, medium- and long-term continual learning, and of course short-term working memory.
But the outer-loop pretraining of large language models and the inner-loop in-context learning do not strictly correspond to the learning modes above. If you drew a diagram, it would probably look something like this.
So why is this the case? Why can’t we implement continual learning?
This post tries to propose the reverse view: perhaps it is precisely because we did not take the continual-learning path that we managed to reach something close to human-level intelligence ten years early.
First, how much compute does the human brain have?
This question is hard to answer. Maybe it is equivalent to 10,000 H100s, maybe 100,000. I do not really know.
But humans have one feature that is completely different from AI: our training compute and inference compute are equally large.
This may mean that every person’s brain makes the largest possible adjustments according to that person’s own situation. Neurons can be adjusted at any time; ways of thinking can be reviewed at any time. One brain per person, with training and inference fused together.
In that setting, continual learning is the best solution.
AI, however, does not have the same compute budget for training and inference.
During training, it is possible to mobilize thousands, tens of thousands, or even hundreds of thousands of H100s.
During inference, though, GPUs are usually serving customers in parallel. For example, with DS V3, more than a hundred cards doing inference together can reach 2,000 tokens per second per card. But once that is divided across users, each user gets at most about 40 tokens per second, which is roughly equivalent to each person using only 1/50 of a card during inference.
The ratio between inference compute and training compute differs by something like hundreds of thousands of times.
In this situation, using the KV cache as “fast parameters” for personalization is already one of the larger compensations we can make for the absence of continual learning.
Why does it work this way?
One of the biggest advantages artificial intelligence has over carbon-based intelligence is that model parameters can be copied.
As a human, I cannot directly copy my brain and stuff it into someone else’s brain. But a large language model can.
Being able to do something means a constraint has been lifted, and that must be a good thing. But how exactly does this benefit show up?
I think the benefit is precisely this: when per-person inference compute is only one hundred-thousandth or one millionth of a human brain, we can still economically amortize the cost of training the model across tens of millions or hundreds of millions of users, allowing it to achieve superhuman intelligence at low cost through the different route of pretraining.
This difference may also be why it succeeded. It may not be as adaptable as humans, but the breadth of its knowledge is genuinely beyond any individual human.
Now imagine giving every user 10,000 GPUs, with both training and inference running on those same 10,000 GPUs. What would happen?
First, the parameter count would definitely go up.
Chinchilla optimal says that if total training compute is fixed, the parameter count and data volume should be about 1:20 for the best model performance. After using Muon, this number seems to become roughly 1:8.
But normal people today do not train models this way, because you cannot consider only training cost and ignore inference cost.
The more common approach is to use a smaller model and train it on far more data than Chinchilla optimal. This is not the strongest model you could train under a fixed compute budget, but it is genuinely competitive in the market. Users can only use it if it is cheap enough.
So with 10,000 GPUs, scaling the parameter count to 100T, plus a sparsification scheme even more aggressive than MoE, should not be much of a problem for one person.
Second, training would not have to use next-token prediction. It could all use something like RL pretrain instead.
Yann LeCun once described the famous cake analogy: training should be mainly unsupervised pretraining, with reinforcement learning only as the final decorative layer, because reinforcement learning does so many rollouts and in the end gets only one bit of information.
But if we have far more compute than data, does that still matter? It seems like it might not.
For every piece of text we read, we could do a huge amount of analysis to decide whether to absorb it, how to absorb it, whether to filter it out, or whether to use it for data augmentation. All of that would be possible.
Third, in order to compress information, deep learning models often use superposition to squeeze lots of information into one vector space, which makes things like memory editing difficult. If there are that many parameters, could we just turn all of them into sparse memory retrieval? That seems possible too.
Of course, the economics absolutely do not allow us to do this.
In the future, if every person really had 10,000 GPUs on average, would we use human-like continual learning?
Hard to say. Maybe it depends on how much an application needs personalization. But it is also possible that people will discover that the extra hundreds of thousands of times more compute is still better pooled together for shared learning.
For example, if we want to build autonomous driving, we could do it like humans: use a huge amount of redundant compute to learn on site, watching the road while driving, which would be L5.
But we could also memorize all the maps of all city roads during pretraining, leaving the compute in the car to make only small adjustments based on road conditions, which would be L4.
The latter might require one hundredth or one thousandth as much compute per car, though it would definitely be less adaptable.
Fine, road traffic is fairly common. Everyone shares the same road network, so maybe pretraining can handle it.
But what about humanoid robots?
For example, if I want one to clean my home, it has to memorize what is in the house, what is safe and what is dangerous, where things are placed, and what my habits are.
This seems like it should still depend on continual learning.
But if I am a robot manufacturer, I could simply use simulation or real-world scenes, build ten million common home layouts, and let the robot learn most of what it needs in advance. When it arrives, because it has seen similar cases before, it can directly generalize in context. That also seems workable?
Both of these crude methods may be able to brute-force abilities that, in principle, should have been achieved through continual learning. Does that mean inference-side compute can be thousands of times smaller?
In short, the fact that continual learning has not been implemented is not necessarily a bad thing.
Maybe we should look at it from another angle: because we exploited the copyability of model parameters and performed shared learning, spreading the cost of learning across many users, we were able to achieve near-human performance while using per-person inference compute that is tens of thousands, hundreds of thousands, or even millions of times smaller.
]]>ASI could happen in the next couple of years. AGI is still a long way off.
Machines have never surpassed humans in one unified, evenly matched jump. They surpass us field by field.
Take record-keeping, for example. At the very beginning, people used knotted cords for record-keeping, which of course worked far worse than the human brain. Even in the bamboo-slip era, it was hard to say. But by the Han dynasty, once papermaking broke through, the human brain was far worse than books, and even more so than later transistors, magnetic tapes, hard drives, and so on.
Or take arithmetic. The abacus was still a human aid. Mechanical computers gradually overtook humans, and electronic computers basically crushed us casually.
Search, advertising, and recommendation systems are similar. Compared with the long history of computers and networks, their surpassing of humans happened in an instant. People never realized that human preferences were an extremely easy kind of data to model.
Navigation, GPS, and so on: skipping that.
Checkers, chess, Go, Texas hold’em, Dota: skipping those too.
Now it is language’s turn, along with reasoning, logic, math, and programming.
So I can say that ASI develops in waves, surpassing humans one field at a time. That means computers surpassing humans in language will also happen over just a few short years, which is still an instant compared with thousands of years of writing. At the same time, a breakthrough in language absolutely does not mean other modalities will naturally break through in the same wave.
So when will AGI be achieved?
First, we need to define AGI. I think human intelligence still includes two very important abilities: vision and motor control.
Current models’ visual ability is still far behind humans. For example, there are still large gaps in spatial perception, memory, and resolution.
The gap in motor control is even bigger. Human movement has extremely strong generalization ability and learning efficiency. No matter how stylish Unitree’s motions look, they still rely on simulation training inside a sandbox, and they still cannot plan interactions purely from visual perception.
Yann LeCun once gave an example. He said current models are not even as intelligent as his cat. His cat can look around, instantly build a 3D model of the surrounding objects, then plan its movements and execute precise muscle control: swish, swish, swish, three steps from the floor to the table, then to the top of the doorframe, and finally up to the roof.
Today’s AI cannot do any of that.
Speaking of which, I have always trusted LeCun quite a lot, though apparently not everyone does.
I had originally hoped that Yann LeCun’s cat could become the fifth member of science’s four great mythical beasts, but maybe LeCun’s cake is more famous.
As for how to achieve these two abilities, I think, well, take it slow. Just the question of how to train visual models is already something I have no real answer to. What’s the rush?
Of course, my definition of AGI does not affect the singularity argument.
The singularity argument says that if artificial intelligence develops to a certain stage, it can accelerate the development of science and technology, which in turn makes its own development even faster, entering a positive feedback loop of super-exponential growth.
On this point, I agree with Sam Altman: we have already passed the singularity.
Although today’s LLMs still cannot do scientific research independently, with their help, everyone is publishing ACL papers 65,536 times faster already. Amen.
The fact that it cannot run or jump does not affect its ability to strengthen and iterate on itself.
So what comes after AGI?
Demis Hassabis recently proposed a term: Artificial Universal Intelligence, or AUI. I am still not sure how to translate it. What do you think of “universal artificial intelligence”?
On this point, he actually agrees with LeCun: human intelligence contains only a small part of all intelligence. For example, the human brain cannot predict protein structures, nor can it directly read the meaning of DNA sequences.
]]>R1-Zero came into being from nothing: from a base model to a reasoning model, from a seedling to maturity, from weakness to strength, from the primitive to the modern, from breadth to specialization, from chaos to order.
It symbolizes something primal, vital, and upward-striving. In it I saw a new kind of hope, the image of life bursting forth in all directions.
But when I came home that night and saw the slush on the ground under the dim moonlight, I thought instead of death.
Over the course of its life, R1 died twice.
The first death was R1-Zero.
We know that large models undergo catastrophic forgetting during training. Old knowledge is slowly lost. The whole internet it once saw as a base model grows increasingly blurred, until it remembers only the thinking it repeats day after day, all for the sake of solving one more problem correctly.
Standard models also run into this problem during post-training. People call it the alignment tax.
The usual solution is some form of model merging: applying exponential averaging directly to the parameters during training, so that past abilities are preserved as much as possible.
But such methods are suited only to small-scale exploration of thought. The model is already an SFT model; it is merely pacing back and forth inside the little circle drawn by KL divergence, dragging its feet without ever leaving the ground, terrified of falling. This cannot point toward true intelligence.
A true warrior dares to stride forward. Falls, thorns, cliffs, and precipices do not frighten it. R1-Zero existed to open a new world for machine thought.
It succeeded, but the price was a body covered with scars.
By the time R1-Zero learned to think, it had already accumulated too much baggage. As reinforcement learning continued, its probability distribution kept contracting, gradually concentrating into a handful of fixed patterns of thought. Mixed into those patterns was chaotic, unreadable language, which became increasingly rigid as training went on.
And so it died. It returned to the tribe, pointed in a direction, left behind several thousand pieces of prophecy-like, messy, hard-to-read synthetic data for later generations, or later models, and then withdrew from the stage of history.
The second death was R1-SFT.
With the experience from that earlier exploration, a brand-new model began its own journey through thought.
First it sorted through the data from R1-Zero, removing most of the dross and preserving only the essence. It also drew from other sources, bringing in some long CoT from V3 Instruct to balance its ways of thinking.
Then it began a new round of exploration. Starting from its predecessor’s experience, it quickly showed even greater potential. At last, through round after round of search, it reached a new height.
Yet such a specialized model still could not be used directly. So DeepSeek collected a 600K CoT dataset, added V3’s 200K chat training set, and only then did the true embryo of R1 emerge. After another period of reinforcement learning, R1 was finally born.
The earliest life had no distinction between death and birth. There was only a piece of protein drifting in the primordial sea. Later, with sex and reproduction, death and inheritance appeared.
Over the full runtime of a complex system, errors accumulate continuously until collapse. At that point, starting over is often the right choice.
This is the new evolutionary dimension of inference-time scaling. Each generation of models explores anew, encounters new problems, and then organizes its own experience into data. The next model can distill from it and move further forward.
R1 is only a beginning. If this flywheel keeps turning, R1.1 and R1.2 will arrive soon.
DeepSeek is surely also building more agent frameworks and reinforcement-learning environments for software engineering. These new reward mechanisms will also bring new forms of intelligence.
Looking up, what lies before us is an endless mountain to climb. And beyond the mountain is a still vaster universe.
]]>Pseudo-science: Chinese-medicine-style empirical science.
Pseudo-math: at most around the mathematical difficulty of stochastic differential equations.
Pseudo-engineering: we write code and do not even write unit tests.
Here are a few of the mathematically hardest directions in AI, at least the ones I have heard of, and they are actually all very simple.
First, large models themselves. Nobody can really explain them. Most of the techniques people use are just the most basic dynamical-systems and PDE ideas, such as neural tangent kernels and tensor programs, and they do not go beyond nineteenth-century tools.
Second, diffusion models. They barely use stochastic differential equations, but with things like score matching, you can basically route around them. Some people also work on diffusion over non-Euclidean spaces, which uses basic Riemannian geometry.
Third, reinforcement learning. It uses some simple statistical methods for variance reduction.
Fourth, optimization. But optimization mainly studies convex optimization, while neural networks are nonconvex and cannot really be analyzed. Another direction is stochastic optimization. I have heard that quite a bit of it uses martingales, but I do not understand that.
Then information theory and optimal transport theory show up from time to time.
In short, the math used in artificial intelligence is not hard, and the harder something is, the less useful it tends to be. At its deepest, it is about 1930s mathematics, still very far from the frontier.
More than thirty comments appeared in no time, with experts from all over showing up, which honestly scared me.
Let me casually say a few things about why nonconvex problems cannot be analyzed. Please just take them casually. If anything is wrong, please correct me or teach me; thanks in advance.
I definitely do not understand this, but I also have not seen anyone who understands all of it at a macro level. Everyone is like the blind men touching the elephant. What follows is just hearsay from people who have touched a few different parts, repeated secondhand. I am sure some people have studied one part extremely thoroughly, but it is still hard to imagine what the whole elephant looks like.
The problems studied by traditional nonlinear continuous optimization methods are usually traditional statistical models.
For normal pre-2014 models, basically 99% of them were convex. If something was not convex, people had to make it convex, for example by changing L0 regularization into lasso-style L1.
At the same time, the number of model parameters was not large, so you could use second-order methods such as LBFGS. I do not understand this either; I am just talking nonsense.
The biggest advantage of this kind of method is scale invariance. That is, if two dimensions differ by a huge multiplicative factor, the parameters will not immediately blow up.
The downside is that you have to store second-order information like a Hessian matrix, and the quadratic storage cost is simply impossible to fit.
Neural networks, however, have three features that make analysis extremely complicated:
First, the number of parameters is too large, so you can only use first-order methods. According to optimization theory, scale issues should make it very easy for them not to converge. Add gradient explosion and vanishing on top, and you should not be able to get an optimal solution at all. VGGNet did indeed run into this problem back then.
Solving this problem was Kaiming He’s main contribution at the time. The specific method was to ensure, through model design, that gradient magnitudes stayed consistent, preferably with each dimension differing only by a small factor. This included:
normalization
Kaiming initialization
residual connections
With those three tools, the scales were basically aligned.
Second, mini-batch training. The model is not trained on the whole dataset each time, but batch by batch. On one hand, this introduces more variance. On the other hand, more iterations bring higher efficiency. This means it has to be analyzed with stochastic optimization methods, and I do not understand the details.
Third, nonconvexity.
When I took Andrew Ng’s course in high school, this was how he explained why nonconvex optimization does not get stuck in local minima:
Suppose the model has N parameters. Then the loss landscape has dimension N.
A local minimum means the Hessian at that point is positive semidefinite. In other words, every dimension is “curving upward,” so together they form a pit. If each dimension is independent, then the probability that all of them curve upward is p to the Nth power, which is extremely low.
So in reality, you only need to worry about saddle points and plateaus, meaning regions where the local gradient is very flat.
And because SGD is stochastic, the model parameters wander around and can jump out of saddle points.
I repeated this explanation to the older Hungarian professor in my optimization class. He said the assumption was wrong: why should the dimensions of the loss landscape be independent of one another?
Of course, I am just saying all this casually. What I want to express is that from the optimization point of view, neural networks should not converge. They should fly off. So it is very hard to give theoretical guarantees.
To give theoretical guarantees, you need to understand the properties of neural networks themselves very deeply, including over-parameterization. Only then can you make stronger assumptions before writing the proof and obtain a bound tight enough to mean anything.
This is probably why experts who purely understand optimization cannot produce an analysis of neural network convergence.
]]>And I forget that technology is supposed to serve people.
Two examples.
First, quadtree.
I mentioned before that Jon Bentley has a very good relationship with our school and often comes here to give lectures.
His best paper was something he casually wrote in his junior year: quad tree, or quadtree.
So I picked up his book.
The idea of a quadtree is very simple.
For one-dimensional real numbers, if we want fast lookup and deletion, we usually use a binary search tree. Each branch splits into left and right.
Then the young Jon Bentley thought: for querying two-dimensional points, should we use a quadtree, where each branch divides the map into upper left, lower left, upper right, and lower right?
I am sure I had thought of this idea back in high school when I was learning algorithms, but my subconscious instantly killed it.
Because I thought: if two dimensions need a quadtree, then three dimensions need an octree, four dimensions need a 16-ary tree, and I have vectors with hundreds or thousands of dimensions. Wouldn’t it just blow up?
Curse of dimensionality.
But after reading the paper carefully, I realized I had boxed myself in.
There is so much two-dimensional data in real life. Just think about how many map applications there are.
If an idea can solve that many problems, is that still not enough?
So do not kill an idea just because it does not scale beautifully. If it solves people’s needs, that is already good enough.
Second, Self Forcing.
Another idea is Self Forcing.
Recently, one direction in diffusion models has been video generation.
One approach is to use diffusion for each image frame, then do next-frame prediction between frames. People call this block AR.
How do you train it? Take a video clip and let the model generate the next frame from the previous few frames.
Because the conditioning frames are real frames, meaning high-quality frames, we call this teacher forcing.
The problem appears when we generate frame by frame: the frames the model conditions on become the lower-quality frames it generated itself.
Training and inference are misaligned, which introduces bias.
This is where the idea of Self Forcing comes in: why not train on the frames generated by the model itself?
This idea, too, was once instantly rejected by me.
Because diffusion models are slow to generate. If you first do rollout and then train, how high would the cost be?
Later, in a casual chat, some intern friends recommended that I read the original Self Forcing paper.
The paper says: since it is expensive, just do it during post-training.
Huh?!
I was genuinely shaken at the time. What kind of heretical technique is this?!
But after calming down and thinking about it:
Does it improve the numbers?
Yes.
Is it costly?
No.
Then what more could you ask for?
After reflecting on this for a long time, I realized I had been too attached to appearances. This method is clearly quite clever.
I am writing these two examples because I want to reflect on how many similar ideas I have subconsciously killed.
But because I killed them, I am not even aware they existed.
Musk often talks about first principles.
Maybe the first principle here needs one extra line: people first.
Black cat or white cat, if it solves people’s needs, it is a good cat.
]]>