A Kind of Idea I Kill Off Subconsciously

I often make one kind of mistake: when an idea lacks mathematical beauty, I subconsciously kill it off.

And I forget that technology is supposed to serve people.

Two examples.

First, quadtree.

I mentioned before that Jon Bentley has a very good relationship with our school and often comes here to give lectures.

His best paper was something he casually wrote in his junior year: quad tree, or quadtree.

So I picked up his book.

The idea of a quadtree is very simple.

For one-dimensional real numbers, if we want fast lookup and deletion, we usually use a binary search tree. Each branch splits into left and right.

Then the young Jon Bentley thought: for querying two-dimensional points, should we use a quadtree, where each branch divides the map into upper left, lower left, upper right, and lower right?

I am sure I had thought of this idea back in high school when I was learning algorithms, but my subconscious instantly killed it.

Because I thought: if two dimensions need a quadtree, then three dimensions need an octree, four dimensions need a 16-ary tree, and I have vectors with hundreds or thousands of dimensions. Wouldn’t it just blow up?

Curse of dimensionality.

But after reading the paper carefully, I realized I had boxed myself in.

There is so much two-dimensional data in real life. Just think about how many map applications there are.

If an idea can solve that many problems, is that still not enough?

So do not kill an idea just because it does not scale beautifully. If it solves people’s needs, that is already good enough.

Second, Self Forcing.

Another idea is Self Forcing.

Recently, one direction in diffusion models has been video generation.

One approach is to use diffusion for each image frame, then do next-frame prediction between frames. People call this block AR.

How do you train it? Take a video clip and let the model generate the next frame from the previous few frames.

Because the conditioning frames are real frames, meaning high-quality frames, we call this teacher forcing.

The problem appears when we generate frame by frame: the frames the model conditions on become the lower-quality frames it generated itself.

Training and inference are misaligned, which introduces bias.

This is where the idea of Self Forcing comes in: why not train on the frames generated by the model itself?

This idea, too, was once instantly rejected by me.

Because diffusion models are slow to generate. If you first do rollout and then train, how high would the cost be?

Later, in a casual chat, some intern friends recommended that I read the original Self Forcing paper.

The paper says: since it is expensive, just do it during post-training.

Huh?!

I was genuinely shaken at the time. What kind of heretical technique is this?!

But after calming down and thinking about it:

Does it improve the numbers?

Yes.

Is it costly?

No.

Then what more could you ask for?

After reflecting on this for a long time, I realized I had been too attached to appearances. This method is clearly quite clever.

I am writing these two examples because I want to reflect on how many similar ideas I have subconsciously killed.

But because I killed them, I am not even aware they existed.

Musk often talks about first principles.

Maybe the first principle here needs one extra line: people first.

Black cat or white cat, if it solves people’s needs, it is a good cat.