Kimi K3 Just Changed The Way AI Models Are Built

01 The problem

AI models remember too much.

What if I told you the most important announcement in Moonshot's Kimi K3 release last week isn't how big or cheap it is but rather 2 important ways they built the model.

Every time a model produces a new token, it looks back at every token that came before it. In order to do this it keeps a running memory of everything it's seen so far, and that memory grows with the conversation. By token 100,000, there are 99,999 look-backs...

That's where the economics start getting pricey. Doubling the document size more than 2X's the cost. That's the reason why "long context windows" (e.g. 1 million) has always been advertised by frontier labs. It's treated like a luxury good.

Look at how Kimi K3 treats context size below. Watch the look-backs pile up in a traditional LLM, then flip to Kimi Delta Attention and observe memory staying fixed in size.

02 Kimi K3's Architecture

Kimi K3 uses a novel memory trick: Kimi Delta Attention

Moonshot's answer to expensive memory? Kimi Delta Attention (KDA), and its signature trait is refusing to let the memory grow… at all. Instead it compresses your history into a fixed-size memory such that the model carries the same amount of state at token 1,000 as it does at token 1,000,000.

Managing that fixed memory is where something called the delta rule comes in. When new information arrives, the model works out what's different from what it already knows and corrects the record in place. Think of this like a business's CRM: when a customer's details change, you update their individual record, you don't create a new copy every time and pray you find the latest version 2 days later.

But a fixed memory obviously has to forget something. Normally this is where the design would fall apart, and for earlier designs, it mostly was. But Kimi K3 does something pretty extraordinary.

It's called fine-grained forgetting, basically every important feature of the model's memory gets its own control panel. This means the model can choose to throw away useless conversational filler (quickly) while holding tightly to a key detail from half a million tokens back. Per Moonshot, this control is what buys decoding speeds up to 6.3× faster at million-token lengths, without the model losing quality. Pretty cool stuff. Here's a visual of how this looks between a regular LLM vs. Kimi K3.

03 Memory Compression

Kimi's Smart Compression Design prevents intelligence loss.

Now the obvious objection (and probably a question you're asking): "isn't compressed memory lossy. Doesn't it forget things? My Claude always forgets things between conversations!"

Yep! Which is why K3 doesn't rely on compression alone. KDA layers are interleaved with a smaller number of conventional, full-attention memory layers that can still perform exact lookups across your entire context. So while compression is used to manage the bulk of everyday work; Kimi K3 can still periodically look up everything. That design is what lets Moonshot claim their model is frontier-level and incredibly efficient.

04 The training breakthrough

How K3 Achieved 2.5x Training Efficiency (beating Claude and GPT)

KDA is only half the story. At the end of last year the team behind K3 wrote a paper on Attention Residuals (AttnRes) focusing on how to improve model training. It was focused on addressing a very specific problem: in very big models, depth (aka parameter count) causes the model to lose focus.

Moonshot calls this PreNorm dilution. Every layer of the model's weights adds a contribution to its shared stream of "consciousness" (I don't mean that literally of course but the analogy helps). As that stream grows, each new layer's voice gets proportionally quieter e.g. like trying to be heard in an increasingly crowded room. At 2.8T in model size, K3 should theoretically suffer this same fate, but it doesn't.

AttnRes rewires the connections so layers can selectively retrieve earlier representations instead of uniformly piling everything up, so deep layers keep being "heard". Per Moonshot's technical report they get roughly a 25% gain in training efficiency for under 2% extra compute. In a world where frontier training runs are priced in the hundreds of millions, that is a lot of money saved. Together with KDA, these 2 designs underpin the company's claim of 2.5× better scaling efficiency vs. Kimi K2.

05 For builders

So why does this matter for you?

Because you don't have to use K3 (or ever touch an open-weights model) for this release to change how you might build your product. If you're building any sort of AI feature, model or tool that reads long documents, holds long conversations, or runs agents with memory, the architecture above could be drawn upon to influence how you build it.