Figure 4 · Muse Spark Open · attention across the layers

Three of every four layers keep only a 2048-token window

Which layers attend to what, and what that does to the KV cache at a 1M-token context. Heights are not to scale.

Pattern [2048, 2048, 2048, whole], counted down from layer 62. The KV share assumes every layer stores the same KV per token.

windowed layer: attends 2048 tokens, keeps 2048 tokens of KV whole-context layer: attends every token, keeps all of them