Figure 1 · Muse Spark Open · decoder layer

Each block sits between two normalizations

One of the 62 identical decoder layers, read top to bottom. The vision and audio encoders are not shown.

Sandwich normalization: x + post_norm(F(norm(x))), around the attention block and again around the expert block.

a standard step the second normalization, after the block