Figure 2 · Muse Spark Open · attention block

Attention normalizes twice more, then gates each head

Inside the attention block of one layer, read top to bottom. The heads are drawn as one.

Rotary position is added only in windowed layers (Figure 4). The gate comes from the same projection as Q, K and V.

a standard attention step normalization added inside attention, and the gate