From the checkpoint config: hidden 2048, vocab 150,144 (untied LM head), 262,144 max positions, RoPE θ = 10,000. Expert grid is schematic.

differs from a standard pre-norm MoE attention, sliding window attention, full context (NoPE) MoE standard op