From the checkpoint config: hidden 5120, vocab 151,936, 1M max positions, final logit softcap 30. Expert grid is schematic.

Beam-specific attention, sliding window attention, full (NoPE) MoE standard op