Module build

Module build 

Source
Expand description

Binding one GLM MLP site for a rank: TP slicing of the dense/shared halves, EP selection of the routed experts.

Takes load closures rather than a WeightStore, for the same reason crate::layers::glm5next_dsa::build does: the slicing is then testable without a checkpoint, and the loader wiring stays one call site.

§🔴 The two axes are different, and mixing them is silent

  • TP splits the width of the dense FFN and the shared expert. gate_proj/up_proj are [inter, hidden] and split by ROW; down_proj is [hidden, inter] and splits by COLUMN. Slicing down_proj by row instead gives a well-formed [hidden/tp, inter] tensor and a plausible, wrong output.
  • EP splits the set of routed experts. An expert is never cut — it is owned whole. The router stays replicated so every rank selects the same ids.

Functions§

build_dense_mlp
TP-slice and upload one BF16 SwiGLU MLP — a dense layer, or a routed layer’s shared expert.
build_moe
Bind one routed MoE site for this rank: replicated router, TP-sharded shared expert, and exactly the local_experts routed experts this EP rank owns.

Type Aliases§

ExpertFn
One routed expert, by GLOBAL id.
LoadFn
A BF16/F32 tensor as host f32, by layer-relative name.