Struct MoeCutlassHostTables 

pub struct MoeCutlassHostTables {
    pub gate_packed: Vec<u64>,
    pub gate_sfb: Vec<u64>,
    pub gate_scale2: Vec<f32>,
    pub up_packed: Vec<u64>,
    pub up_sfb: Vec<u64>,
    pub up_scale2: Vec<f32>,
    pub down: Option<MoeCutlassDownHostTables>,
}

Host snapshots of the per-expert pointer/scale tables for the CUTLASS grouped path, owned by the MoeLayer whose device tables they mirror.

The CUTLASS grouped entry needs these on the HOST to build its per-group problem shapes. They are immutable once build_cutlass_grouped_sfb has run, so re-reading them per call was 6 copies x 2 calls x 47 MoE layers = 564 pointless D2H transfers per prefill. Only expert_offsets genuinely changes per call (it is produced by the expert sort), so only that one is still copied at dispatch time.

Why a layer-owned snapshot and not a process-global cache keyed on the device pointer: an address is not an identity. An in-process model swap (model_swap::swap) tears the outgoing model down — cuMemFree_v2 on every table cached here — and the incoming load’s near-identical cuMemAlloc_v2 sequence reuses those virtual addresses. A pointer-keyed static then hands the NEW model the OLD model’s expert weight pointers, and the grouped GEMM silently reads whatever now lives there as weights. Owning the snapshot on the layer makes staleness structurally impossible: the snapshot dies with the layer, with the model, at teardown.

Fields

gate_packed: Vec<u64>

gate_sfb: Vec<u64>

gate_scale2: Vec<f32>

up_packed: Vec<u64>

up_sfb: Vec<u64>

up_scale2: Vec<f32>

down: Option<MoeCutlassDownHostTables>

None when the checkpoint has no down-projection scale table — the grouped down branch is unreachable in that case.

Implementations

impl MoeCutlassHostTables

pub fn snapshot( gpu: &dyn GpuBackend, num_experts: usize, gate_packed: DevicePtr, gate_sfb: Vec<u64>, gate_scale2: DevicePtr, up_packed: DevicePtr, up_sfb: Vec<u64>, up_scale2: DevicePtr, down: Option<(DevicePtr, Vec<u64>, DevicePtr)>, ) -> Result<Self>

Snapshot the grouped-path tables at load. The SFB pointer vectors are taken by value because build_cutlass_grouped_sfb constructs them on the host in the first place — reading them back from the device would re-derive data this function’s caller already holds. The packed/scale2 tables exist only on the device (uploaded by the pointer-table build), so those are copied down once here.