Struct FastSafetensorsLoader 

pub struct FastSafetensorsLoader {
    pub ep_rank: usize,
    pub ep_world_size: usize,
    pub num_experts: usize,
    pub peak_memory_multiplier: Option<f64>,
    pub skip_activation_scales: bool,
    pub skip_mtp: bool,
    pub try_direct_io: bool,
    pub direct_io_tensor_cap: usize,
    pub prefetch_shards: bool,
    pub skip_vision: bool,
}

Pure-Rust InstantTensor-style loader. Same public shape as crate::weights::SafetensorsLoader.

Fields

ep_rank: usize

ep_world_size: usize

num_experts: usize

peak_memory_multiplier: Option<f64>

skip_activation_scales: bool

Skip the W4A4 *.input_scale activation scales at load.

ModelOpt NVFP4 checkpoints ship one 0-dim F32 scalar per quantized projection. On a 512-expert model that is ~74k four-byte allocations, each taking a full allocation granule — GBs of padding for values Atlas never reads, because it serves w4a16 (BF16 activations) and the NVFP4 loader already treats the key as optional.

OPT-IN: step3p7 reads this key on its own path, so it must stay off unless the model’s loader is known not to need it.

skip_mtp: bool

Skip mtp.* tensors at load.

For models whose loader deliberately does not build an MTP head, uploading its weights is pure waste — on Qwen3.8-Flash-Next that is a 1.49 GB expert shard plus the MTP backbone, held resident while the KV cache goes without.

OPT-IN: a model that DOES build an MTP head must keep them, so this is set only where load_mtp_weights is known to return None.

try_direct_io: bool

When true (default), attempt O_DIRECT; fall back to buffered reads if the filesystem rejects it (tmpfs, overlayfs, some FUSE backends).

direct_io_tensor_cap: usize

Per-shard heuristic cap: if a shard’s tensor count exceeds this, we skip O_DIRECT for that shard and fall back to buffered + pipelined reads even when Self::try_direct_io is true.

Motivation: O_DIRECT’s 4 KiB-aligned per-tensor pread has a fixed syscall + copy overhead that kernel readahead amortises for free on the buffered path. Benchmarks on GB10 showed buffered wins above ~5k tensors/shard; O_DIRECT wins below. Set to usize::MAX to disable.

prefetch_shards: bool

When true, advise the kernel to read a whole buffered shard sequentially before the per-tensor copy loop starts. This helps NFS mounts where many small tensor reads defeat normal readahead.

skip_vision: bool

Skip a multimodal checkpoint’s vision tower.

Set by the caller from ModelWeightLoader::binds_vision_encoder(): false by default, true only when the model’s loader is a text-only port that will never bind the tower. Reading it anyway costs the full tower in unified memory (1.05 GiB/rank on GLM-5.3’s checkpoint) from load time until build_model frees it — which is after the inference -buffer preflight has already refused the serve.

Implementations

impl FastSafetensorsLoader

pub fn new() -> Self

pub fn with_ep(ep_rank: usize, ep_world_size: usize, num_experts: usize) -> Self

Trait Implementations

impl Default for FastSafetensorsLoader

fn default() -> Self

Returns the “default value” for a type.

impl WeightLoader for FastSafetensorsLoader

fn load( &self, model_dir: &Path, gpu: &dyn GpuBackend, oom_reserve_bytes: usize, ) -> Result<WeightStore>