Struct GgufLoader 

pub struct GgufLoader {
    pub ep_rank: usize,
    pub ep_world_size: usize,
    pub num_experts: usize,
    pub peak_memory_multiplier: Option<f64>,
}

Loads weights from a GGUF file, dequantizing every tensor to BF16 on the GPU.

Mirrors super::SafetensorsLoader so the two are interchangeable behind the super::WeightLoader trait and the serve call-site can pick one on file type.

Fields

ep_rank: usize

EP rank (0-based). Only used when ep_world_size > 1.

ep_world_size: usize

EP world size. When > 1, remote expert slices are skipped.

num_experts: usize

Total number of MoE experts in the model (for EP partitioning).

peak_memory_multiplier: Option<f64>

Override for the peak-memory multiplier in the pre-flight OOM check.

Implementations

impl GgufLoader

pub fn new() -> Self

Create a loader with no expert parallelism (loads all tensors).

pub fn with_ep(ep_rank: usize, ep_world_size: usize, num_experts: usize) -> Self

Create a loader with EP-aware expert filtering.

Trait Implementations

impl Default for GgufLoader

fn default() -> Self

Returns the “default value” for a type.

impl WeightLoader for GgufLoader

fn load( &self, model_dir: &Path, gpu: &dyn GpuBackend, oom_reserve_bytes: usize, ) -> Result<WeightStore>