Module mlx_int8 

MLX uint32-packed 8-bit weight format support.

Models published as mlx-community/<name>-MLX-8bit ship safetensors that pack each linear layer’s weights into a triplet:

• {base}.weight — U32 of shape [out_features, in_features / 4]. Each uint32 packs four unsigned 8-bit weight bytes (low byte = column 0, high byte = column 3 of the four-column block).
• {base}.scales — BF16 of shape [out_features, in_features / G]. One scale per group of G (default 64) columns.
• {base}.biases — BF16 of shape [out_features, in_features / G]. One additive bias per group.

The dequantization formula is affine:

w[r, c] = byte * scales[r, c/G] + biases[r, c/G]

where byte is the c%4-th byte of packed[r, c/4] (little-endian).

MlxInt8Weight holds the triplet on the GPU; dequantize_to runs the mlx_int8_dequant kernel to materialize a contiguous BF16 view, and gemv / gemm run the fused dequant-and-multiply kernels for the decode and prefill paths respectively.

Structs

MlxInt8Weight

One MLX-int8 quantized linear weight resident on the GPU.

MlxQuantConfig

Quantization metadata parsed from the model’s config.json.

Functions

gemv_gate_up

Dual-output GEMV: gate_y = gate @ x and up_y = up @ x in one kernel launch. Halves the x-side memory bandwidth and removes one kernel-launch round-trip per FFN. Both projections must share (out_features, in_features, group_size).