Struct SamplingParams 

pub struct SamplingParams {

    pub temperature: f32,
    pub top_k: u32,
    pub top_p: f32,
    pub top_n_sigma: f32,
    pub min_p: f32,
    pub logit_bias: Vec<(u32, f32)>,
    pub repetition_penalty: f32,
    pub repetition_penalty_window: u32,
    pub presence_penalty: f32,
    pub frequency_penalty: f32,
    pub lz_penalty: f32,
    pub dry_multiplier: f32,
    pub dry_base: f32,
    pub dry_allowed_length: u32,
    pub dry_sequence_breakers: Vec<u32>,
    pub max_tokens: usize,
    pub stop_token_ids: Vec<u32>,
    pub seed: Option<u64>,
}

Sampling parameters for a request.

Fields

temperature: f32

Temperature (0.0 = greedy).

top_k: u32

Top-k: keep only the k highest-probability tokens before sampling. 0 = disabled (use all tokens).

top_p: f32

Top-p (nucleus): keep smallest set of tokens whose cumulative probability >= p. 1.0 = disabled.

top_n_sigma: f32

Top-n-sigma: filter tokens in logit space before temperature scaling. Keep only tokens with logit >= mean - n*sigma. Temperature-invariant. 0.0 = disabled. Recommended: 1.0 for NVFP4 models.

min_p: f32

Min-p: keep tokens with prob >= min_p * max_prob (post-softmax). 0.0 = disabled. Recommended: 0.05-0.1.

logit_bias: Vec<(u32, f32)>

Per-token logit bias: (token_id, bias_value) pairs. Applied additively to raw logits before any filtering.

repetition_penalty: f32

Repetition penalty: multiply logits of previously-seen tokens. 1.0 = disabled. Recommended: 1.05-1.1.

repetition_penalty_window: u32

Repetition penalty window: only consider the last N tokens. 0 = full history (default). Recommended: 64 for long-form generation.

presence_penalty: f32

Presence penalty (OpenAI-style): flat additive penalty for each token that appeared at least once. Range [-2.0, 2.0], 0.0 = disabled.

frequency_penalty: f32

Frequency penalty (OpenAI-style): additive penalty proportional to occurrence count. Range [-2.0, 2.0], 0.0 = disabled.

lz_penalty: f32

LZ penalty: penalize tokens that extend repeated n-gram patterns. 0.0 = disabled. 1.0 = moderate (default). Based on arXiv:2504.20131.

dry_multiplier: f32

DRY (Don’t Repeat Yourself) penalty multiplier. From llama.cpp. Uses Z-algorithm O(n) sequence matching with exponential penalty. 0.0 = disabled. Recommended: 0.8.

dry_base: f32

DRY penalty base for exponential scaling. penalty = multiplier * base^(match_len - allowed_len). Recommended: 1.75.

dry_allowed_length: u32

DRY minimum match length before penalty applies. Sequences shorter than this are ignored. Recommended: 2.

dry_sequence_breakers: Vec<u32>

DRY sequence breaker token IDs. Delimiters (newlines, colons, quotes, braces) that reset sequence tracking. Critical for JSON/tool call output where structural tokens repeat.

max_tokens: usize

Maximum tokens to generate.

stop_token_ids: Vec<u32>

Stop token IDs.

seed: Option<u64>

Seed for deterministic sampling. When Some, the RNG is seeded with this value for reproducible output. None = non-deterministic (thread_rng).

Implementations

impl SamplingParams

pub fn greedy(max_tokens: usize) -> Self

Greedy sampling with a max token limit.

pub fn is_greedy(&self) -> bool

Trait Implementations

impl Clone for SamplingParams

fn clone(&self) -> SamplingParams

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for SamplingParams

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.