generate_streaming

Function generate_streaming 

Source
pub fn generate_streaming<F>(
    model: &dyn Model,
    prompt_tokens: &[u32],
    params: &SamplingParams,
    on_token: F,
) -> Result<GenerateResult>
where F: FnMut(u32),
Expand description

Generate response tokens with per-token callback.

Same as generate but calls on_token(token_id) after each token is produced (including the first token from prefill). The callback is synchronous — designed for the caller to send tokens through a channel without pulling in an async runtime dependency.