Struct ChatTokenizer 

pub struct ChatTokenizer { /* private fields */ }

Implementations

impl ChatTokenizer

pub fn from_model_dir( model_dir: &Path, eos_token_id: u32, supports_thinking: bool, model_type: &str, repo_root: Option<&Path>, disable_template_overrides: bool, ) -> Result<Self>

pub fn inner(&self) -> &Tokenizer

Returns a borrowed reference to the underlying HF tokenizer (for callers that need to drive low-level encode/decode directly).

pub fn encode(&self, text: &str) -> Result<Vec<u32>>

pub fn decode(&self, ids: &[u32]) -> Result<String>

pub fn decode_with_special(&self, ids: &[u32]) -> Result<String>

Decode without stripping special tokens. Use when tool calling is active — some tokenizers register <tool_call> as a special token, and skip_special would strip it, breaking tool call detection.

pub fn incremental_decode( &self, toks: &[u32], prefix_offset: &mut usize, read_offset: &mut usize, ) -> String

Incremental detokenizer (vLLM detokenize_incrementally scheme). Returns the newly-STABLE decoded bytes of toks since the last call and advances the offsets. Only the suffix window toks[prefix_offset..] is decoded each call (a handful of tokens since the last stable boundary), so streaming a full response is O(n) rather than re-decoding the whole history every token (O(n²)).

Byte-identical to decode(&all_toks) + trim_end_matches('\u{FFFD}') for byte-level BPE and SentencePiece tokenizers: a token’s decoded bytes do not depend on tokens before it, so decode(toks[prefix_offset..]) is exactly the corresponding suffix of decode(toks). A token whose window decode ends in U+FFFD (incomplete multibyte) is held back — the offsets stay put, so the window naturally extends until a later token completes the codepoint (same deferral the old trim_end_matches did). Uses the skip-special-tokens decode, matching the full-decode it replaces.

pub fn streaming_decoder( &self, skip_special_tokens: bool, ) -> StreamingDecoder<'_>

Create a stateful streaming decoder wrapper. Each step(token_id) returns Ok(Some(chunk)) when enough bytes have accumulated for valid UTF-8, or Ok(None) for incomplete multi-byte sequences.

pub fn apply_chat_template_jinja( &self, messages: &[Value], tools: Option<&[Value]>, enable_thinking: bool, disable_tool_steering: bool, ) -> Result<Vec<u32>>

Apply the Jinja chat template and encode to token IDs.

messages: Vec of serde_json::Value objects with role, content, and optionally tool_calls, reasoning_content. tools: Optional tool definitions (passed to Jinja context). enable_thinking: Controls <think> generation prompt behavior.

pub fn apply_chat_template_jinja_with_effort( &self, messages: &[Value], tools: Option<&[Value]>, enable_thinking: bool, disable_tool_steering: bool, reasoning_effort: Option<&str>, preserve_thinking: Option<bool>, ) -> Result<Vec<u32>>

pub fn apply_chat_template_openai( &self, messages: &[Value], tools: Option<&[Value]>, enable_thinking: bool, disable_tool_steering: bool, ) -> Result<Vec<u32>>

Apply the OpenAI-variant template (if available), falling back to the default. The OpenAI variant gates historical <think> wrappers on enable_thinking, preventing the model from learning a “always think” pattern during tool use.

pub fn apply_chat_template_openai_with_effort( &self, messages: &[Value], tools: Option<&[Value]>, enable_thinking: bool, disable_tool_steering: bool, reasoning_effort: Option<&str>, preserve_thinking: Option<bool>, ) -> Result<Vec<u32>>

pub fn apply_chat_template( &self, messages: &[(String, String)], enable_thinking: bool, _image_pad_counts: &[usize], ) -> Result<Vec<u32>>

Legacy apply_chat_template for callers that pass (role, content) tuples. Converts to JSON messages and delegates to apply_chat_template_jinja.

pub fn eos_token_id(&self) -> u32

pub fn think_end_token_id(&self) -> Option<u32>

pub fn supports_thinking(&self) -> bool

pub fn uses_deepseek_v4_encoding(&self) -> bool

pub fn image_pad_token_id(&self) -> Option<u32>

Encode the <|image_pad|> placeholder token and return its ID. Returns None when the tokenizer doesn’t have this token (text-only models). Cheap to call repeatedly — the underlying tokenizer caches single-token encodes.

pub fn video_pad_token_id(&self) -> Option<u32>

<|video_pad|>, the temporal sibling. None on a tokenizer without it — every text-only model, and any VL model that predates video.

pub fn expand_vision_pads( &self, tokens: Vec<u32>, pad_counts: &[usize], ) -> Vec<u32>

Post-process a rendered token sequence to expand <|image_pad|> placeholders. The Qwen3-VL / Qwen3.6 chat template emits exactly one <|image_pad|> per image, but the vision encoder produces grid_h * grid_w patches per image. At embed-injection time the server expects one pad token per patch so each patch’s embedding lands at the right hidden-state position — this helper does the fan-out.

pad_counts[i] is the number of patches the i-th image produces. Extra or missing <|image_pad|> occurrences (vs pad_counts.len()) pass through unchanged, matching counts are replicated in place.