Expand description
Tokenizer wrapper using HuggingFace tokenizers + minijinja chat template.
Loads the model’s official Jinja template from tokenizer_config.json and
renders it with minijinja for byte-exact alignment with the model’s training
format. No fallback — if there’s no Jinja template, the model is misconfigured.
Structs§
- Chat
Tokenizer - Streaming
Decoder - Wrapper around tokenizers::DecodeStream that hides the generic parameters. O(1) per step vs O(n) for full re-decode.