Module tokenizer 

Tokenizer wrapper using HuggingFace tokenizers + minijinja chat template.

Loads the model’s official Jinja template from tokenizer_config.json and renders it with minijinja for byte-exact alignment with the model’s training format. No fallback — if there’s no Jinja template, the model is misconfigured.

Structs

ChatTokenizer

StreamingDecoder

Wrapper around tokenizers::DecodeStream that hides the generic parameters. O(1) per step vs O(n) for full re-decode.