clamp_trainable_to_vocab

Function clamp_trainable_to_vocab 

Source
pub fn clamp_trainable_to_vocab(
    trainable: &[u32],
    r: usize,
    vocab: usize,
) -> Result<(Vec<u32>, usize)>
Expand description

Clamp trainable ids to the served vocab, preserving list order (the delta tensor’s rows align positionally to it).

Returns (kept_ids, skipped_extension_count):

  • idx >= r → hard error (id outside the adapter’s own [R,h] embedding).
  • vocab <= idx < r → vocab-extension token the served tokenizer can’t emit; dropped and counted (caller warns).
  • idx < vocab → kept, but a kept id smaller than a previously-kept id is a hard error: PEFT appends extension tokens as the largest indices with delta rows in the same order, so the kept prefix must stay positionally aligned to the delta rows after the extension tail is dropped.