draft_conf_tau

Function draft_conf_tau 

Source
pub fn draft_conf_tau() -> f32
Expand description

A draft token proposer for speculative decoding.

The engine calls propose() after each target decode to get draft tokens, then verifies them with the target model. after_verify() lets the proposer trim state (e.g., KV cache) based on how many drafts were accepted. Confidence floor for submitting drafts to verification (ATLAS_MTP_DRAFT_CONF, default 0.0 = disabled). When the drafter’s chain confidence (min top-1 softmax prob across the drafts of one propose) is below this, the drafts are discarded and the next step decodes serially — skipping a verify that would most likely reject. Economics at K=1 on the 35B MoE: verify ≈ 35 ms for 1+acc tokens vs decode+propose ≈ 21 ms for 1, so a draft is only worth verifying when p(accept) ≳ 0.66 — the threshold to calibrate around. Staged OFF until its measured A/B (same discipline as ATLAS_SNAP_EVICT_ALPHA).