gdn_verify_fused_conv_kn_batched

Function gdn_verify_fused_conv_kn_batched 

Source
pub fn gdn_verify_fused_conv_kn_batched(
    gpu: &dyn GpuBackend,
    kernel: KernelHandle,
    conv_state: DevicePtr,
    new_input: DevicePtr,
    weight: &DenseWeight,
    output: DevicePtr,
    conv_state_inter: DevicePtr,
    num_tokens: u32,
    d_inner: u32,
    d_conv: u32,
    qk_channels: u32,
    head_dim: u32,
    input_stride: u32,
    output_stride: u32,
    inter_stride: u32,
    l2_eps: f32,
    n_seq: u32,
    conv_state_seq_stride: u32,
    input_seq_stride: u32,
    output_seq_stride: u32,
    inter_seq_stride: u32,
    stream: u64,
) -> Result<()>
Expand description

Fused generic-K DFlash-verify conv1d+L2norm: ALL K draft positions in one launch, with every per-token conv-state rollback snapshot written inline to a strided intermediates array (replaces the per-token conv1d_update_l2norm ×K + copy_d2d ×K sequence — 34 serialized ops at K=17). conv_state is left holding the committed (post final-position) window, which the kernel also duplicates as snapshot K-1, so the caller issues NO copies.

Same numerics as the per-token path (identical accumulation order under –fmad=false; the K=2 twin is proven bit-identical by gdn_verify_fused_microtest).

Kernel: gdn_verify_fused_conv_kn(conv_state, new_input, weight, output, conv_state_inter, num_tokens, dim, d_conv, qk_channels, head_dim, input_stride, output_stride, inter_stride, l2_eps) Grid: (ceil(dim/256), 1, 1) Block: (256, 1, 1) BATCHED verify conv: n_seq sequences x num_tokens positions in ONE launch.

Step 1 of batched speculative decoding. spec_step.rs:94 currently calls decode_verify one sequence at a time, so MTP at C=n runs n full model forwards and re-reads ~9.6 GB of weights n times — measured as a 3.4% LOSS at C=2 and a HALVING of C=4. One launch over gridDim.y = n_seq is the first piece of making n*(K+1) rows share a single weight read.

Bit-identical to n separate gdn_verify_fused_conv_kn calls: each sequence’s conv window is independent, so only the base addresses differ.

Grid: (ceil(d_inner/256), n_seq, 1) Block: (256, 1, 1)