padded_batch_n

Function padded_batch_n 

Source
pub fn padded_batch_n(n: usize) -> usize
Expand description

The multi-sequence batch padding ladder — the SSOT for padded_n.

Batched decode pads the live sequence count up to a small set of captured sizes so that (a) CUDA graphs (ATLAS_DECODE_GRAPHS_MULTISEQ) are keyed by a handful of stable shapes instead of one per exact n, and (b) the batched kernels see a bounded set of widths. Padding rows point at the dummy SSM slot / dummy KV block and cost one wasted lane each.

This expression used to be duplicated at FOUR call sites (decode_a2.rs:168, decode_b.rs:52 and :110, phase_continue_prefills.rs:142 — the last one in a different crate), which is exactly how the ladder would have drifted when a step was added. All four now call here.

12 and 16 were added for the C=[1,2,4,8,16] concurrency work (2026-07-25): previously any n ≥ 9 fell through to padded_n = n, so at C=16 every distinct batch composition minted its OWN CUDA graph (n=9, 10, … 16 each a separate capture) and the buffer-fit guards were computed on exact n.

24 and 32 were added for native bs=32 (2026-07-30): n=17..32 now pads to two stable graph shapes instead of minting one graph per exact n.

48, 64, 96 and 128 were added for native bs=64+ (2026-07-31, wave-14a): the decode-metadata layout is now DERIVED from the serve max_batch_size (spark_runtime::buffers::DecodeMetaLayout, rows = max(32, bs), ceiling DECODE_META_MAX_ROWS), and upload_batch_metadata_fixed ensures padded_n <= rows. Rungs above the boot’s max_batch_size are unreachable (the scheduler admits at most max_batch_size active sequences), so every bs<=32 boot never pads past 32 — byte-identical by construction. Rungs <=32 unchanged. Above 128 the fall-through behaviour is unchanged (guarded downstream).