pub fn padded_batch_n(n: usize) -> usizeExpand description
The multi-sequence batch padding ladder — the SSOT for padded_n.
Batched decode pads the live sequence count up to a small set of captured
sizes so that (a) CUDA graphs (ATLAS_DECODE_GRAPHS_MULTISEQ) are keyed by
a handful of stable shapes instead of one per exact n, and (b) the batched
kernels see a bounded set of widths. Padding rows point at the dummy SSM
slot / dummy KV block and cost one wasted lane each.
This expression used to be duplicated at FOUR call sites
(decode_a2.rs:168, decode_b.rs:52 and :110,
phase_continue_prefills.rs:142 — the last one in a different crate), which
is exactly how the ladder would have drifted when a step was added. All four
now call here.
12 and 16 were added for the C=[1,2,4,8,16] concurrency work
(2026-07-25): previously any n ≥ 9 fell through to padded_n = n, so at
C=16 every distinct batch composition minted its OWN CUDA graph (n=9, 10,
… 16 each a separate capture) and the buffer-fit guards were computed on
exact n.
24 and 32 were added for native bs=32 (2026-07-30): n=17..32 now pads
to two stable graph shapes instead of minting one graph per exact n.
48, 64, 96 and 128 were added for native bs=64+ (2026-07-31,
wave-14a): the decode-metadata layout is now DERIVED from the serve
max_batch_size (spark_runtime::buffers::DecodeMetaLayout, rows =
max(32, bs), ceiling DECODE_META_MAX_ROWS), and
upload_batch_metadata_fixed ensures padded_n <= rows. Rungs above the
boot’s max_batch_size are unreachable (the scheduler admits at most
max_batch_size active sequences), so every bs<=32 boot never pads past
32 — byte-identical by construction. Rungs <=32 unchanged.
Above 128 the fall-through behaviour is unchanged (guarded downstream).