pub fn prewarm(stream: u64)Expand description
Force cuBLASLt’s one-time costs at MODEL LOAD instead of on request 1.
The lazy ctx() means the first GEMM pays cublasLtCreate, the 64 MB
workspace alloc, and — the expensive part — the library’s kernel-image
load and heuristic warm-up. Measured on the 35B flagship (2026-08-22,
dgx1): the first in-serve request read ~0.9 s slower than warm requests
once QKVZ routed through cuBLASLt, and cold TTFT is a headline metric.
One 64x64x64 BF16 GEMM here is trivial GPU work and moves that cost to
load time, where it overlaps the operator’s mental model of “loading”.
Never fails the serve: a pre-warm failure is logged and swallowed — the lazy path remains and request 1 simply pays the old cost.