REDUCE_ATTN_BAR

Constant REDUCE_ATTN_BAR 

Source
pub const REDUCE_ATTN_BAR: usize = 17;
Expand description

A 2-BYTE collective issued immediately before the real one, PROFILING ONLY. It is a rendezvous: neither rank leaves it until both have arrived, so it absorbs the per-call arrival jitter and charges the minimum-payload NCCL latency. The real 8 KB reduce that follows therefore starts with both ranks synchronised, which is what makes REDUCE_ATTN/REDUCE_MLP readable as network-and-kernel cost rather than “network plus whatever the other rank was still doing”.

🪤 Aggregate rank skew being ~0 does NOT mean per-call wait is ~0 — the two ranks trade the lead call by call, so the NET cancels while every individual call still pays |jitter|. That is exactly why this probe exists and why the both-rank profile diff was not enough.