pub const REDUCE_ATTN_BAR: usize = 17;Expand description
A 2-BYTE collective issued immediately before the real one, PROFILING ONLY. It is a
rendezvous: neither rank leaves it until both have arrived, so it absorbs the per-call
arrival jitter and charges the minimum-payload NCCL latency. The real 8 KB reduce that
follows therefore starts with both ranks synchronised, which is what makes
REDUCE_ATTN/REDUCE_MLP readable as network-and-kernel cost rather than “network
plus whatever the other rank was still doing”.
🪤 Aggregate rank skew being ~0 does NOT mean per-call wait is ~0 — the two ranks trade the lead call by call, so the NET cancels while every individual call still pays |jitter|. That is exactly why this probe exists and why the both-rank profile diff was not enough.