Struct NcclBackend 

pub struct NcclBackend { /* private fields */ }

NCCL communication backend for multi-GPU / multi-node EP.

Implementations

impl NcclBackend

pub fn new( rank: usize, world_size: usize, master_addr: &str, master_port: u16, stream: u64, recv_capacity: usize, ) -> Result<Self>

Initialize NCCL with TCP bootstrap.

Rank 0 listens on master_addr:master_port, generates a unique ID, and sends it to all connecting ranks. All ranks then call ncclCommInitRank which internally synchronizes. recv_capacity is the largest all-reduce payload this backend will ever be asked to carry, in bytes — compute it with required_recv_bytes from the serve configuration. It is only consulted when world_size == 2 (the send/recv fast path); other world sizes reduce in-place via ncclAllReduce and allocate no receive buffer.

Trait Implementations

impl CommBackend for NcclBackend

fn all_reduce(&self, ptr: u64, bytes: usize) -> Result<()>

All-reduce: sum across all ranks, result on all ranks.

fn all_reduce_async( &self, ptr: u64, bytes: usize, compute_stream: u64, ) -> Result<()>

Async all-reduce using GPU-side event synchronization.

fn register_buffer(&self, ptr: u64, bytes: usize) -> Result<u64>

Pre-register a GPU buffer with the communication backend.

fn deregister_buffer(&self, handle: u64) -> Result<()>

Deregister a previously registered buffer.

fn symmetric_alloc(&self, bytes: usize) -> Result<u64>

Allocate a GPU buffer in NCCL’s symmetric-memory window (NCCL ≥ 2.28 / ncclMemAlloc). Returns the device pointer as u64.

fn symmetric_free(&self, ptr: u64) -> Result<()>

Free a buffer previously returned by symmetric_alloc.

fn set_add_kernel(&self, handle: u64)

Provide a kernel handle for the BF16 in-place addition kernel.

fn all_gather(&self, send_ptr: u64, recv_ptr: u64, bytes: usize) -> Result<()>

All-gather: each rank contributes a chunk, all ranks get full buffer.

fn reduce_scatter( &self, send_ptr: u64, recv_ptr: u64, bytes: usize, ) -> Result<()>

Reduce-scatter: reduce + scatter (inverse of all-gather).

fn broadcast(&self, ptr: u64, bytes: usize, root: usize) -> Result<()>

Broadcast from root rank to all ranks.

fn barrier(&self) -> Result<()>

Barrier: block until all ranks reach this point.

fn send_to( &self, ptr: u64, bytes: usize, dest_rank: usize, stream: u64, ) -> Result<()>

Send tokens to a specific rank (for EP token dispatch).

fn recv_from( &self, ptr: u64, bytes: usize, src_rank: usize, stream: u64, ) -> Result<()>

Receive tokens from a specific rank (for EP token combine).

fn group_start(&self) -> Result<()>

Begin a group of point-to-point operations (send_to/recv_from).

fn group_end(&self) -> Result<()>

End a group of point-to-point operations.

fn is_healthy(&self) -> bool

Check if the communicator is healthy (no async errors, no timeouts).

fn attempt_reconnect(&self) -> Result<()>

Attempt to recover a degraded communicator.

fn rank(&self) -> usize

This rank’s index (0-based).

fn world_size(&self) -> usize

Total number of ranks.

impl Drop for NcclBackend

fn drop(&mut self)

Executes the destructor for this type.

impl Send for NcclBackend

impl Sync for NcclBackend