Module gpu 

GPU backend abstraction (SBIO IORouter for GPU operations).

All CUDA interactions flow through GpuBackend. Business logic (model forward pass, KV cache management) never calls cuLaunchKernel or cuMemAlloc directly.

Structs

DevicePtr

Opaque device pointer wrapping a CUDA CUdeviceptr (u64).

GraphHandle

Handle to an instantiated CUDA graph (CUgraphExec).

KernelHandle

Handle to a loaded CUDA kernel function.

Enums

KernelArg

Typed kernel argument, used by launch_typed.

Traits

GpuBackend

GPU backend trait — SBIO IORouter for all CUDA operations.

Functions

baseline_free_bytes

The free-memory baseline captured at context init, or None if never set.

pack_kernel_args

Pack typed args into u64 slots, returning the slots and each argument’s STARTING slot index.

set_baseline_free_bytes

Record the free-memory baseline at GPU-context init. Call once, early, before weight loading. Idempotent-last-write; intended to be set exactly once.