Struct OpCache 

pub struct OpCache { /* private fields */ }

Memoized kernel handles and scratch allocations for one backend.

Implementations

impl OpCache

pub fn new() -> Self

pub fn kernel( &self, gpu: &dyn GpuBackend, module: &'static str, func: &'static str, ) -> Result<KernelHandle>

Resolve module::func, memoized. Equivalent to gpu.kernel(..) on a miss; a map read on a hit.

pub fn scratch( &self, gpu: &dyn GpuBackend, tag: &'static str, bytes: usize, ) -> Result<DevicePtr>

A scratch allocation of at least bytes, memoized under tag.

Grow-only: a request larger than the current buffer allocates a new one and abandons the old, which is bounded because the sizes that drive it (batch × hidden) have a ceiling per model. The abandoned block is reclaimed when the context goes, which is the point of scoping the cache to the backend.

pub fn alloc_fell_back(&self) -> bool

Has device allocation already failed on this backend?

Retrying a failing cuMemAlloc per tensor wastes minutes of load time and fragments what is left, so the first failure latches the loader onto managed memory. Per BACKEND rather than per process: after a model is unloaded the pressure is gone, and the next model’s load should try device memory again instead of inheriting a UVM sentence from a model that is no longer resident.

pub fn note_alloc_fallback(&self)

Latch the managed-memory fallback for the rest of this model’s load.

pub fn first_n(&self, key: &'static str, n: u32) -> bool

true for the first n times this backend reaches key.

For the diagnostics that report the first few of something and then go quiet. Counted per backend, so a second model reports its own.

pub fn once(&self, key: &'static str) -> bool

true the first time this backend reaches key, false after.

For the log/dump gates whose call site holds a GpuBackend and nothing else. Backend-scoped, so a second model re-arms them.

pub fn first_shape(&self, name: &str, m: u32, n: u32, k: u32) -> bool

true the first time this backend dispatches (name, m, n, k). Diagnostic de-duplication for the GEMM route/shape log lines.

Trait Implementations

impl Debug for OpCache

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for OpCache

fn default() -> OpCache

Returns the “default value” for a type.

impl ModelResource<dyn GpuBackend> for OpCache

Release the scratch allocations.

The kernel handles are not freed here: they are module-scoped and die with the AtlasRegistry the backend holds, which cuda_host::release unloads once every handle to it is gone. Freeing them here would be a double-unload.

fn label(&self) -> &'static str

Human name, for the teardown report and for attributing a failure.

fn release(&mut self, gpu: &dyn GpuBackend) -> Result<()>

Release everything this owns. Must be idempotent: the host calls it, and a Drop backstop may call it again.