Module fault 

Process-fatal GPU fault latch.

Why this exists (issue #429)

A large class of CUDA errors — CUDA_ERROR_MISALIGNED_ADDRESS (716), CUDA_ERROR_ILLEGAL_ADDRESS (700), CUDA_ERROR_LAUNCH_FAILED (719) — are sticky: they do not merely fail the call that produced them, they destroy the CUDA context. Every subsequent driver call in the process returns the same status, forever. There is no in-process recovery; the context cannot be re-created while the primary context is retained.

Before this module, Atlas treated such a failure as a per-request error. The forward pass returned Err, the scheduler failed that batch, the handler emitted a 500 — and then kept serving. /health still said ready, because a model was still published, and every following request died deep in the driver (observed at cuMemsetD8Async). The process was alive, advertised itself as healthy, and could only produce errors.

How fatality is decided — probed, never guessed

Classification does not match on the error code or its text. Sticky-ness is a property of the context, so it is measured directly: after a failed operation, issue a call that must succeed on a healthy context (a no-op synchronize). If that also fails, the context is gone.

This is why classify takes a probe result rather than an error code. It buys three things a code allowlist cannot:

• it covers every sticky status, including ones not yet enumerated;
• it does not kill the server for a status that merely looks fatal — an isolated invalid argument from a bad launch config leaves the context healthy, and the probe says so;
• it is a pure function of the probe, so both verdicts are unit-testable with no GPU.

Contract

The latch is one-shot and first-writer-wins: the first fault is the diagnostic one, and everything after it is that fault echoing through the remaining call sites. Reporting the tenth cuMemsetD8Async failure instead of the launch that poisoned the context would bury the cause.

Both properties come from OnceLock rather than from code that maintains them. A flag-plus-reason pair (AtomicBool + Mutex<Option<String>>) has a window in which the flag is visible and the reason is not, and a health endpoint that lands in it reports “faulted, reason unknown” — the least useful of the three possible answers. A single OnceLock<String> makes “is faulted” and “has a reason” the same word, so the window does not exist and set supplies first-writer-wins atomically.

Structs

FaultLatch

A one-shot, first-writer-wins fault flag.

Enums

Fatality

The verdict for one failed GPU operation.

Constants

EXIT_GPU_FAULT

Exit status for a process that died because its CUDA context was lost.

Functions

classify

Decide whether a failed GPU operation destroyed the context.

exit_code

The process exit status for a run that is ending.

global

The process-wide latch. A destroyed CUDA context is a property of the process, not of any one backend handle, so this is deliberately global — a per-backend flag would report healthy from a second handle onto the same dead context.