Module traits 

Model trait (SDD: single trait, multiple implementations possible).

The Model trait defines the interface for running inference. Business logic (scheduler, engine) programs against this trait, not concrete types.

Structs

BeamReq

One beam-search request for a translation model (NLLB). Carries the resolved per-request parameters the scheduler stamps onto the sequence; the model runs the whole beam search to completion and returns the winning hypothesis.

ChunkedPrefillPageMetadata

Per-sequence paged attention metadata for chunked prefill.

EpCommandFailed

A worker command that was received and then FAILED TO EXECUTE.

MixedBatchResult

Result of a fully-batched mixed forward pass: M decode tokens + N prefill chunks in one pass.

MixedForwardResult

Result of a mixed forward pass (decode + prefill in one pass).

PrefillSlice

Per-stream input slice for batched prefill.

PromptTokenLogprob

One scored prompt position: log P(tokens[i+1] | tokens[..=i]) plus the top-k alternative tokens under the same distribution.

SequenceState

Sequence state tracked across decode steps.

Traits

Model

Functions

bf16_to_f32

BF16 → FP32 (upper 16 bits of the IEEE-754 f32 pattern).

extract_bf16

Extract one position’s PromptTokenLogprob from a BF16 [vocab] logits slice (the layout lm_head_batched writes per row).

logprob_of

Log-softmax over f32_logits; returns the target token’s logprob and the top-k alternatives sorted descending by logprob. k=0 returns an empty top vector (chosen-token logprob only). A target outside the vocab yields -inf (fail-visible, never panics).

padded_batch_n

The multi-sequence batch padding ladder — the SSOT for padded_n.