Module safetensors

Module safetensors 

Source
Expand description

The one place that turns a safetensors data_offsets pair into a byte span.

A checkpoint is third-party data: Atlas loads it by URL, so every number in the header is attacker-controlled until it has been checked. The header declares each tensor as "data_offsets": [start, end] relative to the data section, and the naive end - start is a u64 subtraction that WRAPS on a crafted or truncated file — a reversed pair yields a length near u64::MAX, which downstream becomes an allocation size, a pread window, or an RDMA len published to a peer.

Two loaders parse the same header format for different transports (spark_runtime::fast_weights reads it with O_DIRECT, and spark_storage::weight_peer republishes it as an RDMA manifest). They live in crates that cannot depend on each other, so the rule lives here and both call it, rather than each keeping its own copy of the arithmetic.

Structs§

TensorSpan
A validated tensor byte span: an absolute file offset and a length that is known to fit inside the file it came from.

Functions§

tensor_span
Validate one tensor’s data_offsets against the file that declared it.