Expand description
The one place that turns a safetensors data_offsets pair into a byte span.
A checkpoint is third-party data: Atlas loads it by URL, so every number in
the header is attacker-controlled until it has been checked. The header
declares each tensor as "data_offsets": [start, end] relative to the data
section, and the naive end - start is a u64 subtraction that WRAPS on a
crafted or truncated file — a reversed pair yields a length near u64::MAX,
which downstream becomes an allocation size, a pread window, or an RDMA
len published to a peer.
Two loaders parse the same header format for different transports
(spark_runtime::fast_weights reads it with O_DIRECT, and
spark_storage::weight_peer republishes it as an RDMA manifest). They live
in crates that cannot depend on each other, so the rule lives here and
both call it, rather than each keeping its own copy of the arithmetic.
Structs§
- Tensor
Span - A validated tensor byte span: an absolute file offset and a length that is known to fit inside the file it came from.
Functions§
- tensor_
span - Validate one tensor’s
data_offsetsagainst the file that declared it.