Benchmarking

Atlas's performance claims are measurable. This chapter shows what the benchmarks measure, how to reproduce them, and how to read the numbers.

The headline numbers

From the repo README, distilled:

And the kernel micro-benchmark summary: Atlas wins 32/32 against PyTorch on attention, GEMM, SSM, RMSNorm, RoPE, SiLU×Mul, and conv1d, with speedups from 1.04× up to 18.2×.

Two kinds of benchmark

Atlas has two benchmark surfaces:

1. End-to-end HTTP throughput — atlas-spark-bench (client-side Criterion harness targeting a running server). This is what "131 tok/s" means.
2. Per-kernel micro-benchmarks — Criterion benches in each primitive crate, run with cargo bench. This is where "4.95× prefill attention" comes from.

Different things; both are meaningful. The E2E number is what an operator sees. The per-kernel number is what tells the kernel engineer where effort is paying back.

Running end-to-end benchmarks

Start a server:

sudo docker run -d --name atlas-35b \
  --network host --gpus all --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  avarok/atlas-gb10:latest \
  serve Sehyo/Qwen3.5-35B-A3B-NVFP4 \
    --max-seq-len 8192 --kv-cache-dtype nvfp4 \
    --scheduling-policy slai \
    --speculative --mtp-quantization nvfp4

Wait for listening. Then:

export ATLAS_BENCH_URL=http://localhost:8888
cd /path/to/atlas
cargo bench -p atlas-spark-bench

Criterion saves results to target/criterion/. The stable JSON snapshots that the README quotes are pinned under bench/.

The scripts/sweep_all_models.sh helper boots each model in turn, runs the canonical short-prompt bench, and writes the README.md throughput table. That's how the table in the README gets regenerated.

Running per-kernel benchmarks

cargo bench -p spark-runtime        # KV cache ops, sampler micro
cargo bench -p atlas-spark-bench    # end-to-end client benchmarks

Criterion-driven, from each crate's benches/*.rs. Reference shapes come from Qwen3-Next-80B (hidden=2048, 16 Q-heads, 2 KV-heads, head_dim=256, intermediate=512, num_experts=256, topk=10).

The full kernel numbers table:

32/32 wins. Peak achieved memory bandwidth in this set: 599 GB/s (2.2× the 273 GB/s LPDDR5X spec — that's the L2 cache effect for small SiLU×Mul inputs).

Concurrency sweep

scripts/run_conc_benchmark.sh drives N parallel streams against one server. Reveals the scheduler + KV allocator under load. Typical pattern on Qwen3.5-35B:

Aggregate throughput grows super-linearly up to the batch size cap (where graph amortisation kicks in) and then super-linearly until KV pool pressure; after that latency degrades more than throughput improves. Sweet spot on 35B: concurrency 4–8.

TTFT and prefix-cache behaviour

Run a request, note the TTFT. Run the same request again — with --enable-prefix-caching, TTFT drops to ~40ms (prefix cache hit). Agent workloads observe this as the difference between "the first response was slow" and "everything after is fast". The bench harness's prefix-warmup suite measures cold vs warm TTFT.

Where raw results live

• Pinned snapshots (tracked): result files under bench/. These feed the book and the README.
• Ephemeral Criterion runs (gitignored): target/criterion/.
• Historical benchmark journeys: docs/ATLAS_SPARK_JOURNEY.md — the benchmark retrospective across the Spark line.

Apples-to-apples notes

When comparing Atlas to vLLM or TensorRT-LLM:

• Same hardware. GB10 SM121 numbers do not transfer to H100 / B200 / MI300X.
• Same model. "Qwen3.5-35B-A3B at 36 tok/s" is vLLM's NVIDIA GB10 benchmark on the NVFP4 CUTLASS MoE path, same HF checkpoint.
• Same prompt shape. The 131 tok/s number is on a short prompt ("What is the capital of France?", max_tokens ≤ 30). Longer prompts show slightly different numbers because prefill cost amortizes differently.
• Same precision. Atlas NVFP4 vs vLLM NVFP4; Atlas FP8 vs vLLM FP8. Never compare across quant schemes.

The headline "3.6× faster than NVIDIA's 36 tok/s" is apples-to-apples against NVIDIA's own vLLM numbers on the same (GB10, Qwen3.5-35B-A3B, NVFP4) target.

Files to read

From the CLI

The benchmark suite the dashboard runs is also a subcommand, so a benchmark can be scripted, run in CI, or driven over SSH with no terminal attached.

spark benchmark list                      # the suite
spark benchmark list concurrency-sweep    # one benchmark's parameters
spark benchmark run  concurrency-sweep --model <served-model>
spark benchmark history

run drives an endpoint that is already serving — it neither loads a model nor touches the GPU. The one exception is --pull-request-gate, which does start a server: it serves the benchmark's own recipe on a free port (900 s boot timeout, for a cold NVFP4 load) and tears it down on drop (cli/bench_selfstart.rs).

A benchmark can be defined on more than one model variant — one BENCH.toml entry per checkpoint, each carrying its own serve recipe and its own thresholds (spark benchmark list <id> prints them, and the TUI shows them as a step after selecting the benchmark). A gate run serves the variant marked default = true unless you name another:

spark benchmark run agentic-webserver --yes --pull-request-gate   --checkpoint unsloth/Qwen3.8-27B-NVFP4

Records are keyed by variant too — a non-default variant's record gets the checkpoint slug in its filename, and only the default variant's records can discharge the required gate. Numbers never compare across variants: the dense 27B's wall band is roughly 2× the 35B MoE's, which is exactly why the thresholds live per checkpoint.

Point run somewhere else with --url:

spark benchmark run concurrency-sweep \
  --url http://10.10.10.3:8888 --model Qwen/Qwen3.6-35B-A3B-FP8 \
  --param concurrencies=1,2,4 --param isls=128 --param osl=64

--param takes any key from spark benchmark list <id>; anything you leave out takes the schema default. An unknown key is an error listing the valid ones, because a silently-ignored override produces a run measuring something other than what you asked for.

Exit codes

1 and 2 are distinct so a script can tell "the harness broke" from "the model missed the bar". --no-fail-on-verdict collapses 2 into 0 when you are collecting numbers rather than gating on them.

Run history

Every run — from the CLI or the dashboard — is recorded under ~/.atlas/runs/<benchmark-id>/, carrying the result, every parameter used (not just the ones you overrode), the target, the source, and the Atlas version. So a stored run says what it measured and can be reproduced.

spark benchmark history --id concurrency-sweep --limit 5
spark benchmark history --run run-1785000000123456789 --format json | jq .params

One store, both directions: a CLI run appears in the dashboard's Benchmarks → History pane, and a dashboard run appears in spark benchmark history marked tui.

Machine-readable output goes to stdout, progress to stderr, so --format json > run.json is a clean file. ATLAS_HOME relocates the store.

• crates/atlas-spark-bench/src/lib.rs — E2E harness.
• Each primitive crate's benches/*.rs — per-kernel micro.
• bench/*.json — pinned result snapshots.
• scripts/sweep_all_models.sh, scripts/run_conc_benchmark.sh — automation.
• docs/ATLAS_SPARK_JOURNEY.md — benchmark journey and retrospective.
• README "Benchmark Results" section — the authoritative long-form table.