OpenAI-Compatible Server

Atlas serves via spark-server — an OpenAI and Anthropic compatible HTTP API over axum. This chapter is the operator's reference for CLI flags, protocols supported, and the knobs that matter in production. The authoritative flag list is always spark serve --help; the headings below match the groupings in the CLI so cross-referencing is easy.

CLI structure

spark serve <MODEL> [--flags...]
spark serve --model-from-path <PATH> [--flags...]
spark --version
spark --help

Every runtime configuration flag has a long-form name. Most are documented inline with #[arg] doc-strings in crates/spark-server/src/cli/serve_args.rs.

Model selection and I/O

Memory / budget

Production rule of thumb for tight single-GPU deployments of 100B+ models: drop --max-prefill-tokens to 2048 and --max-batch-size to 1. The default 8192 sizes the scratch arena, not the KV pool; tuning down frees hundreds of MB.

KV cache precision

The flag takes a number or one of three words: auto (a fixed alias for 2, not a heuristic) and max/all (every attention layer). Anything else that fails to parse warns and falls back to 0.

--kv-high-precision-layers 0 is not "no promotion" for the turbo* family. 0 means defer to the per-dtype automatic value (main_modules/kv_dtypes.rs::auto_high_precision_layers, applied at serve_phases/kv_cache.rs). For bf16 / fp8 / nvfp4 the automatic value is None, so 0 really is zero. For every turbo* and asymmetric variant it is not: turbo2 and bf16k_turbo3v promote max(4, ⌈4·L/5⌉) layers, and all the others promote max(2, ⌈L/3⌉) — roughly a third of attention layers forced to BF16, which also shrinks the KV pool. Pass an explicit non-zero value if you want to control it; there is no spelling of this flag that promotes nothing under a turbo dtype.

See FP8 and NVFP4 for the trade-offs. Atlas's recommendation per model family:

• Qwen3.5 family → nvfp4 KV.
• Qwen3.6 / Nemotron-H → fp8 with calibration.
• 122B-class (memory-constrained) → nvfp4 + --kv-high-precision-layers 2.
• Everything else → fp8 (safe default).

Speculative decoding

See the MTP deep dive. Use only one of --speculative, --self-speculative, --ngram-speculative — but note this is guidance, not an enforced constraint: none of the three carries a clap conflicts_with and cli/validate.rs has no rule for the combination, so passing several parses and serves. The scheduler then resolves them by silent precedence (ngram → self-spec → MTP) rather than rejecting the config. --dflash is enforced — it declares conflicts_with = "speculative".

--num-drafts is also not a plain constant: when it is still 1, the model's MODEL.toml default_num_drafts replaces it (serve_phases/config.rs). On qwen3.6-27b that is 3, i.e. K=4.

Scheduling / caching

Agent workloads (Claude Code, OpenCode): always enable --enable-prefix-caching and --scheduling-policy slai. The prefix cache dominates wall-clock for system prompts + tool schemas; SLAI keeps streaming smooth under concurrent load.

Multi-GPU (Expert Parallelism)

See Multi-GPU & EP=2 for the full setup, including the NCCL env vars that matter on GB10.

Reasoning / tools

Observability / experimental

Endpoints

Rate limiting and auth

• --require-auth (with --auth-token <key> or --auth-tokens-file <path>) — requires an Authorization: Bearer <key> header on write endpoints. The presented token must match one of the loaded tokens (constant-time compare); there is no "accept any key" mode.
• Token-bucket rate limiter per key (crates/spark-server/src/rate_limiter.rs). Off by default; enable by setting ATLAS_RATE_LIMIT_RPM (requests/min) and/or ATLAS_RATE_LIMIT_TPM (tokens/min) > 0 (bursts via ATLAS_RATE_LIMIT_BURST_RPM / ATLAS_RATE_LIMIT_BURST_TPM, default = the cap). A MAX_KEYS DoS guard bounds the key table.
• Body-size limit env-configurable via ATLAS_MAX_BODY_BYTES (default 32 MiB — main_modules/serve_router.rs).

Changing the model without restarting

A running server can replace its model in place. Three routes reach it, and they share one code path:

• The dashboard. spark serve with no MODEL starts the listener and opens the Library, where a model and one of its recipes can be picked. Selecting one loads it and returns to the Main view.
• The Library, on a server that is already serving. Same flow; the running model is released first.
• A client request naming a different model, Ollama-style — off unless --auto-swap is passed.

--no-auto-swap forbids request-triggered loading outright and wins over --auto-swap regardless of order. For deployments where the served model is part of the contract, pass it: no client can then change what the endpoint is running, whatever else is on the command line.

Even with --auto-swap, a swap needs a request whose model resolves to a DIFFERENT model with a known recipe. A name that is absent, unrecognised, or already live is served by the current model, exactly as before the flag existed.

What a swap preserves. Stored responses and conversations, rate-limiter buckets, the API-key policy and --dump all belong to the process, not the model, and survive unchanged — including while no model is loaded, when GET /v1/conversations/{id} still answers rather than reporting the model missing.

What it cannot change. The listening socket is bound once for the process lifetime, so a recipe's host/port are ignored with a warning and the model serves on the address already bound. Hot-swap is single-node: a recipe needing more than one rank is refused rather than half-applied.

While it runs. In-flight requests finish on the model they started on. New requests that need a model get 503 model_not_loaded — retriable, and the same shape /health already reports during startup. A load that fails restores the previous model, and one this build has no kernels for is refused before anything is released.

Chat templating

Tokenization uses the HF tokenizers crate plus minijinja for chat templates. Atlas ships its own template overrides for a handful of models in jinja-templates/<family>.j2 when the upstream template has known issues (e.g. template-forced <think> seeding). Naming convention: filename matches the HF repo.

Observability

Prometheus-style metrics are exposed on /metrics (optional — gated behind a build feature):

• spark_requests_total{status, model}
• spark_tokens_generated_total
• spark_ttft_seconds{model}
• spark_decode_throughput_tok_per_sec{model}
• spark_kv_pool_utilization_ratio
• spark_active_sequences

Structured logs go to stderr; they're the primary operational signal. Atlas logs a brief line per completed request (model, prompt tokens, generated tokens, TTFT, elapsed, tools used). Per-token DECODE spam is deliberately not logged — it's useless.

A safe production config (Qwen3.5-35B, agents)

serve Sehyo/Qwen3.5-35B-A3B-NVFP4 \
  --port 8888 \
  --max-seq-len 16384 \
  --max-batch-size 4 \
  --kv-cache-dtype nvfp4 \
  --gpu-memory-utilization 0.88 \
  --scheduling-policy slai \
  --tbt-deadline-ms 100 \
  --enable-prefix-caching \
  --speculative --mtp-quantization nvfp4 \
  --auto-compact 0.85 \
  --adaptive-sampling

Claude Code uses 16k+ context for tool use; running with --max-seq-len 4096 will make agents fail mid-session. Always size up when running an agent workload.