NVIDIA and Princeton University’s Dao AI Lab published ReplaySSM on October 12, 2026, a technique that addresses memory-bandwidth bottlenecks in hybrid language models by caching recent inputs instead of storing full recurrent state at every decoding step. The authors report kernel speedups of 1.46× to 1.80× and whole-decode-step speedups of 1.20× to 1.47× across four models at batch size 256, according to the paper ReplaySSM: Cache SSM Inputs, Not State available on arXiv and the NVIDIA Developer Blog post by Michelle Horton.
Hybrid Models and the State-Traffic Problem
Hybrid models interleave state-space model (SSM) layers—such as Mamba-2, Gated DeltaNet, or Kimi Delta Attention—with a smaller number of attention layers. NVIDIA Nemotron 3, Qwen3.5 through Qwen3.8, Kimi-K3, and GLM-5.3-Flash all follow this architecture. SSM layers use fixed-size recurrent state rather than key-value caches that grow with context length, a design intended to accelerate decoding.
At serving batch sizes, however, SSM layers can become the latency bottleneck. On NVIDIA Nemotron-3-Super-NVFP4 at batch size 256 on one NVIDIA Blackwell Ultra B300 GPU, SSM layers accounted for the largest component of decode-step latency up to 100,000 tokens of context, according to the October 12 blog post. Each conventional SSM decoding step loads the full state from high-bandwidth memory, applies a data-dependent decay and rank-one update, computes the output, and writes the updated state back. The state for one Mamba-2 layer is 4 MiB per sequence in FP32, while an FP8 key-value entry is 0.5 KiB per token; an SSM layer’s state equals the size of an attention layer’s KV cache for 8,000 tokens. Nemotron-3-Super-NVFP4 has five times as many SSM layers as attention layers.
How ReplaySSM Works
ReplaySSM represents the recurrent state as a checkpoint plus a buffer of recent input records—the decay parameter, input vector, and key vector for Mamba-2. On most steps, called non-flush steps, the method appends the current inputs to the buffer and computes the output from the checkpoint and buffer without storing a full state matrix. When the buffer of L records fills, a flush step folds the buffered inputs into the checkpoint, writes the new checkpoint to memory once, and clears the buffer. Over a window of L tokens, conventional decoding reads the state L times and writes it L times; ReplaySSM reads the checkpoint on every step but writes it only once during the flush, roughly halving dominant state memory traffic, according to the paper.
The October 12 blog post states that the authors evaluated four hybrid models: Nemotron-3-Nano-4B and Nemotron-3-Super-120B-A12B (both with Mamba-2 layers), and Qwen3.5-4B and Qwen3.5-122B-A10B (both with Gated DeltaNet layers). Tests used window size L = 16 and an FP32 SSM state, comparing ReplaySSM in vLLM against vLLM’s own kernels at their best launch configuration. At batch size 256, kernel speedups ranged from 1.46× to 1.80×, and decode-step speedups from 1.20× to 1.47×. At batch size one, other sources of latency dominated and ReplaySSM provided no speedup in these tests.
Speculative Decoding and Rollback
Speculative decoding uses a drafter to propose T tokens, verifies them in one pass through the target model, and rolls back to the last accepted token. Attention models support rollback by moving a KV-cache pointer, but once an SSM has summarized draft tokens into its state, the update has no easy inverse. Serving engines such as vLLM store a state snapshot after every draft token and select the snapshot matching the accepted prefix—one read and T writes per step on a memory-bound path. The snapshots also reduce how many requests fit in GPU memory.
ReplaySSM appends the inputs of all T draft tokens to the buffer before verification and commits the accepted prefix by advancing the buffer pointer; records beyond the pointer are ignored. No state snapshot is saved for each draft token and no state is restored. A flush folds only committed records into the checkpoint. Because a non-flush step produces only the output—not an updated state—the authors use output-only evaluation: the checkpoint is read once for all T queries, and the remainder is two small matrix products over cached keys, cached values, and draft queries, without forming intermediate states. This computation can use Tensor Cores and shares work across value heads in a group.
On GSM8K prompts with T = 4 at batch size 512, ReplaySSM delivered 1.85× and 2.10× the throughput of vLLM’s speculative decoding on Nemotron-3-Super-120B-A12B-NVFP4 and Qwen3.5-122B-A10B-NVFP4, respectively, with matched acceptance lengths, according to the October 12 blog post. The gain came from faster verification—ReplaySSM verifies eight tokens in about the time of one standard step, flushes included, while vLLM’s kernel latency at T = 8 is almost five times that of its standard decoding step—and higher concurrency, because draft snapshots consume GPU memory and cap batch size.
Quantization and Error Accumulation
A quantized SSM state produces the output and is the input to the next update, so rounding error re-enters the recurrence at every step. A KV cache has no such loop: a quantized entry is read many times but never written back into itself. ReplaySSM separates the two jobs of the state—output computation and recurrent update—into two copies with different precisions. A high-precision checkpoint is maintained for the recurrence, and a low-precision copy is used for non-flush reads. A flush starts from the high-precision checkpoint, incorporates buffered inputs, and casts the result independently into both formats. The low-precision copy is never read back into the recurrence, so its rounding error affects only the outputs of the current window and does not propagate across the recurrence, according to the October 12 blog post.
The authors measured output error as Kullback–Leibler divergence from FP32-state decoding under teacher forcing on PG-19 text, out to 128,000 generated tokens. A shared NVFP4 or FP8 checkpoint accumulated error with length. On Qwen3.5-4B at 128,000 tokens, NVFP4 reads with an FP32 recurrence reduced KL from 0.136 to 0.0038 compared to a shared NVFP4 checkpoint. An FP16 recurrence with stochastic rounding tracked FP32 closely.
On the GPQA-Diamond benchmark, a shared NVFP4 checkpoint lost 3 to 31 percentage points of accuracy across the four models compared to FP32. NVFP4 reads with an FP16 or FP8 recurrence recovered most of that loss, keeping three out of four models within about 2 percentage points of FP32, according to Table 2 in the blog post. On Nemotron-3-Super-NVFP4, the shared checkpoint tracked FP32 accuracy for shorter generations but diverged as generation length grew, while the separate-precision configuration stayed close to FP32 throughout. On MMLU-Pro and Math-L5, where generations are generally shorter, three out of four models lost at most 3 percentage points with a shared NVFP4 checkpoint, while separate checkpoint configurations stayed within half a percentage point of FP32 on average.
Production Deployment and Availability
ReplaySSM is in production in GLM-5.3-Flash and Kimi-K3, according to the October 12 blog post. Z.ai’s infrastructure report describes adding ReplaySSM to the KDA decode kernel for GLM-5.3-Flash. Kimi-K3 caches the SSM inputs of draft tokens for verification in its KDA layers and cites ReplaySSM as a concurrent approach. The method is integrated into NVIDIA TensorRT-LLM, vLLM, SGLang, and FlashInfer. Code is available on GitHub at github.com/Johnny-Liou/ReplaySSM, and the paper is available on arXiv at arxiv.org/abs/2610.13778.
The performance figures reported are workload-specific author benchmarks, not universal guarantees. Batch-size-one tests showed little end-to-end improvement. The Dao AI Lab published an earlier walkthrough of standard and speculative decoding on its blog.
Source: developer.nvidia.com.
Be the first to comment