Mamba State-Space Model

Mamba is a sequence-modeling architecture based on selective state-space dynamics rather than attention . It runs in linear time and constant memory per token at inference, processes arbitrarily long contexts without a growing KV cache , and is the most credible attention-alternative to emerge in years. As of 2026, it has not displaced the transformer at frontier scale — but it’s the architecture every long-context-curious team is watching.

The core idea

A state-space model maintains a hidden state that evolves over time according to a linear recurrence:

This is structurally a state-space model from control theory, and structurally also an RNN — except the matrices are designed so the recurrence can be reformulated as a parallel scan, making training as fast as a transformer over the same sequence.

Earlier SSMs (S4, S5) kept these matrices fixed across time. That made them efficient but expressively limited — the model couldn’t, for example, dynamically choose to “pay attention to” or “ignore” a particular token, because the dynamics were the same regardless of input.

Mamba’s contribution (2023, Gu and Dao) is to make , and a discretization step size depend on the input . This selective mechanism gives the model the ability to gate which inputs propagate into the state — a much-cheaper analog of what attention’s softmax does. The architecture name “S6” (selective S4) and the algorithm name “selective scan” both refer to this.

Why linear time matters

Attention is in sequence length. Mamba is . For short contexts (under 4K), the constants favor attention — modern GPUs are highly optimized for the dense matrix multiplies attention requires, and FlashAttention narrows the gap further. But beyond ~16K tokens, the asymptotic gap dominates, and beyond ~64K tokens Mamba is dramatically faster per FLOP.

The bigger asymptotic difference is at inference. A transformer maintains a KV cache that grows linearly with context length and dominates GPU memory at long contexts. Mamba’s hidden state is fixed size — a few hundred floats per layer — regardless of how much context has been processed. At 1M tokens of context, the difference is multiple gigabytes per sequence vs. multiple megabytes.

Where it underperforms transformers

The honest assessment as of 2026: Mamba matches or beats transformers on language modeling perplexity at a given parameter count, but lags on tasks that require associative recall — fetching a specific earlier token by content. The toy benchmark is the multi-query associative recall (MQAR) task: store many key-value pairs in the prefix, then query them. Attention solves this trivially because it can directly look up any past token. Mamba’s fixed-size state can only store so many key-value pairs before losing precision.

In practical terms: long-document question-answering, retrieval over a long context, and exact citation tasks all favor transformers. Pure Mamba has not reached frontier-LLM quality on these.

The hybrid play

The architectures that have shipped at meaningful scale are hybrids. Jamba (AI21, 2024) interleaves Mamba blocks with sparse attention layers and mixture-of-experts . Zamba and similar models do the same. The hybrid keeps Mamba’s cheap long-range mixing while restoring transformers’ precise lookup, with a few attention layers’ worth of KV cache instead of every layer.

Why it might still matter long-term

The asymptotic argument doesn’t go away. As context windows push past 1M tokens and per-token decode latency becomes the binding constraint, the linear-time / constant-memory profile gets more attractive. The trajectory of selective SSMs since 2023 has been steady improvement; the architecture is not a dead end. If a hybrid Mamba-transformer at frontier scale closes the recall gap while keeping the long-context efficiency, the production calculus changes.

For now, Mamba is the best-credentialed alternative architecture and the field’s hedge against “what if attention isn’t actually optimal.” Worth understanding even if you’re not deploying it.