Nemotron 3: How a Mamba-Transformer Hybrid Runs 1M Tokens

TL;DR

NVIDIA’s Nemotron 3 Nano is a 30B-parameter model that only turns on about 3.5B parameters per token, and only 6 of its 52 layers use attention at all. The other 46 are Mamba-2 state-space layers and Mixture-of-Experts blocks. That single design choice is why the model advertises a 1-million-token context window and claims up to 3.3x the throughput of similarly sized open models. If you have been treating “Transformer” and “LLM” as synonyms, Nemotron 3 is a reason to separate the two. Below I walk through what Mamba-2 does, why NVIDIA kept a handful of attention layers instead of dropping them, and how to run the open weights yourself.

Why anyone is bolting Mamba onto a Transformer

Attention has one expensive property: every token looks at every other token. Double the sequence and you roughly quadruple the compute for the attention step. At a few thousand tokens nobody cares. At a million tokens, the attention math can dominate the forward pass, and the KV cache (the running memory of all those past keys and values) balloons until it no longer fits on the GPU. I’ve written before about how the field keeps attacking this from the compression side in TriAttention’s KV-cache work and the broader sparse attention literature. Those methods shave the cost of attention. Nemotron 3 takes a blunter route: use a lot less of it.

State-space models are the tool that makes that possible. Mamba, and its successor Mamba-2, process a sequence in linear time and carry a fixed-size hidden state instead of a cache that grows with every token. Feed one a million tokens and the memory footprint of the sequence model barely moves. The catch is that a fixed state is a lossy summary, so you give up some of the precise, look-anything-up recall that attention gives you for free. The engineering question of the last two years has been how much attention you can throw away before the model gets noticeably dumber, and NVIDIA’s answer with Nemotron 3 is: quite a lot.

What I actually looked at

I spent an afternoon with the Nemotron 3 Nano technical report and the Hugging Face model card rather than trusting the launch-day summaries, and one detail jumped out that the marketing glosses over: the released config ships with max_position_embeddings set to 256k, not the headline 1M. The million-token window is real, but you opt into it. The default checkpoint is tuned for a quarter of that. It’s the kind of thing you only notice if you open config.json, and it changes how you’d budget memory for a deployment. That’s the lens for this piece: what the architecture is, stripped of the press release.

Mamba-2 in plain language

Think of attention as a person reading a long document who can flip back to any page instantly to re-check a fact. Precise, but the more pages there are, the longer each flip-back takes, and they have to keep every page on the desk.

A state-space model is a person reading the same document while keeping a single running set of notes. They never flip back. When a new sentence arrives, they update the notes and move on. Reading speed stays constant no matter how long the document is, and the desk never overflows, because there is only ever one page of notes. The cost is obvious: if a detail didn’t make it into the notes, it’s gone.

Mathematically, an SSM keeps a hidden state h and updates it one step at a time:

# Toy linear state-space recurrence (the idea, not the optimized kernel)
import numpy as np

def ssm_step(h, x, A, B, C):
    # h: fixed-size hidden state; x: current token embedding
    h = A @ h + B @ x     # carry forward + absorb the new token
    y = C @ h             # read an output out of the state
    return h, y

h = np.zeros(16)          # the "notes": fixed size, regardless of sequence length
A, B, C = init_matrices() # learned; A controls what the state remembers vs forgets
for x in token_stream:    # one pass, left to right, O(n) total
    h, y = ssm_step(h, x, A, B, C)

The two things to take away: the loop is O(n) in sequence length, not O(n²), and h never grows. Mamba-2’s real contribution over the original Mamba was showing that this kind of recurrence is mathematically close to a restricted form of attention, which let the authors write it as big matrix multiplications that saturate a GPU’s tensor cores. That reformulation runs 2–8x faster than the original Mamba’s own selective scan while staying competitive with Transformers on language modeling, per the Mamba-2 paper. The state stays small, and the hardware stays busy.

Why keep 6 attention layers at all?

The sharp design choice is keeping a few attention layers at all. Out of 52 layers, NVIDIA kept exactly 6, using grouped-query attention with 2 groups; the other layers are 23 Mamba-2 blocks and 23 MoE blocks.

The reason is that pure state-space models are bad at one specific, important thing: precise recall of something they saw far back in the context. Copying a long string verbatim, retrieving an exact figure from page 400, following an instruction that referenced a variable defined thousands of tokens ago — these lean on the ability to reach back and grab the exact token, which is what attention does and what a compressed state struggles with. A handful of full-attention layers, sprinkled through the stack, restores that ability cheaply. You pay the quadratic cost on 6 layers instead of 52, and you keep most of the recall.

This “mostly Mamba, a little attention” recipe is not unique to NVIDIA. Jamba and other 2024-era hybrids explored the same idea. But Nemotron 3 pushes the attention fraction down about as far as anyone has shipped in a production-grade model. Roughly one in nine layers does global attention. The rest run in linear time.

The MoE half: 30B on paper, 3.5B in practice

The other 23 layers are where the “30B model that acts small” claim comes from. Each MoE (Mixture-of-Experts) layer holds 128 routed experts plus 1 shared expert, but a lightweight router picks only 6 of them to run for any given token. So the model stores 31.6B parameters total, yet a single forward pass only lights up about 3.5B of them (roughly 3.2B if you exclude embeddings, which is where the “A3B”, for active 3B, in the name comes from).

The practical effect: you get the knowledge capacity of a big model with the inference cost closer to a small one. It’s the same trick behind most of 2026’s efficient open models, and it stacks neatly with the Mamba layers: one axis of sparsity in the sequence dimension (Mamba’s linear scan), another in the parameter dimension (MoE’s routing).

The numbers NVIDIA reports

I want to be careful here, because these are vendor-reported figures measured on NVIDIA’s own H200 hardware, not something I benchmarked. Treat them as claims to verify against your own workload, not gospel.

The headline is throughput. On an 8K-input / 16K-output setting, NVIDIA reports Nemotron 3 Nano running about 3.3x higher throughput than Qwen3-30B-A3B and 2.2x higher than GPT-OSS-20B, two comparable open MoE models. The hybrid design is doing the work: fewer attention layers means less of the forward pass scales quadratically with output length.

On accuracy, the reported benchmark card looks like this:

That RULER score is the one that carries the architecture argument. RULER stress-tests whether a model can actually find things across its full context rather than just accepting a long prompt and quietly ignoring the middle. Holding 86.3% at the 1M mark is the evidence that the 6 attention layers are pulling their weight. A pure state-space model tends to sag badly on that test. NVIDIA also reports it beats both comparison models on RULER across context lengths, which, if it holds up, is the practical payoff of the whole design.

The training bill behind it: 25 trillion tokens of pretraining (about 3 trillion of them new relative to Nemotron 2), followed by supervised fine-tuning and large-scale reinforcement learning. That is a lot of tokens for a 30B model, and it is a reminder that architecture is only half the story; data still does most of the heavy lifting.

Running it yourself

The weights are openly downloadable under the NVIDIA Nemotron Open Model License, which is one reason I bothered digging in. There are two easy paths.

The quickest is Ollama, which pulls a quantized build:

ollama run nemotron-3-nano

For the real checkpoint with full control, the Hugging Face route works. Note trust_remote_code=True, since the Mamba-2 layers need custom modeling code that isn’t in stock Transformers yet:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="auto",
)

messages = [{"role": "user", "content": "Explain a state-space model in one sentence."}]
chat = tokenizer.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

out = model.generate(chat, max_new_tokens=512, temperature=0.6, top_p=0.95)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Two things worth knowing before you commit hardware to it. First, that 256k default context from config.json. If you need the full million tokens, you have to raise it explicitly and pay the memory for the KV cache on those 6 attention layers, which is exactly the cost the architecture was designed to minimize but not eliminate. Second, the model is a reasoning model by default: it emits a thinking trace before its final answer, so budget your max_new_tokens accordingly or you’ll truncate the answer mid-thought. If you want a lighter reasoning trace, the free hosted endpoint on OpenRouter is a cheap way to feel out the behavior before downloading 60GB of weights.

Where this fits in the bigger picture

Hybrids like this are the clearest sign that the “everything is a Transformer” era is softening. For a couple of years the interesting architecture work went into making attention cheaper without replacing it, which is the thread running through recursive language models and the diffusion-based text models I’ve covered, and the whole LLM inference efficiency cluster. Nemotron 3 sits at the other end of that spectrum: it keeps just enough attention to stay sharp and replaces the rest wholesale. NVIDIA is not alone here. The same instinct shows up in DeepSeek’s hyper-connection work and across the 2026 crop of long-context models. But Nemotron 3 is the most quotable production example of how far the swap can go.

For practitioners, the takeaway is narrow and useful. If your workload is long-context (repository-scale code, long agent traces, document QA over big corpora), a Mamba-heavy hybrid is now a serious option, and an open-weights one at that. If you’re doing short prompts, the architecture buys you less; a dense small model may be simpler. The whole point of the design is that its advantage grows with sequence length, and that is exactly where a plain Transformer hurts most.

FAQ

Is Mamba better than a Transformer?

Not universally. Mamba (a state-space model) is faster and far cheaper in memory on long sequences because it runs in linear time and carries a fixed-size state. But it’s weaker at precise recall of distant tokens, which attention handles naturally. That trade-off is exactly why Nemotron 3 uses both instead of picking a side.

Why combine Mamba and attention in one model?

Each covers the other’s weakness. Mamba layers handle the bulk of sequence processing cheaply, while a small number of attention layers restore the exact long-range lookup that a compressed state struggles with. The result keeps most of the efficiency of a state-space model and most of the recall of a Transformer.

What is Nemotron 3 Nano?

It’s NVIDIA’s open-weights hybrid Mamba-Transformer MoE model: 31.6B total parameters, about 3.5B active per token, 52 layers (23 Mamba-2, 23 MoE, 6 attention), and support for context up to 1M tokens. It’s trained as a reasoning model, emitting a thinking trace before its final answer.

How is Mamba-2 different from attention?

Attention compares every token to every other token (quadratic cost, growing KV cache) so it can look anything up precisely. Mamba-2 instead maintains a single fixed-size state it updates token by token (linear cost, constant memory), trading some precise recall for large gains in speed and memory on long inputs.

Is Nemotron 3 free to use?

The weights are released under the NVIDIA Nemotron Open Model License and are downloadable from Hugging Face, runnable locally via Ollama or Transformers, and available on hosted endpoints like OpenRouter’s free tier. Check the license terms for your specific commercial use before shipping.

Sources

• Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning — arXiv:2512.20848 — the technical report this explainer is based on
• NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 — Hugging Face model card — architecture spec, benchmark numbers, and run instructions
• NVIDIA Nemotron 3 family overview — variant lineup (Nano, Super, Ultra) and reported throughput
• Transformers are SSMs: Mamba-2 — arXiv:2405.21060 — the state-space model this architecture builds on
• Jamba: A Hybrid Transformer-Mamba Language Model — arXiv:2403.19887 — earlier work on interleaving Mamba and attention

Bottom line

Nemotron 3 is worth understanding even if you never deploy it, because it’s a clean demonstration of a bet the whole field is now making: attention is a precision tool you spend on a few layers while state-space math carries the rest. Six attention layers out of 52, and the model still holds 86% retrieval at a million tokens. The next time someone calls a model “just a Transformer,” that’s now an assumption worth checking.