Nemotron 3: How a Mamba-Transformer Hybrid Runs 1M Tokens
Nemotron 3 swaps most attention layers for Mamba-2 and MoE to reach 1M-token context at 3.3x throughput. Here is how the hybrid architecture actually works.
Nemotron 3 swaps most attention layers for Mamba-2 and MoE to reach 1M-token context at 3.3x throughput. Here is how the hybrid architecture actually works.