TL;DR

This guide covers five recent research papers that each attack LLM inference costs from a different angle: quantizing the KV cache (TurboQuant), replacing autoregressive decoding with diffusion (Mercury), fixing residual connections so models train better at scale (DeepSeek mHC), evicting unimportant keys using trigonometry (TriAttention), and fusing block-level and token-level sparse attention (AsyncTLS). TriAttention and AsyncTLS have full breakdowns on this blog; the other three are covered here, with links straight to the papers. This page connects them, explains where they overlap, and lays out which combinations actually make sense if you’re building or running inference infrastructure.

Two broader sections round it out: the sparse-attention lineage from DeepSeek’s natively trained NSA to the sparse attention now shipping in DeepSeek V3.2 and V4, and NVIDIA’s Nemotron 3, which drops most attention layers in favor of Mamba-2.

Paper numbers, model versions and tooling support last verified: October 25, 2026.

The Inference Cost Problem

Training a large model is expensive once. Serving it is expensive forever.

In 2026, the economics of LLM deployment are dominated by three resources: GPU memory for the KV cache, compute for attention, and bandwidth for moving data between GPU and CPU memory. As context windows stretch past 100K tokens and reasoning models generate long chains of thought before answering, these costs grow superlinearly. Going from a 32K to a 96K context multiplies prefill attention compute by about 9 (three times the tokens, squared). The KV cache grows linearly but starts big: a Qwen3-32B-class model (64 layers, 8 KV heads of dimension 128) stores 256 KB per token in fp16, so 96K tokens take roughly 25 GB of VRAM before a single weight is loaded.

The five papers I’m covering here each attack a different piece of this cost structure. Some shrink what’s stored. Some skip computation entirely. One rethinks how tokens are generated in the first place. None of them is a complete solution on its own, but taken together they sketch out what efficient LLM inference will look like over the next year.

Here’s the map:

ApproachPaperWhat It AttacksHeadline Number
KV cache quantizationTurboQuantMemory per cached entry6x+ smaller cache at ~3 bits (reported)
Parallel decodingMercurySequential token generation1,000+ tokens/sec (reported)
Training architectureDeepSeek mHCTraining instability at scaleStable from 3B to 27B, +7.2 points on BBH
KV cache evictionTriAttentionNumber of cached entries10.7x less KV memory at full-attention accuracy (reported)
Sparse attentionAsyncTLSCompute per attention step1.3-4.7x end-to-end throughput
Trainable sparse attentionDeepSeek NSAAttention cost, learned in training11.6x expected decoding speedup at 64K (reported)
Hybrid architectureNemotron 3 NanoMost attention layers3.3x throughput vs Qwen3-30B-A3B (reported)

Quantization: TurboQuant Shrinks Every Cache Entry by 6x

The most straightforward way to fit more context into memory: make each cached key-value pair smaller.

Google’s TurboQuant does this by converting KV vectors from Cartesian to polar coordinates (PolarQuant), which eliminates the per-block normalization metadata that limited previous quantization methods. For keys, where quantization errors get amplified during attention, it adds a 1-bit correction layer (QJL) based on the Johnson-Lindenstrauss transform. Google’s announcement reports at least a 6x smaller KV cache at about 3 bits per value, down from 16, with no measured accuracy loss across LongBench, Needle-in-a-Haystack, ZeroSCROLLS, RULER and L-Eval on Gemma and Mistral. The TurboQuant paper is more conservative about where the line sits. On Llama-3.1-8B, 3.5 bits per channel matches the full cache exactly (LongBench average 50.06 vs 50.06), while 2.5 bits gives up a little (49.44). The authors call that “marginal quality degradation”.

The design choice that makes this work is what it refuses to store. Previous KV cache quantizers spent 1-2 extra bits per number on bookkeeping (scale factors, zero points, per-block metadata), which ate a large share of the savings. TurboQuant sidesteps that: after a random rotation, each coordinate follows a predictable, tightly concentrated distribution, so a fixed precomputed quantizer works without per-block statistics. The method is data-oblivious, meaning no calibration set and no per-model tuning. The 6x figure still comes from the models Google tested, so treat it as a target to check on your own model.

It’s no longer a research-only artifact. vLLM merged a TurboQuant attention backend in April 2026 (PR #38479), and current releases expose it as KV-cache dtypes:

# TurboQuant KV cache in vLLM; other options include turboquant_4bit_nc,
# turboquant_k3v4_nc and turboquant_3bit_nc
vllm serve Qwen/Qwen3-8B --kv-cache-dtype turboquant_k8v4

llama.cpp hasn’t merged a TurboQuant cache type (several community PRs were closed), so local GGUF users still depend on forks. One caveat on the speed numbers: Google’s “up to 8x” compares 4-bit TurboQuant keys against 32-bit unquantized keys for the attention-logit computation on H100s. That’s one kernel, not end-to-end inference, so expect smaller real-world gains that still add up on long-context workloads.

Primary sources: TurboQuant paper (arXiv:2504.19874) and the Google Research write-up

Parallel Decoding: Mercury Hits 1,000 Tokens/Sec With Diffusion

Every major LLM you’ve used generates tokens one at a time. Token 50 requires token 49, which requires token 48. This creates a speed ceiling that faster hardware raises but never removes: hosted autoregressive models typically stream somewhere between tens and a few hundred tokens per second per request.

Mercury, from Inception Labs, throws that constraint away. It uses diffusion to generate the whole response in parallel, refining a noisy draft into coherent text over a small number of steps. Mercury 2 reached 1,009 tokens per second on Blackwell GPUs. In the launch comparison relayed by The Decoder, it finished a reasoning task in 1.7 seconds end to end, against 23.4 seconds for Claude Haiku 4.5 and 22.8 for GPT-5 Mini, and scored 91 on AIME.

The line has moved twice since. Mercury 2.5 claimed 1,107 tokens per second with a 260K context, and Mercury 3 is now Inception’s flagship: 1,068 tokens per second, a 1M-token context, and quality the company says matches or beats GPT-6 Luna and Claude Haiku 5.5. Those are vendor claims, but the direction is consistent: each generation got smarter without getting slower.

This matters for a different reason than the other four papers. TurboQuant, TriAttention, and AsyncTLS all optimize the existing autoregressive pipeline. Mercury replaces the pipeline entirely. If diffusion LLMs mature to frontier quality, the entire optimization stack for autoregressive inference becomes irrelevant for those models. That’s a big if, though. Even Mercury 3 positions itself against speed-optimized tiers like Luna and Haiku, not against the frontier models at the top of the coding rankings. The quality gap is closing, but it hasn’t closed.

Diffusion also gives you capabilities autoregressive models can’t replicate. Infilling (generating text in the middle of existing text) is natural. Error correction is built in because the model refines the entire sequence at each step instead of committing to tokens left-to-right. And the arithmetic intensity is higher, meaning the GPU tensor cores stay busy instead of waiting on memory reads.

Primary sources: Inception’s Mercury 2 and Mercury 3 announcements

Architecture Innovation: DeepSeek’s mHC Fixes Scaling With a 1967 Algorithm

This paper sits upstream of inference. It doesn’t make a deployed model faster. It makes models better during training, which means the model you deploy performs better per parameter.

The problem: standard residual connections (the skip connections in every transformer since 2017) are a bottleneck. ByteDance’s hyper-connections replace the single residual stream with multiple parallel streams, giving richer information flow between layers. They work great at small scale. At 27B parameters, the learned mixing matrices amplify signals by 3,000x and training explodes.

DeepSeek’s fix, detailed in their mHC paper, constrains those mixing matrices to be doubly stochastic using the Sinkhorn-Knopp algorithm from 1967. A doubly stochastic matrix has rows and columns that each sum to 1, so multiplying by it redistributes signal without amplifying it. The composite gain across all layers stays at 1.6 instead of 3,000. Training is stable at 3B, 9B and 27B parameters. At 27B, mHC beats standard residuals by 7.2 points on BBH (51.0 vs 43.8), 6.9 on DROP (53.9 vs 47.0) and 7.1 on GSM8K (53.8 vs 46.7). It also edges out the unconstrained hyper-connections it fixes, by a smaller margin: 2.1, 2.3 and 0.6 points on the same three benchmarks.

The overhead is 6.7%. A training run that takes 100 GPU-hours with standard residuals takes 107 with mHC. For the performance gains you get, that’s a trade anyone would take.

I’m including mHC in this hub even though it’s a training technique because better-trained models need fewer parameters for the same quality. Fewer parameters means less memory, faster inference, and lower serving costs. If mHC lets you get GPT-5-level performance from a model that’s 20% smaller, every inference optimization in this guide benefits from that.

Primary source: mHC: Manifold-Constrained Hyper-Connections (arXiv:2512.24880)

KV Cache Eviction: TriAttention Drops 90% of Keys Using Trigonometry

Where TurboQuant makes each KV pair smaller, TriAttention asks a different question: which pairs can you throw away entirely?

Previous eviction methods (H2O, SnapKV, PyramidKV) estimate key importance from recent attention scores. The problem is that modern LLMs use RoPE, which continuously rotates query and key vectors based on position. A recent query’s attention scores give you a rotated, biased view of key importance. The TriAttention paper shows what that costs: on AIME25 with Qwen3-8B and a 2,048-token KV budget, SnapKV scores 20.0% and R-KV 17.5%, against 40.8% for full attention, so the leading baselines keep about half the accuracy. TriAttention scores 32.9% at the same budget, and the authors report matching full-attention accuracy with 10.7x less KV memory on AIME25 with 32K-token generation.

The trick: before RoPE rotates them, query and key vectors cluster tightly around fixed centers. That clustering turns attention into a trigonometric function of positional distance. TriAttention uses this stable signal to score keys instead of relying on post-RoPE attention, which shifts with every token.

The practical impact is concrete. In the paper’s OpenClaw demo, an agent backed by Qwen3-32B (AWQ INT4) on a single 24 GB RTX 4090 runs out of memory mid-session with full attention; with TriAttention the multi-turn session finishes. The vLLM plugin installs from the GitHub repo and activates itself once installed. There’s no CLI flag: you set the budget, an absolute token count rather than a fraction, through environment variables. The repo’s recommended chat setup:

git clone https://github.com/WeianMao/triattention.git
cd triattention && pip install -e .

# Precomputed Q/K frequency stats are required; the repo ships this one
export TRIATTN_RUNTIME_SPARSE_STATS_PATH=triattention/vllm/stats/qwen3_32b_int4_stats.pt
export TRIATTN_RUNTIME_KV_BUDGET=12000   # tokens kept per request (default 2048)

vllm serve <model_path> --dtype bfloat16 --max-model-len 32768 \
    --enforce-eager --trust-remote-code \
    --enable-prefix-caching false --max-num-batched-tokens 1024

That stats file is the catch. Scoring needs per-model frequency statistics, and the repo lists only three verified models (Qwen3-8B, DeepSeek-R1-Distill-Llama-8B, DeepSeek-R1-Distill-Qwen-7B) plus the Qwen3-32B INT4 stats used for OpenClaw. Anything else needs a calibration run first, and prefix caching has to stay off. Even so, it’s the most deployable paper in this hub: open source and already wired into vLLM. The repo’s results table also covers GPT-OSS-20B, and the paper checks that the pre-RoPE clustering holds on GLM-4.7-Flash’s MLA attention.

Read the full breakdown: TriAttention Compresses KV Cache 10.7x: How Trigonometry Fixed Long-Context Reasoning

Sparse Attention: AsyncTLS Gets 4.7x Throughput by Attending to Less

Sparse attention has been around for years. The idea is obvious: at any decoding step, most past tokens don’t contribute much to the prediction, so skip them. The execution has been stuck in a tradeoff. Block-level methods (divide context into blocks, score blocks, attend to top-k blocks) are fast but miss individual important tokens inside low-scoring blocks. Token-level methods (score every token individually) are accurate but the sorting and scatter-gather overhead kills the GPU.

AsyncTLS (paper) argues you don’t have to pick. Run block-level filtering first to shrink the candidate pool by 10x, then run token-level selection only on the survivors. The expensive per-token math runs on a fraction of the original tokens, so the indexing overhead stays manageable. You get token-level accuracy with block-level speed.

The second half of the system is an asynchronous KV cache offloading engine that exploits temporal locality (if a block was important at step t, it’s probably important at step t+1) to prefetch data from CPU memory during the current step’s compute. This hides PCIe transfer latency behind work the GPU is already doing. It’s the less exciting part of the paper but arguably does more for production throughput than the attention algorithm itself.

End-to-end results on Qwen3 and GLM-4.7-Flash at 48k-96k tokens: 1.3-4.7x throughput improvement. The gap between the operator speedup (up to 10x) and the end-to-end number tells you something important: attention is no longer the only bottleneck. MLP layers, sampling, and non-attention overhead together account for roughly half the wall-clock time even after attention gets dramatically cheaper.

Read the full breakdown: AsyncTLS: 4.7x Faster Long-Context LLM Inference With Two-Level Sparse Attention

The Wider Sparse-Attention Landscape

AsyncTLS sits in a lineage worth knowing, because the three generations fail in different ways:

  • Static patterns (2020). Longformer and BigBird fix the mask at architecture time: a sliding window, a few global tokens, some random links. Simple and fine for classification, but the same window applies to a legal contract and a code file.
  • Dynamic, inference-time (2023-2024). H2O keeps recent tokens plus the “heavy hitters” that soak up most attention weight, and reported up to 29x throughput on OPT-6.7B keeping 20% of tokens. MInference classifies each head’s pattern offline and reported up to 10x prefill speedup at 1M tokens on one A100. SnapKV and Quest prune the cache per query. All of them are bolted onto a model trained with full attention, which caps accuracy at high compression.
  • Natively trainable (2025). DeepSeek’s NSA trains with sparsity from the start through three branches (compressed tokens, selected blocks, a sliding window) blended by a learned gate. On a 27B MoE model it beat full attention on LongBench (0.469 vs 0.437). Its kernels ran 9.0x faster forward and 6.0x faster backward at 64K against a Triton FlashAttention-2 baseline on A100s. The headline 11.6x decoding figure is an expected speedup computed from memory traffic: at 64K a decode step reads about 5.6K cached KV pairs instead of 65K.
  • Shipped in production (2025-2026). NSA’s descendants now run in DeepSeek’s own models. DeepSeek-V3.2-Exp introduced DeepSeek Sparse Attention (DSA) in September 2025, and DeepSeek V4 (April 2026) mixes compressed and sparse attention for 1M-token contexts. Per Hugging Face’s V4 write-up, V4-Pro at 1M tokens needs 27% of V3.2’s single-token FLOPs and 10% of its KV cache, and V4-Flash cuts that to 10% and 7%.

The Sparse Frontier, the largest study of training-free methods (Qwen 2.5, Llama 3.1 and Gemma 3 families, up to 128K tokens), adds the budget rule: at equal cost, a bigger model running sparse attention beats a smaller dense one. Its findings on limits shifted between versions, which is itself a lesson. The first version reported that the 32B and 72B models held about 17x compression during decoding while the 7B fell from 12x at 16K to 5x at 128K. The current revision, with more model families, finds that on average model size shows “no clear correlation with sparsity tolerance”. What survived every version: decoding tolerates sparsity better than prefill, longer sequences tolerate more, and some task types (high scope or high dispersion) degrade at sparsity as low as 0.5-0.67, i.e. a 2-3x budget cut. Check the tasks your pipeline depends on, not the average.

What to use today: MInference for long prompts (training-free, prefill only), H2O or Quest for decode-time cache pruning with per-task monitoring, and for 32K-plus serving, either NSA-style training or a model that already ships it, such as DeepSeek V3.2 or V4.

Hybrid Architecture: Nemotron 3 Swaps Attention for Mamba-2

Everything above makes attention cheaper. NVIDIA’s Nemotron 3 Nano mostly removes it. Of its 52 layers, only 6 use attention (grouped-query, 2 groups). The rest are 23 Mamba-2 state-space blocks, which mix the sequence in linear time, and 23 Mixture-of-Experts blocks with 128 experts and 6 active per token. So a 30B model runs about 3.5B parameters per token.

Layer typeCountJob
Mamba-223Linear-time sequence mixing; carries the long-range state
MoE23Sparse feed-forward; 128 experts, 6 fire per token
Attention (GQA)6Exact long-range recall where the state model falls short

Why keep six? A compressed state is bad at exact recall: copying a long string, pulling a figure from page 400, honoring a variable defined thousands of tokens back. A few attention layers restore that, and you pay the quadratic cost on 6 layers instead of 52. Jamba tried the same recipe in 2024. Nemotron 3 keeps attention to about one layer in nine, among the lowest shares shipped so far.

NVIDIA reports 3.3x the throughput of Qwen3-30B-A3B and 2.2x GPT-OSS-20B at 8K in / 16K out on H200s, plus 86.3% on RULER-100 at the full 1M context. These are vendor numbers. The RULER score is the one that backs up the design, since pure state-space models sag badly on that test.

Nano is the smallest of three. NVIDIA has since released Nemotron 3 Super and Ultra, the latter at 550B total parameters with 55B active (model card). Both keep the hybrid Mamba-Transformer layout and add LatentMoE experts and multi-token prediction layers. The Nano report is still the place to read the architecture in detail.

The weights are open. ollama run nemotron-3-nano pulls a quantized build; the full checkpoint (nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16) needs trust_remote_code=True for the Mamba-2 layers. The config defaults to a 256K context, so a million tokens has to be set explicitly, and the attention layers still need KV cache for it. It’s a reasoning model by default, so leave max_new_tokens room for the thinking trace.

How These Approaches Interact

The most interesting question isn’t which of these techniques is best. It’s which ones stack.

TurboQuant + TriAttention: These are directly complementary. TurboQuant compresses the values stored in each KV entry. TriAttention decides which entries to keep. Use TriAttention to keep the top 10% of keys, then apply TurboQuant to compress those remaining entries from 16-bit down to 2.5-3 bits. In theory, that’s 60-100x total KV memory reduction. The TriAttention paper doesn’t benchmark the pairing, but the project README points to a community llama.cpp port for AMD GPUs (triattention-ggml) that reports about 6.8x total KV reduction with TurboQuant composed on top. That’s far below the back-of-envelope ceiling, which tells me the two compose but don’t multiply cleanly, at least in that implementation. A controlled benchmark of the pair is still missing.

TriAttention + AsyncTLS: Both reduce the effective number of tokens the attention layer processes. TriAttention evicts keys from the cache permanently. AsyncTLS selects which cached keys to attend to at each step. They could compose, but there’s a risk of double-filtering, where AsyncTLS’s block scoring misses tokens that TriAttention kept specifically because they scored high on the trigonometric metric. Whether the two selection criteria align or conflict is an open empirical question.

TurboQuant + AsyncTLS: AsyncTLS’s offloading engine streams KV blocks between CPU and GPU memory. Compressed blocks transfer faster over the PCIe bus. This should compose well because TurboQuant is format-level compression while AsyncTLS is a scheduling and selection system. The prefetch predictions don’t depend on the precision of the cached values.

Mercury + everything else: Mercury replaces autoregressive decoding entirely. It doesn’t use a KV cache in the same way because it processes the full sequence at each diffusion step rather than generating tokens incrementally. TurboQuant, TriAttention, and AsyncTLS are all designed for the autoregressive pipeline, so they don’t apply to Mercury directly. That said, diffusion inference has its own memory challenges (storing the full sequence during refinement steps), and compression techniques adapted for diffusion will likely emerge.

mHC + everything: mHC operates at training time. It produces a better model, and every inference optimization works on whatever model you give it. mHC is orthogonal to everything in this list.

The practical takeaway: for autoregressive serving, the stack that’s most ready today is TriAttention (eviction) on top of TurboQuant (compression), with AsyncTLS-style hierarchical sparse attention for the longest contexts. For latency-sensitive applications where you can tolerate Haiku-class quality, Mercury’s diffusion approach is worth evaluating separately.

Reading Order

If you want to work through all five papers, here’s the order I’d suggest. TriAttention and AsyncTLS link to our full breakdowns; the other three link to the papers themselves.

  1. Start with TurboQuant. KV cache quantization is the most immediately useful technique, it’s conceptually clean, and it sets up the context for understanding the other KV-focused papers.

  2. Then TriAttention. After understanding how TurboQuant compresses each entry, TriAttention shows you how to select which entries to keep. Together they cover the full KV cache optimization picture.

  3. Then AsyncTLS. Once you understand the KV cache problem, sparse attention adds the computational dimension. AsyncTLS’s two-level approach builds on ideas from both the compression and eviction camps.

  4. Then Mercury. After seeing all the work needed to optimize autoregressive inference, Mercury’s “throw out the whole paradigm” approach hits differently. It’s the contrast that makes the story complete.

  5. Finish with DeepSeek mHC. This one sits upstream of the others. After reading about inference optimization, mHC shows what’s happening on the training side to produce models that need less optimization in the first place.

If you only have time for one, read TriAttention. It’s the most immediately actionable (a vLLM plugin you can run today on its verified models) and the core insight is beautiful.

What’s Coming Next

Four areas of LLM efficiency research we haven’t covered yet but are watching:

Reasoning token efficiency. Reasoning models often overthink simple problems, spending thousands of tokens on chains of thought that don’t improve accuracy. Techniques like Sketch-of-Thought, token budgets, and short-m@k have shown up to 84% token savings by controlling when and how much a model reasons. This is orthogonal to the inference-level optimizations in this hub — you can cut reasoning tokens and compress what’s left with TurboQuant or TriAttention. Agents have their own version of the same lever: context engineering prunes and summarizes what goes into the window on each step, and one 2026 paper reports higher task completion with 63% fewer tokens.

Speculative decoding at scale. Small draft models that propose tokens for large models to verify. This is already shipping in production at several providers, but the research on optimal draft model selection and multi-token verification is evolving fast. It’s orthogonal to everything in this hub and potentially composable with all of it.

MoE routing efficiency. Mixture-of-experts models activate only a fraction of their parameters per token, but routing overhead and load imbalance eat into the theoretical savings. DeepSeek’s V3 architecture pushed this forward, and follow-up work on dynamic routing and expert merging is active.

Hardware-software co-design for inference. Groq’s LPU, Cerebras’s wafer-scale chips, and Etched’s Sohu are all purpose-built for transformer inference. As inference-specific silicon matures, the software optimization picture will shift. Techniques designed around GPU memory hierarchies may need rethinking for new hardware.

FAQ

Can I combine TurboQuant and TriAttention today?

Only through community code. Both pieces now live in or around vLLM: TurboQuant as built-in --kv-cache-dtype options, TriAttention as a plugin. Nobody has documented running the two together in one vLLM server, though. The closest thing to a combined result is the community triattention-ggml port for llama.cpp on AMD GPUs, which reports about 6.8x total KV reduction with both enabled. The two operate at different levels (per-entry compression vs. entry selection), so they shouldn’t fight each other, but measure on your own workload before counting on the theoretical stacking.

Which of these techniques helps most for local LLM users?

TriAttention, if your model is on its verified list or you’re willing to run the calibration step. It installs from GitHub as a vLLM plugin and targets the most common local problem: OOM crashes during long conversations on 24GB GPUs. TurboQuant is usable today through vLLM’s TurboQuant cache dtypes; llama.cpp users still need a community fork, since mainline hasn’t merged a TurboQuant cache type. Mercury is API-only: Inception serves Mercury 3 through its API and enterprise deals and hasn’t released weights.

Do these techniques work for fine-tuned models?

TurboQuant doesn’t need model-specific adaptation; it’s data-oblivious by design. TriAttention relies on RoPE, which nearly all open-weight models use, but it also needs per-model Q/K frequency statistics. For a fine-tune, regenerate those with the repo’s calibration guide rather than reusing the base model’s file. AsyncTLS has been tested on both GQA and MLA architectures, covering the two main attention variants. mHC requires retraining from scratch with the modified residual connections, so it’s not something you’d apply to an existing fine-tuned model.

Newer additions to this cluster: efficient LLM reasoning on cutting chain-of-thought cost, and Nemotron 3’s Mamba hybrid, now covered in this guide, on the architecture side.

Sources