4.7x Faster Long-Context LLM Inference: AsyncTLS Explained
Long-context LLM inference collapses past 96k tokens. AsyncTLS pairs two-level sparse attention with async KV offloading for 1.3-4.7x more throughput.
Long-context LLM inference collapses past 96k tokens. AsyncTLS pairs two-level sparse attention with async KV offloading for 1.3-4.7x more throughput.