TL;DR

Recursive Language Models (RLMs) load the whole long prompt into a Python REPL as a variable the model can grep, slice, and hand to sub-model calls, instead of stuffing it into the context window. In the MIT paper by Zhang, Kraska, and Khattab (accepted to NeurIPS 2026, per the authors), RLM(GPT-5) scores 91.3 on a BrowseComp-Plus setup with 6M to 11M-token inputs, where plain GPT-5 can’t even fit the prompt. A post-trained 8B model, RLM-Qwen3-8B, beats its own base model by a median 28.3% and approaches vanilla GPT-5 on three of the four tasks. The cost is real: the paper’s sub-calls were blocking, there’s no prefix caching, and outlier runs inflate the average bill. For retrieval and aggregation over huge inputs, the results are hard to ignore.

The long-context problem hasn’t gone away

Model makers keep raising context limits. Gemini 1.5 hit 2M tokens in 2024. As of October 2026, OpenAI’s GPT-6 Astra lists a 1.05M-token window, current Claude models (Opus 4.6 and later, Sonnet 4.6 and later) default to 1M, and open-weight models like Kimi K3 ship a million-token context too. Benchmarks keep finding that quality falls off well before the advertised limit. Needle-in-haystack tests are generous because the needle is usually a single factual string. Aggregation tasks are harsher. The OOLONG benchmark makes a model classify thousands of individual entries (TREC-style questions, lines from Dungeons & Dragons transcripts) and then answer distributional questions about them, and GPT-5, Claude-Sonnet-4, and Gemini-2.5-Pro all score under 50% at 128K.

The usual workaround is a retrieval pipeline. You index chunks, embed a query, pull the top-k, stuff them in a prompt. It helps but forces you to decide up front what the model will see. The model never gets to ask “wait, let me read section 4 of that second document again” — you picked for it.

RLMs flip the control. Instead of choosing chunks in advance, you give the model a Python REPL, stash the whole prompt as a variable, and let it decide what to look at.

The idea in plain terms

Here’s the loop. The root model gets a system prompt saying, roughly: you have a variable context holding the full input, a Python REPL, and a function llm_query() that sends any string you build to a sub-model with a fresh context window. That covers the whole interface.

Inside that REPL, the model runs as a programmer. It can:

  • Run len(context) or print(context[:2000]) to skim.
  • Use regex to find candidate passages.
  • Chunk the text into sections and call llm_query() on each chunk in a loop.
  • Keep intermediate answers in REPL variables and combine them with ordinary Python.
  • Finish with FINAL(answer), or with FINAL_VAR(name) to return a variable it built up in the REPL.

Instead of reading the context in its own window, the model writes code that reads the context and pulls only relevant snippets back into its prompt. When a section needs close reading, a sub-call gets a window that holds only that section plus the sub-question. The v3 paper and the current library add rlm_query(), which spawns a full RLM with its own REPL instead of a plain model call, as long as the depth budget allows it.

Mental picture: the root call behaves like a research assistant with a library card, a keyword index, and a habit of writing Python. It can page through a giant book by calling a junior assistant for each chapter and compile the answers. The junior never sees the whole book — only the chapter plus instructions.

What the paper measured

The paper runs four long-context tasks: CodeQA (code-repository understanding, from LongBench-v2), BrowseComp-Plus with 1,000 documents per query (deep research), OOLONG’s trec_coarse split (aggregation), and OOLONG-Pairs, a synthetic pairwise-reasoning variant the authors built. The main experiments use GPT-5 as the root model with GPT-5-mini handling sub-calls, plus the open Qwen3-Coder-480B-A35B.

Task (input size)Vanilla GPT-5RLM(GPT-5), depth 1
CodeQA (23K–4.2M tokens)24.0* ($0.13)62.0 ($0.11)
BrowseComp-Plus, 1K docs (6M–11M tokens)0.0*91.3 ($0.99)
OOLONG trec_coarse (131K tokens)44.0 ($0.14)56.0 ($0.43)
OOLONG-Pairs (32K tokens)0.1 ($0.16)58.0 ($0.33)

Scores from Table 1 of the v3 paper; brackets show average API cost per query. An asterisk marks runs that (sometimes) hit the model’s input context limit.

Two rows stand out. OOLONG-Pairs is only 32K tokens, well inside GPT-5’s 400K window, yet the vanilla model scores 0.1 and the RLM scores 58.0. Part of the gain comes from splitting an information-dense task into pieces, which helps even when everything fits. BrowseComp-Plus is lopsided by design: the input runs more than an order of magnitude past the window, so vanilla GPT-5 never really gets to try. Against other scaffolds the margin is smaller but still there. On GPT-5, the abstract reports median gains of 26% over context compaction, 130% over CodeAct with sub-calls, and 13% over Claude Code.

The numbers that first got RLMs noticed came earlier, from Alex Zhang’s October 2025 blog post. There, RLM(GPT-5-mini) beat plain GPT-5 by over 34 points (about 114%) on OOLONG trec_coarse at 132K tokens, and by over 15 points (about 49%) at 263K, at a similar or lower per-query cost. Those runs used a cheaper root model and an earlier setup, so treat them as the preliminary result and Table 1 as the reference.

The 8B result is the other headline. The authors fine-tuned Qwen3-8B on 1,000 filtered RLM trajectories that Qwen3-Coder-480B-A35B produced on LongBenchPro tasks, which are unrelated to the evaluation set, using 48 H100 hours. The resulting RLM-Qwen3-8B beats base Qwen3-8B run as an RLM by a median of 28.3% across the four tasks and, in the abstract’s words, “even approaches the quality of vanilla GPT-5 on three long-context tasks.” Part of that says more about vanilla GPT-5 than about the small model: GPT-5 reading the full prompt scores 0.0 and 0.1 on two of the four tasks.

Cost is the counterintuitive piece. Because code filters out irrelevant chunks before any sub-model reads them, much of the prompt never gets tokenized into a language model at all. The authors report “comparable cost” with a skew: the median RLM run is cheaper than the median base-model run, but the average is higher because of outlier trajectories where the RLM struggles to find an answer. The table shows both sides, with RLM(GPT-5) cheaper than vanilla on CodeQA ($0.11 vs $0.13) and about three times pricier on OOLONG ($0.43 vs $0.14).

Root call and recursive calls

Depth is a parameter. Depth 0 is a REPL with no sub-calls. Depth 1, the default for the paper’s main results, means sub-calls are plain LM calls. Depth 2 and up lets a sub-call be a full RLM with its own REPL. The v3 paper adds an ablation: for GPT-5, deeper recursion helped on the information-dense OOLONG-Pairs task (58.0 at depth 1, 65.5 at depth 2, 76.0 at depth 3), while higher depths made Qwen3-Coder worse on average, because its syntax errors propagated into the sub-RLMs. Deeper recursion pays off only when the model writes reliable code.

What using it looks like

The authors ship a drop-in library. The README pitches it as replacing llm.completion(prompt, model) with rlm.completion(prompt, model), and the library handles the REPL, sub-call routing, sandboxing, and depth, iteration, and budget limits. It’s on PyPI as rlms (Python 3.11+). As of October 2026 it supports OpenAI, Anthropic, Gemini, OpenRouter, Portkey, and vLLM backends, and REPL environments from a local exec (the default) up to Docker, Modal, Prime Intellect, Daytona, and E2B sandboxes.

Your application code barely changes. The difference is that the model, when handed a giant prompt, stops trying to read it and starts scripting against it. If you already call the Anthropic API, pass backend="anthropic"; no custom wrapper needed.

A minimal call, following the README’s quick-start:

pip install rlms

from rlm import RLM

long_transcript = open("earnings_call_transcript.txt").read()
question = "Which product line had the biggest YoY revenue swing?"

rlm = RLM(
    backend="openai",
    backend_kwargs={"model_name": "gpt-5-mini"},
    max_depth=1,       # sub-calls are plain LM calls, as in the paper's main runs
    max_timeout=600,   # seconds; stop runaway trajectories
)
# The transcript goes into the REPL as `context`; root_prompt is the
# short text the root model sees directly.
result = rlm.completion(long_transcript, root_prompt=question)
print(result.response)

That snippet isn’t executed for this article, since it needs a paid API key, so there’s no output to show. Under the hood, the root model might write something like hits = [i for i in range(0, len(context), 20000) if "revenue" in context[i:i+20000].lower()] and then call llm_query() on each matching 20K-character window.

The idea has also landed in DSPy, the framework Khattab created. dspy.RLM is an experimental module that implements the paper’s approach behind a DSPy signature:

import dspy

dspy.configure(lm=dspy.LM("openai/gpt-5"))
rlm = dspy.RLM("document, question -> answer", max_iters=10)

document = open("large_report.txt").read()  # 500K+ characters
result = rlm(document=document, question="What were the key findings from Q3?")
print(result.answer)

The real costs

The design has sharp edges. Some are called out in the paper or the blog post; some fall out of how a REPL plus recursion works in practice.

  • Sub-calls were blocking in the paper. The authors write that they “implemented all sub-LM queries naively as blocking / sequential calls,” which made their RLM runs slow next to the base model. The library has since added batched sub-calls (llm_query_batched, rlm_query_batched) that run in parallel up to max_concurrent_subcalls (default 4), so this is now partly an implementation choice.
  • No prefix caching across recursive calls. Zhang’s blog post notes that each recursive call “does not take advantage of any kind of prefix caching.” Each sub-call is a cold start, so the cache hit rates you rely on with a single long prompt evaporate.
  • Runtime is unpredictable. The blog reports queries ranging “from a few seconds to several minutes,” and the paper notes a wide range of runtimes. You cannot give an SLA up front.
  • Cost has a long tail. The median run is cheaper than a vanilla call and the average isn’t, because the root model decides how many sub-calls to make. The paper warns that the missing guardrails “may lead to unintentional side-effects like exploding sub-call costs.” The library now exposes max_budget (USD, on cost-tracking backends such as OpenRouter), max_timeout, and max_tokens; set them.
  • Debugging is worse. When a final answer is wrong, you have to inspect a REPL trace plus whatever sub-call outputs fed back up. It is closer to debugging a program than a prompt.

Most of these are engineering problems, and the library has started on them: parallel batched sub-calls and budget and timeout caps exist now. Prefix caching across recursive shards is still open, and the paper itself lists asynchronous sub-calls and sandboxed REPLs as future work.

What this means for practitioners

If you work on retrieval pipelines, agent memory, or long-document QA, RLMs are worth a weekend. A few takes:

  • RLMs are RAG with the roles flipped. RAG picks chunks before the model sees them. RLMs let the model pick its own chunks at inference time. The same idea (agentic memory is adjacent) keeps showing up across recent long-context research. Either way, garbage tokenization upstream still ruins everything downstream — see MarkItDown vs Docling vs Marker for the converter side of that pipeline.
  • They complement compression work, not replace it. Methods like TriAttention’s KV cache compression or TurboQuant shrink the window you do send, and sparse-attention systems like AsyncTLS cut the cost of attending across it. RLMs sidestep the window entirely by only sending the parts the question actually needs.
  • Orchestration can stand in for window size. The blog post’s headline was GPT-5-mini in an RLM wrapper beating GPT-5 at 132K tokens; the paper’s is RLM(GPT-5) going from 0.0 to 91.3 on inputs of 6M to 11M tokens. If the orchestration is cheap enough, a smaller model called more often can beat paying for a bigger context.
  • Short prompts can benefit too, when the task is dense. OOLONG-Pairs is only 32K tokens and the RLM still went from 0.1 to 58.0, because the task needs comparisons across the whole input. If your prompt fits comfortably and the model already handles it, a REPL layer only adds latency and variance.

The harder question is whether post-training a model specifically for RLM use becomes a standard step. The RLM-Qwen3-8B result suggests a native RLM model handles the REPL protocol better than a general model asked to be clever about it, and the repo now ships a training environment built on Prime Intellect’s prime-rl. Prime Intellect itself called RLMs “the paradigm of 2026” and implemented a variant as RLMEnv in its verifiers library. Theory is catching up too: a July 2026 paper by Yang, Li, McAllester, and Srebro, “Recursive Language Models Generalize Out of Domain”, argues that solving each subtask in an isolated context rules out shortcuts that chain-of-thought can learn from surrounding tokens, which is why recursion helps out of distribution even when it adds little in-distribution. Expect more of that.

For the same problem inside coding agents, context engineering covers why smaller contexts often win and how explorer sub-agents that return file paths instead of file contents keep the main window clean.

Sources

FAQ

What is a recursive language model in one sentence?

An inference setup where the LLM treats a long prompt as a Python variable and writes code to grep, slice, and call sub-models over relevant pieces, instead of reading the whole thing in one window.

Who wrote the RLM paper?

Alex L. Zhang, Tim Kraska, and Omar Khattab, all at MIT CSAIL. The arXiv ID is 2512.24601, with initial submission December 31, 2025 and revisions on January 28, 2026 (v2) and May 11, 2026 (v3). The authors’ blog and the repo README say it was accepted to NeurIPS 2026.

How is RLM different from RAG?

RAG picks which chunks the model sees before inference. RLM lets the model pick chunks during inference by running code against the raw prompt. RAG is still faster and cheaper for common cases; RLM wins when the right chunk is context-dependent.

Does RLM need a special model?

No. The paper runs RLMs on off-the-shelf GPT-5 (with GPT-5-mini handling sub-calls) and Qwen3-Coder-480B-A35B. The authors also post-train RLM-Qwen3-8B, which beats base Qwen3-8B used as an RLM by a median 28.3% across the four tasks.

What are the main limitations?

In the paper’s runs, sub-calls were blocking and sequential, nothing is prefix-cached between calls, and runtime ranges from seconds to minutes. Cost control is weaker than a single API call because the root model decides how many sub-calls to make; the library’s budget, timeout, and token caps help.

Where can I try it?

pip install rlms installs the reference implementation from github.com/alexzhang13/rlm. It supports several sandbox backends and drops in as a rlm.completion(...) replacement for a standard completion call. DSPy users can try the experimental dspy.RLM module instead.

Bottom line

RLMs are a rare case of an inference-time trick that does not need bigger models, longer windows, or new attention math. The move is conceptual: stop pretending the model reads million-token prompts, and let it script against them instead. The paper’s numbers say a model with a REPL handles inputs its window can’t hold, and beats the same model reading everything at once even on some prompts that fit. Whether your stack can afford minute-long trajectories with a long cost tail is a different question, but the research direction is coming whether you opt in or not.