Introduction

The impact of Large Language Models (LLMs) across diverse fields cannot be overstated [1, 2]. LLMs have evolved from simple next-token prediction systems into sophisticated assistants for business [3], coding [4], and research [5]. Recently, the agentic paradigm has enabled LLMs to interact continuously with their environment in a loop, allowing models to react to environmental changes and distribute tasks and responsibilities across multiple agents [6]. Despite these advances, LLMs still exhibit several fundamental limitations: they generate false or misleading information [7], reproduce existing human biases [8], and are constrained by limited context windows [9, 10]. The transformer architecture restricts models to learning “in context” through a fixed-size window, without mechanisms for updating internal parameters after deployment. This limitation is compounded by two additional factors: first, performance degrades as more tokens are added to the context, a phenomenon known as “context rot” [11], second, inference costs scale linearly with context [12]. Together, these factors make the processing of large amounts of text both qualitatively and economically prohibitive.

While context window sizes have increased substantially in recent years [10], with some models now supporting millions of tokens [13], current agentic applications often require LLMs to process and interact with billions of tokens over extended periods. Two prominent approaches have emerged to cope with context window constraints: Retrieval Augmented Generation (RAG) [14, 15] and context compaction [16, 17]. RAG enables models to retrieve relevant chunks of information from external data sources at inference time, selectively enriching the prompt with pertinent contextual information rather than requiring all information to fit within the context window. Context compaction techniques, conversely, address the problem by condensing information through summarization or selective filtering, offering efficiency gains at the cost of information loss. While current agentic frameworks and RAG have enabled the development of complex and capable agentic systems, these solutions are not scalable in the long term.

Examining current agentic architectures through the lens of the “bitter lesson” [18, 19] reveals that these approaches may not optimally leverage models and their probable future evolution. Agent frameworks require human intervention to embed models within human-designed loops or graphs, with architectures often schematically organized from a human perspective, dividing agent roles as one would assign tasks to employees [20]. RAG and context compaction, meanwhile, fundamentally assume that portions of the context are less important than others and can be either omitted or summarized away.

Recursive Language Models (RLMs) [21] offer a compelling alternative to these paradigms. RLMs treat the user prompt as part of an external environment that they can control programmatically. Their defining characteristic is the ability to manipulate input through symbolic handles without copying text into their context window, enabling models to recurse on portions of the prompt. In practice, this is implemented by providing the model with a persistent REPL environment where the initial prompt is loaded as a variable, and by instructing or training the model to generate code that facilitates understanding and transformation of chunks of the prompt. The model builds intermediate values and the final response into new variables, potentially invoking sub-RLMs within loops. RLMs have achieved impressive results on long-context benchmarks [21, 22], and their applications in fields such as coding [23] and literature synthesis appear particularly promising. Figure 1.1 illustrates all three approaches.

However, the true capabilities of RLMs remain understudied. We have identified three key limitations in the original RLM work:

1. Training methodology: The original paper primarily studies RLMs using prompting rather than explicit training, and when training is employed, it relies on distillation. Both approaches are limited, prompting prevents models from learning from their mistakes, while distillation only enables smaller models to learn from more capable ones without explicitly improving model capabilities beyond the teacher’s knowledge.

2. Recursion depth constraints: The original paper restricts maximum recursion depth to 1, imposing an artificial human constraint on RLMs. Following the general principle of the approach, it would be beneficial to allow models to determine optimal recursion depth based on task requirements.

3. Synchronous execution: All calls to sub-RLMs in the original implementation are synchronous, leading to prohibitively long inference times. Allowing models to autonomously choose between synchronous and asynchronous calls could yield substantial efficiency gains.

In light of these limitations, we have devised a research program to study the efficient training and inference of recursive language models. Making RLMs more accessible is crucial for lowering barriers to AI adoption, particularly as interactions with these models become increasingly ubiquitous and costly.

Our approach employs Reinforcement Learning from Verifiable Rewards (RLVR) with the Dr. GRPO [24] algorithm, combined with Low-Rank Adaptation (LoRA) [25], to train cost-aware RLMs. The goal is to create models optimized to maximize answer accuracy while minimizing computational cost and wall-clock inference time.

This research aims to contribute to two broader contexts in LLM development. First, it advances approaches for enabling models to work effectively over long tasks involving extensive text or extended time periods, allowing agents to operate longer without suffering from context degradation. Second, it participates in the larger effort to make LLM customization more accessible and cost-effective [26]. Reinforcement learning with LoRA has garnered considerable recent interest [27–29], yet its applications and best practices remain underexplored, making this an opportune moment for systematic investigation.

References