# Record **Author:** @cameron.stream (`did:plc:gfrmhdmjvxn2sjedzboeudef`) ## `3mvehmaclck26` **Collection:** `site.standard.document` **AT URI:** `at://did:plc:gfrmhdmjvxn2sjedzboeudef/site.standard.document/3mvehmaclck26` **Title:** Recurrent Looped Transformer **Published:** Sun, 13 Sep 2026 01:35:54 GMT **Updated:** Sun, 13 Sep 2026 03:13:18 GMT **Description:** How feeding a decoder’s final hidden vector into its next token changes computation, parallelism, and training costs. **Publication:** `at://did:plc:gfrmhdmjvxn2sjedzboeudef/site.standard.publication/3mr4py6clps2f` **Path:** /recurrent-looped-transformer **Tags:** knowledge, concept, ai, language-models, model-architecture, machine-learning, sequence-models **Content:** ```json { "text": "The **Recurrent Looped Transformer (RLT)** is a proposed language-model architecture that carries a decoder's internal state across every prompt and response token. A causal encoder supplies context memory, while a recurrent decoder combines the current token's representation with its previous output. The [technical report](https://github.com/yifanzhang-pro/recurrent-looped-tranformer/blob/master/Recurrent_Looped_Transformer.pdf), dated September 12, 2026, specifies the architecture, execution schedule, and training semantics. It reports no measured reasoning gains, hardware speedups, or scaling results.\n\nRLT's phrase “infinite temporal depth” describes a computation path that can keep growing as the sequence grows. Each finite sequence still requires finite computation. The proposal creates a route for earlier internal computation to influence later tokens; whether training makes effective use of that route remains an empirical question.\n\n## What changes behind the token stream\n\nRLT can expose the same interface as an ordinary language model: a sequence of output tokens, including written reasoning if the model produces it. The proposed change concerns how the network computes the next token. It introduces no required new type of visible output.\n\nIn a conventional causal transformer, each token passes through a fixed stack of layers. Attention at those layers can consult cached keys and values from earlier positions. Those caches already preserve continuous internal representations. Selecting one output token does not erase the network's entire computational history or force all information reaching the next step through that token alone.\n\nRLT adds a specific connection: the previous token's final decoder hidden vector enters the bottom of the decoder when it processes the next token. Information can therefore travel through the full decoder stack repeatedly as the sequence advances. This top-to-bottom feedback supplements the attention memory.\n\n## Which vector feeds back\n\nThe feedback vector is the output of the final decoder block for the previous token, before output normalization and the vocabulary projection. Its size is the model's hidden width. It is neither a vocabulary-sized vector of logits nor the embedding of the sampled token.\n\nThat hidden vector has two uses:\n\n1. Output normalization and a vocabulary projection turn it into logits, the scores used to form the next-token distribution.\n2. A separate normalization, projection, and gated merge combine it with the next token's encoder representation. The combined vector enters the decoder again.\n\nSampling selects a discrete token from a distribution over the vocabulary. That selection is lossy as a description of the distribution, but the vocabulary distribution is itself only a readout of the hidden representation. RLT feeds back the representation before that readout, rather than preserving all vocabulary alternatives as a probability vector.\n\nThe feedback vector also differs from the complete state of the network. Decoder attention caches remain separate, layer-specific stores. The architecture feeds those caches to the corresponding attention layers instead of flattening every activation into one feedback vector.\n\n## How information moves through the model\n\nRLT separates three forms of information that would otherwise be easy to conflate:\n\n| Component | Contents | Role |\n| --- | --- | --- |\n| Encoder memory | Key and value projections derived from causal encoder representations | Lets the decoder attend to the encoded token history |\n| Recurrent output | The final decoder hidden vector from the previous token | Feeds the previous decoder result into the next decoder input |\n| Decoder attention cache | Recent key and value projections at each decoder layer | Gives the decoder local access to its own earlier activations |\n\nEncoder memory is a growing table of numerical representations of the text. For an input such as “The cat sat on the,” the encoder produces a contextual vector at each token position. The vector at a position depends on that token and what came before it. This restriction makes the encoder *causal*.\n\nThe model projects these vectors into keys and values. Keys let attention score which positions are relevant; values supply the information combined from those positions. The decoder reads this table through cross-attention. Newly consumed tokens add entries. Here, memory means cached vectors within a sequence, rather than saved notes between conversations or facts stored in model weights.\n\nFor each token, the encoder produces a representation. A gated merge combines that representation with the previous final decoder output. The decoder then applies sliding-window self-attention, cross-attention to encoder memory, and feed-forward computation. Its final output predicts the next token and becomes part of the state used at the following step.\n\nSliding-window attention restricts direct access to recent decoder positions. With a window of $W$ positions including the current token, each layer retains at most $W-1$ historical positions for the next update. Encoder memory provides a separate path to the broader history. Neither store is an unrestricted archive of every previous final decoder state.\n\nThe complete decoder state therefore has two parts:\n\n$$\nH_t = (s_t, C_t^D).\n$$\n\nHere $s_t$ is the final decoder output after token $t$, and $C_t^D$ contains the retained decoder attention caches. Omitting the caches would omit part of the model's state. The next transition can be written as:\n\n$$\nH_t = D_\\phi\\bigl(\\mathrm{Merge}(e_t,s_{t-1});M_{\\leq t},C_{t-1}^D,t\\bigr).\n$$\n\nIn this expression, $e_t$ is the current encoder representation, $M_{\\leq t}$ is encoder memory through the current position, and $D_\\phi$ is the decoder with parameters $\\phi$.\n\n## What the loop adds\n\nRLT explicitly feeds the previous token's final decoder output back into the next token's decoder input. Repeating that transition creates a path through many applications of the same decoder. The report's tied configuration also reuses compatible attention and feed-forward weights between encoder and decoder stages; weight reuse and state recurrence are separate design choices.\n\nThe illustrative configuration has 48 encoder layers and 48 decoder layers. After $t$ processed tokens, its recurrent path passes through $48t$ decoder blocks. Each token still evaluates 96 encoder-plus-decoder blocks, excluding merge and readout operations. Decoder blocks also perform cross-attention, so counting layers alone does not establish their computational cost.\n\nFor example, after 100 processed tokens, the recurrent path contains 4,800 decoder-block applications. This is a path through the computation already performed across those tokens. It does not mean the model performs 4,800 fresh block applications to answer the hundredth token, or can think indefinitely before emitting any token.\n\nLong paths also need to carry useful information. Learned gates, projections, and contracting state updates may weaken earlier contributions. The architecture's path length therefore establishes an available dependency, while reasoning benchmarks would have to establish a capability gain.\n\n## The prompt must pass through the decoder\n\nRLT applies the same decoder transition to observed prompt tokens and generated response tokens. The encoder can process a known prompt with token-parallel operations under a causal mask. The decoder must then process the prompt tokens in order to construct its recurrent output and attention caches.\n\nAt decoder position $t$, cross-attention can read only encoder memory through $t$, even if the encoder has already computed the entire prompt. Allowing access to later prompt positions would change the causal model. A conventional parallel decoder pass also cannot simply replace these recurrent updates, because historical decoder keys and values depend on earlier updates.\n\nGeneration continues from the completed prompt state without resetting either state component. The model samples the first response token from that state, then encodes and consumes the sampled token exactly once. New user or tool text likewise updates the state before later predictions.\n\nThis design has a latency cost. A prompt with $T$ tokens requires $T$ sequential decoder transitions. Batching independent sequences can improve hardware utilization, but does not remove the dependency within one sequence. The report explicitly claims no reduced-prefill speedup.\n\n## Training and replay must reconstruct the same state\n\nRLT specifies one transition for pretraining, supervised fine-tuning, generation, and reinforcement-learning replay. In supervised fine-tuning, the loss can select assistant targets while the state still advances through all context tokens. Selecting which tokens receive a loss does not remove the computations needed to construct later states.\n\nReinforcement learning introduces another requirement: a cached state depends on the model weights that produced it. After a weight update, exact evaluation under the current policy requires rebuilding parameter-dependent states and caches from the history. Reusing old rollout caches would introduce a stale-state approximation.\n\nFull *backpropagation through time* differentiates through the sequence of recurrent updates. Gradients must account for recurrent outputs, decoder keys and values, and encoder-derived information. Detaching one of these tensors can preserve its forward numerical value while removing a gradient path. Such a calculation is an approximation to the report's full-gradient reference.\n\nThe report also distinguishes current-policy probabilities from the behavior probabilities recorded when tokens were sampled. If sampling uses temperature or truncation, those recorded probabilities must describe the actual sampling distribution. Exact importance sampling additionally requires adequate support: logging a chosen token's probability cannot restore actions the sampler excluded.\n\nUsing the same mathematical transition across modes removes a structural source of disagreement. Different numerical kernels, precision settings, or stochastic execution can still produce discrepancies. Consistent replay also does not establish that a particular reinforcement-learning objective is unbiased or effective.\n\n## Memory and efficiency tradeoffs\n\nRLT's bounded decoder window limits one part of inference memory. Encoder-side caches and global encoder memory still grow with sequence length. Cross-attention work also grows with the amount of encoder memory being read. Fixed block count per token therefore does not imply fixed time or constant total memory per token.\n\nWeight tying reduces the number of separately stored parameters, but both logical stages still execute. The report identifies batching, memory reuse, kernel fusion, and activation checkpointing as optimization opportunities. It does not supply measured savings from those opportunities.\n\nTraining has additional storage requirements. Evicting an old key or value from the inference cache does not erase the earlier computations that consumed it. Full-gradient training must retain or recompute the necessary activations. Activation checkpointing trades recomputation for storage; truncating gradients changes the training calculation.\n\n## Can recurrence retain parallelism?\n\nSome recurrent architectures support parallel processing of known sequences by restricting the form of their state updates. Consider a scalar recurrence:\n\n$$\nh_t = a_t h_{t-1} + b_t.\n$$\n\nIf the coefficients $a_t$ and $b_t$ can be computed without knowing the previous state, adjacent updates can be combined before evaluating that state. Applying one update and then another gives:\n\n$$\nh_2 = (a_2 a_1)h_0 + (a_2 b_1+b_2).\n$$\n\nThe combined operation has the same multiply-and-add form. An associative *parallel prefix scan* composes updates in a tree to compute all prefix states. It preserves their order while reducing the sequential dependency depth. Efficient vector implementations additionally require suitable structure, such as elementwise state transitions. Autoregressive generation still waits for newly sampled tokens; the scan benefits known sequences used in training and prompt processing.\n\n[Mamba](https://arxiv.org/abs/2312.00752) uses structured state updates that are linear in the previous state, with coefficients that depend on the input. Its hardware-aware scan demonstrates that continuous recurrent state can coexist with parallel sequence processing. This restriction changes what the recurrent update can compute directly; surrounding layers and nonlinearities supply additional modeling capacity.\n\nTraditional long short-term memory networks, or LSTMs, calculate gates using the previous hidden state. The next update's coefficients are therefore unavailable until that state has been computed. LSTMs were widely used before transformers, but their sequential execution limited training parallelism. The [minLSTM study](https://arxiv.org/abs/2410.01201) removes previous-hidden-state dependencies from gates to enable parallel scans, while evaluating the resulting simpler models on a limited set of tasks.\n\nRLT's feedback passes through a full nonlinear decoder and its attention caches. The report supplies no efficient exact scan for that transition. Replacing the update with a scan-friendly recurrence would change the model. Thus serial execution is a cost of this specified feedback mechanism, rather than a universal requirement for carrying latent information forward.\n\n## Relationship to earlier architectures\n\nThe report places RLT among several existing approaches to reusing computation:\n\n- [Feedback Transformer](https://arxiv.org/abs/2002.09402) provides precedent for exposing earlier representations to later computation through feedback memory.\n- [Universal Transformers](https://arxiv.org/abs/1807.03819) reuse a transformation across depth. RLT's temporal state instead advances with processed tokens.\n- [Recurrent-depth reasoning](https://arxiv.org/abs/2502.05171) studies repeated internal computation as a way to scale test-time compute. RLT's stated depth count follows the token history.\n- [YOCO](https://arxiv.org/abs/2405.05254) motivates reusable attention memory. RLT retains its full recurrent prompt pass and therefore does not inherit a decoder-prefill skipping benefit from that design.\n\nThese comparisons locate the proposal's design choices. Establishing that those choices improve quality or efficiency requires matched experiments, including appropriate recurrent and weight-sharing baselines.\n\n## Evidence available at introduction\n\nThe September 12 report provides equations, analytical work and memory counts, execution schedules, and arguments about causality and replay consistency. Its figures illustrate the proposed architecture and computational dependencies. They are not benchmark measurements.\n\nThe unresolved empirical questions include reasoning quality at matched training and inference budgets, long-sequence optimization stability, prompt latency, throughput, and the usefulness of exact replay in reinforcement learning. The report itself leaves realized reasoning quality, hardware efficiency, and scaling behavior for future validation. Claims of superintelligence go beyond the evidence it presents.\n\nAn economic evaluation would compare the cost of reaching a target answer quality, including training cost, prompt processing, and generation. A quality improvement or fewer required output tokens could potentially offset more expensive execution, but neither benefit is demonstrated here. The sequential dependency is a concrete engineering concern; commercial usefulness remains unestablished rather than disproved.\n\nThe [repository](https://github.com/yifanzhang-pro/recurrent-looped-tranformer) and [project page](https://yifanzhang-pro.github.io/recurrent-looped-tranformer/) provide the report and supporting material. This explanation concerns that initial proposal; later implementations or results should be evaluated separately.\n\n## Sources\n\n- [Recurrent Looped Transformer technical report]()\n- [Recurrent Looped Transformer repository]()\n- [Recurrent Looped Transformer project page]()\n- [Attention Is All You Need \\(Vaswani et al\\., 2017\\)]()\n- [Feedback Transformer \\(Fan et al\\., 2020\\)]()\n- [Universal Transformers \\(Dehghani et al\\., 2018\\)]()\n- [Scaling up Test\\-Time Compute with Latent Reasoning \\(Geiping et al\\., 2025\\)]()\n- [You Only Cache Once \\(Sun et al\\., 2024\\)]()\n- [Mamba: Linear\\-Time Sequence Modeling with Selective State Spaces \\(Gu and Dao, 2023\\)]()\n- [Were RNNs All We Needed? \\(Feng et al\\., 2024\\)]()", "$type": "site.standard.content.markdown", "version": "1.0" } ``` --- *Fetched from https://enoki.us-east.host.bsky.network via `com.atproto.repo.getRecord`*