REPO: Language Models with Context Re-Positioning

This document introduces REPO (Context Re-Positioning), a novel mechanism for Large Language Models (LLMs) addressing the limitations of rigid, fixed contextual structures. Current LLM architectures assign linear or constant positional indices, which, inspired by Cognitive Load Theory (CLT), is argued to increase extraneous cognitive load and hinder deep reasoning. REPO proposes to reduce this load by allowing LLMs to dynamically re-organize token positions. It employs a differentiable module, fϕ, to assign continuous, non-linear position values based on contextual dependencies rather than pre-defined integer ranges. This approach enables LLMs to free up "working memory" capacity for more effective "germane processing" by intelligently restructuring context. Continually pre-trained on the OLMo-2 1B backbone, REPO demonstrates significant performance enhancements. It shows improvements on tasks involving noisy contexts, structured data, and extended context lengths, while maintaining strong performance on short-context tasks. Analysis reveals REPO's ability to allocate higher attention to distant yet relevant information, assign positions in a dense and non-linear space, and capture intrinsic input structure.
#LLMs #ContextWindow #PositionalEncoding #CognitiveLoad #REPO #MachineLearning #NLP #DeepLearning

paper - https://arxiv.org/abs/2512.14391
subscribe - https://t.me/arxivpaper
donations:
USDT: 0xAA7B976c6A9A7ccC97A3B55B7fb353b6Cc8D1ef7
BTC: bc1q8972egrt38f5ye5klv3yye0996k2jjsz2zthpr
ETH: 0xAA7B976c6A9A7ccC97A3B55B7fb353b6Cc8D1ef7
SOL: DXnz1nd6oVm7evDJk25Z2wFSstEH8mcA1dzWDCVjUj9e
created with NotebookLM