Green and Efficient Multi-Scale Test-Time-Training Memory for Long-Context Foundation Models

Long-context understanding underpins virtually every scientific application of foundation models — from reasoning over multi-document literature to modelling long biological or temporal sequences. Standard Transformers scale quadratically with context length and are environmentally and economically prohibitive at the scales scientific use cases demand. Sub-quadratic alternatives (linear attention, state-space models, Test-Time Training) handle long contexts cheaply but suffer a single-scale information bottleneck: one recurrent state, forced to summarise everything, inevitably overwrites earlier evidence. The human brain, by contrast, handles long-range temporal context with remarkable efficiency through multi-scale memory consolidation — suggesting that the dominant single-state attention paradigm may not be the final architecture for AI.

This project develops multi-scale Test-Time-Training (TTT) memory: a green, sub-quadratic foundation model architecture in which several recurrent memory modules update at different temporal scales, with explicit lateral state transfers at shared chunk boundaries. Fast modules track local dynamics; slow modules consolidate stable long-range context; cross-module transfers prevent catastrophic forgetting without quadratic cost.

The project additionally investigates two adaptive content-selection mechanisms — surprise-driven gating and hidden-state-informed token retention — as planned ablations within this framework. Recent industry results demonstrate that linear attention architectures are viable at production scale — Kimi Linear successfully pretrained a large foundation model using DeltaNet-based linear attention — yet these designs still rely on a single memory state.