Metis: Memory Foundation Model
Abstract
Recent advances in AI agents have increasingly internalized native capabilities into their underlying foundation models, giving rise to multimodal foundation models and large reasoning models. However, agent memory is still primarily implemented through external modules, leaving the native memory capability largely unexplored. In this paper, we take a first step toward this direction by introducing memory foundation models, which empower foundation models with native memory capabilities. We formalize native memory from two perspectives: a persistent and dynamically evolving memory state within the backbone, and native memory procedures that autonomously store and utilize information through model computation. We show that native memory offers advantages in architecture, end-to-end optimization, and efficiency. Based on this formulation, we propose Metis, the first prototype of memory foundation models. Metis introduces a new architecture that equips a foundation model with a native memory state, allowing historical information to be compressed into the model and accessed through memory attention. We construct large-scale memory-specific training data and introduce multiple optimization objectives to acquire these native memory procedures through mid-training. The online memory maintenance of Metis is gradient-free, and the memory update requires only a forward pass. At inference time, all learned model weights remain frozen, while the native memory states are autonomously transformed through standard forward computation. Through extensive experiments, we show that Metis exhibits native memory capabilities and further provide a detailed analysis of its strengths, limitations, and behaviors. To facilitate future research on memory foundation models, we release our project and model checkpoints.
Community
Metis: The first prototype of a memory foundation model, equipping foundation models with a persistent and dynamically evolving native memory state.
What if memory were a native capability of foundation models, rather than an external module?
Recent advances have increasingly internalized capabilities such as multimodal perception and reasoning into foundation models. Agent memory, however, is still primarily implemented through external retrieval, storage, and prompt-construction pipelines.
We introduce memory foundation models, which treat memory as a first-class capability of the model itself. We formalize native memory from two complementary perspectives:
- Native memory state: a persistent and dynamically evolving state within the model backbone that participates directly in subsequent computation.
- Native memory procedures: learned procedures that autonomously determine what to store and how to utilize the stored information.
At its core, each Metis Block combines a Local Memory Block, which maintains the persistent memory state, with a Hyper Memory Block, which learns the storage and utilization procedures. Historical information is compressed into a compact, fixed-size session state and accessed through memory attention, allowing later queries to use the memory without replaying the original history.
Memory maintenance is gradient-free and requires only forward computation. At inference time, all learned model weights remain frozen, while the native memory state evolves through standard model computation. We release the implementation and Qwen3.5-based checkpoints at 4B, 9B, and 27B scales.
💡 Overview
Metis is the first prototype of a memory foundation model, equipping foundation models with a persistent and dynamically evolving native memory state.
It learns to autonomously store and utilize information through model computation, compressing historical context into native memory and accessing it through memory attention. At inference time, all model weights remain frozen, while memory is updated through gradient-free forward computation.
The project explores three central ideas:
- Native memory state. Dynamic parametric states live inside the backbone and participate directly in later forward passes.
- Native memory procedures. Storage and utilization are learned from data instead of being implemented as separate retrieval, reranking, and prompt-construction rules.
- Fixed-size session state. Historical information is compressed into a compact state, so later queries do not need to replay the original text history.
Architecture
A Metis Block is inserted into Transformer layers and contains two components:
- The Local Memory Block maintains the dynamic memory matrix and normalization state that persist across interaction steps.
- The Hyper Memory Block learns token selection, memory key/value projections, a dedicated memory query, and the state-update procedure.
After each memory step, Metis selects informative hidden states and updates the local memory. During a later query, memory attention reads that state and fuses the result with the original attention branch. The default implementation uses a Gated Delta Network (GDN) update. In the Qwen3.5 hybrid implementation, Metis is attached to full-attention layers, while linear-attention layers keep their original computation path.
Roadmap
The paper frames native memory as a progression from stateful capability to self-managing memory, experience-driven learning, persistent cognition, and ultimately self-evolving capability.
Metis is an early research system rather than a complete replacement for external memory. Hybrid native–external memory remains an important direction.
Get this paper in your agent:
hf papers read 2607.26760 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 3
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper