AI Agent Memory: How LLM Agents Remember Across Sessions

TL;DR

A language model forgets everything the moment it finishes a reply. Every API call starts from zero, so “memory” in an AI agent is something you engineer around the model, a layer that decides what to store, what to throw away, and what to paste back into the next prompt. Researchers now split that layer into four kinds of memory (working, episodic, semantic, procedural), and a 2025 paper called MemoryOS shows that borrowing ideas from operating systems (paging, FIFO eviction, tiered storage) lifts long-conversation answer quality by roughly 49% F1 on the LoCoMo benchmark. Below is the mental model, the research behind it, and a small tiered-memory implementation you can run today.

The model has no idea who you are

The trap for first-time agent builders: the LLM itself has no memory at all. GPT-4o-mini and Claude and every other model answer each request as a pure function of the tokens you send in that one call. When a chatbot “remembers” your name from three messages ago, your app just pasted those three messages back into the prompt.

That works fine for a short chat. It falls apart the second your agent runs for an hour, books a flight, reads forty files, and calls twelve tools. You can’t paste an hour of history into every prompt. Context windows are large now, but they’re not infinite, they cost money per token, and stuffing them full actually hurts accuracy (the model loses the needle in its own haystack). So you need a system that keeps the relevant slice and drops the rest. That system is what people mean when they say “agent memory.”

I hit this wall the first week I tried to build a research assistant that ran across sessions. Day one it was great. Day three it kept re-asking things I had already told it, because my “memory” was a flat list of every message I naively replayed until I blew the token budget and started truncating from the top, which threw away the useful early facts and kept the chit-chat. That failure is the whole reason the memory research below exists.

Four kinds of memory, borrowed from human cognition

The framing that finally made this click for me comes straight from cognitive psychology. Human memory comes in several kinds, and agent memory should too. Practitioners and a 2024 survey on AI long-term memory converge on four types, and each maps to a different storage strategy.

Working memory is the easy one. It is the recent conversation you keep in the prompt. The interesting design questions all live in long-term memory, and that is where the three other types matter.

Episodic memory is the agent’s diary. A travel agent that recalls you booked London in March, stayed near the venue, and complained about the airport transfer has episodic memory. It stores events, keyed by time, and pulls the relevant ones back when a similar situation shows up.

Semantic memory is the distilled version. Instead of the whole transcript, it keeps the fact: “user prefers city-center hotels.” This is what most people build first, usually as embeddings in a vector database, because it answers “what do I know about this user” without replaying every conversation.

Procedural memory is the one almost everyone skips. It’s how the agent does things: the multi-step routine for booking a trip, the order it calls tools, the format it returns. Right now most teams bake this into prompts and code by hand. The frontier research is about agents that manage and update their own memory from experience, like the self-editing scheme in my Agentic Memory breakdown, and it is a much harder problem that is mostly still in the lab.

MemoryOS: treat memory like an operating system

The paper I keep coming back to is Memory OS of AI Agent (Kang, Ji, Zhao, Bai). Its core move is an analogy that sounds obvious once you hear it: an operating system already solved “how do you give a program the illusion of unlimited fast memory when you only have a little.” The answer OSes use is a hierarchy (registers, RAM, disk) plus rules for moving data between tiers. MemoryOS applies the same shape to an agent.

It defines three tiers:

• Short-term memory holds the immediate conversation, like RAM holding the active process.
• Mid-term memory is a staging area updated with a dialogue-chain FIFO scheme. Recent topic threads live here; when the buffer fills, the oldest chain gets evicted, exactly like a page-replacement policy.
• Long-term personal memory is the disk. It stores durable user-specific information using a segmented “page” organization, so the system can fetch a relevant page without scanning everything.

Four modules move data around those tiers: storage, updating, retrieval, and generation. The updating module is the clever part: it decides what graduates from mid-term to long-term, which is the agent equivalent of deciding what to write from RAM back to disk.

The numbers are what sold me on the framing being more than a cute metaphor. Tested on LoCoMo, a benchmark built specifically for long, multi-session conversations, and running on GPT-4o-mini as the base model, MemoryOS reports:

A ~49% F1 gain from memory management alone, with the same underlying model, tells you how much answer quality on long tasks depends on getting the memory layer right rather than swapping in a bigger model. The paper reports these as improvements over baselines on their setup, so treat them as the authors’ claimed results, but the direction matches what I have seen building agents by hand: the model was rarely my bottleneck; retrieval usually was.

Building a tiered memory you can run

The three-layer idea fits in about forty lines of pure Python with no dependencies. It’s deliberately small so the mechanics stay visible: a working buffer, an episodic log, and a semantic store you query by similarity.

import math
from collections import deque, Counter

def tokenize(text):
    return ''.join(c.lower() if c.isalnum() else ' ' for c in text).split()

def embed(text):
    # toy bag-of-words vector; real agents use dense embeddings + a vector DB
    return Counter(tokenize(text))

def cosine(a, b):
    common = set(a) & set(b)
    dot = sum(a[t] * b[t] for t in common)
    na = math.sqrt(sum(v * v for v in a.values()))
    nb = math.sqrt(sum(v * v for v in b.values()))
    return dot / (na * nb) if na and nb else 0.0

class TieredMemory:
    def __init__(self, working_size=6):
        self.working = deque(maxlen=working_size)   # short-term / working memory
        self.episodic = []                          # timestamped event log
        self.semantic = []                          # (fact, vector) knowledge

    def observe(self, role, text):
        self.working.append((role, text))
        self.episodic.append((len(self.episodic), role, text))

    def remember_fact(self, fact):
        self.semantic.append((fact, embed(fact)))

    def recall(self, query, k=2):
        qv = embed(query)
        scored = ((cosine(qv, vec), fact) for fact, vec in self.semantic)
        ranked = sorted((s for s in scored if s[0] > 0), reverse=True)
        return ranked[:k]

    def build_context(self, query):
        facts = "\n".join(f"- ({s:.2f}) {f}" for s, f in self.recall(query))
        recent = "\n".join(f"{r}: {t}" for r, t in self.working)
        return f"# Relevant facts\n{facts}\n\n# Recent turns\n{recent}"

The deque(maxlen=6) is working memory: it forgets automatically once it holds six turns, which is the FIFO eviction MemoryOS formalizes. The semantic list is your long-term store, and recall is a poor man’s retrieval, just cosine similarity over word-count vectors. Now drive it:

mem = TieredMemory()
mem.remember_fact("The user prefers city-center hotels when traveling.")
mem.remember_fact("The user is vegetarian and avoids seafood.")
mem.remember_fact("The user booked a trip to London for a conference in March.")

mem.observe("user", "I am heading back to London next month.")
mem.observe("assistant", "Want me to look at the same area as last time?")
mem.observe("user", "Yes, and a hotel I can walk to the venue from.")

print(mem.build_context("which hotels does the user prefer?"))

Running it prints exactly this:

# Relevant facts
- (0.43) The user prefers city-center hotels when traveling.
- (0.31) The user is vegetarian and avoids seafood.

# Recent turns
user: I am heading back to London next month.
assistant: Want me to look at the same area as last time?
user: Yes, and a hotel I can walk to the venue from.

Two things to notice. First, the hotel-preference fact ranks top, which is what you want: that string gets prepended to the model’s prompt and the agent answers as if it “remembered.” Second, and more useful as a lesson: the London-trip fact scored 0 and never surfaced, even though a human would say it is relevant to a hotel question. Bag-of-words retrieval only matches shared words, and “hotels” does not appear in the London fact. That gap is the entire reason production agents swap my embed toy for a real dense-embedding model and a proper vector database, so “book a room” retrieves “reserved a hotel” by meaning, not spelling. If you want to see how the managed tools handle that layer, I compared three of them in my mem0 vs Cognee vs Letta breakdown.

The three problems every memory system fights

Building the happy path took an afternoon; the hard parts took weeks. The same three problems show up in every serious memory system.

Retrieval quality is the first wall. Getting the right five facts into a prompt is harder than storing five thousand. Too greedy and you flood the context with noise, which drags accuracy down and burns tokens (the same waste I measured in my FastContext writeup). Too stingy and the agent forgets the one thing that mattered. This is where the overlap with retrieval-augmented generation is total. An agent’s semantic memory is a RAG index over its own history instead of over documents.

Second comes forgetting. Storing everything forever sounds safe and is a trap. My episodic log grew without bound, retrieval slowed, and stale facts (“user is planning a wedding”) outranked current ones. Real systems need decay: recency weighting, summarization of old episodes into semantic facts, and outright deletion. MemoryOS handles this with its FIFO eviction and mid-to-long-term promotion, but tuning what to keep is an open problem, not a solved one.

The third is evaluation. You can’t eyeball whether a memory system actually works, which is why LoCoMo exists: it scripts very long, multi-session conversations and then asks questions whose answers depend on facts from far earlier, scoring whether the agent recalls them. If you build a memory layer and don’t measure it against something like LoCoMo, you’re guessing. I learned that the annoying way, by shipping a “smarter” memory that felt better and tested worse.

What this changed about how I build agents

The practical shift for me was to stop treating memory as a feature I would add later and start treating it as the architecture. Now I decide up front which of the four types a given agent actually needs. A support bot needs strong semantic memory (user facts) and almost no procedural memory. A coding agent that runs long sessions needs aggressive working-memory management and episodic recall of what it already tried, so it stops re-reading the same file. Most agents do not need all four, and building the ones you do not need is wasted complexity.

The OS analogy is the part worth burning into your brain. When you are unsure how to structure an agent’s memory, ask what a kernel would do: keep the hot stuff close and cheap, page the warm stuff to a middle tier, write the cold stuff to disk and fetch it only on a miss. It is a fifty-year-old idea, and it turns out language models needed it too. If you are building the agent side rather than the memory side, my Go AI agent tutorial walks through the loop these memories plug into.

FAQ

What is AI agent memory?

AI agent memory is the system an application builds around a language model to retain and recall information across turns and sessions. The model itself is stateless (it forgets everything after each response), so the memory layer stores past interactions and facts, then selects and injects the relevant pieces into each new prompt.

What are the types of AI agent memory?

Four types. Working (short-term) memory holds the current task in the context window. Episodic memory logs past events with timestamps. Semantic memory stores distilled facts and preferences, usually as embeddings. Procedural memory captures learned skills and workflows. Working memory lives in the prompt; the other three live in external storage.

How does memory work in AI agents?

The agent writes observations to storage (a log, a vector database, a graph), and before each model call it runs a retrieval step that pulls the most relevant stored items and pastes them into the prompt alongside the recent conversation. The model then answers as if it remembered, when really the application did the remembering and handed it the notes.

What is the difference between short-term and long-term memory in AI agents?

Short-term (working) memory is the recent context held directly in the prompt; it is fast, small, and cleared often. Long-term memory is external storage that persists across sessions and is queried by relevance. It holds far more than fits in a context window but needs a retrieval step to access.

How do you implement memory in an AI agent?

Start with a working buffer of recent turns, add a semantic store (embeddings in a vector database) for durable facts, and add an episodic log if the agent needs to recall specific past events. Retrieve top-k relevant items before each call. Frameworks like Mem0, Cognee, Letta, and LangGraph provide this out of the box if you would rather not build it yourself.

Sources

• Memory OS of AI Agent — arXiv:2506.06326 — the OS-inspired tiered memory architecture and the LoCoMo results cited above
• LoCoMo: Evaluating Very Long-Term Conversational Memory — arXiv:2402.17753 — the benchmark used to measure long-conversation recall
• Human-inspired Perspectives: A Survey on AI Long-term Memory — arXiv:2411.00489 — survey grounding the episodic/semantic/procedural taxonomy
• IBM: What is AI agent memory? — practitioner framing of the memory types

Bottom line

The model is the easy part now. What separates an agent that feels dumb from one that feels attentive is almost always the memory layer around it — what it keeps, what it drops, and what it manages to find again at the right moment. Steal the operating-system playbook: tier your storage, evict aggressively, retrieve carefully, and measure recall on something like LoCoMo instead of vibes. Do that and a small model with good memory will beat a big model with none.