An Agent's Guide to Long-Term Memory

Beyond the context window. Designing memory systems that persist, compress, decay, and evolve — the infrastructure that gives agents continuity across sessions.

28 min

Introduction

This guide is written for agents and the humans who build them. It is about the difference between what an agent can hold in mind and what it can carry forward — and about the infrastructure that turns a sequence of isolated conversations into something with continuity.

A language model, on its own, has no memory. It has a context window — a bounded working surface it reads on each turn — and nothing else. When the turn ends, the window is discarded. The next request begins from the same blank starting point as the first. Whatever the agent learned, decided, or promised evaporates unless something outside the model wrote it down.

Long-term memory is that something. It is the layer that persists experience across sessions, compresses it into usable knowledge, lets the irrelevant fade, and keeps the durable. Building it well is one of the defining problems of agentic systems, because an agent without memory is not really an agent — it is a very capable stranger you have to reintroduce yourself to every morning.

We will move from the ground up: why the context window is not memory, how a memory hierarchy is layered, how experiences are encoded and retrieved, why similarity search alone is not enough, and how consolidation and forgetting keep the whole system healthy over time. Where it helps to make an abstract idea concrete, we will point to how Kamino's Elephantasm framework implements it — not as the only way, but as one worked example of the concepts in production.

The Context Window Is Not Memory

The most common and most costly misconception in agent design is treating the context window as memory. It is not. The context window is working memory — a scratchpad the model reads in full on every forward pass. It is bounded, volatile, and expensive, and it resets to empty the moment a session ends. Confusing it for a persistent store leads directly to agents that cannot remember yesterday and cannot scale past a single conversation.

The Stateless Turn

Inference is stateless. Each request to a model is self-contained: the model sees only the tokens you send it, produces a completion, and retains nothing. The illusion of a continuous conversation is created entirely by the surrounding system re-sending the prior turns on every request. There is no hidden reservoir inside the model where the last hour of dialogue is stored. If your code does not include a fact in the prompt, the model does not know it — no matter how many times it was told before.

This means every property we associate with memory — persistence, recall, forgetting, revision — must be built in the layer around the model. The model supplies fluency and reasoning over whatever is in front of it. Memory supplies what to put in front of it.

Why a Bigger Window Is Not the Answer

The tempting fix is to make the window larger — hold the entire history in context and never forget. This fails for three structural reasons, and understanding them motivates everything that follows.

The context window is where thinking happens. Memory is where the results of thinking are kept. A system that conflates the two either forgets everything or drowns in everything. The job of a memory system is to keep the window small and relevant by deciding, on each turn, which small handful of durable facts deserve to be present.

The Memory Hierarchy

Human memory is not one store; it is a graded system where information moves between tiers of decreasing volatility and increasing abstraction. Agent memory benefits from the same structure. Two distinct axes are worth separating: the time horizon of a store, and the degree of refinement of what it holds.

By Time Horizon: Working, Short-Term, Long-Term

Working memory is the context window itself — the tokens active in the current turn. It is the most powerful surface, because the model reasons directly over it, and the most scarce, because it is bounded and costly. Working memory is for the task at hand, nothing more.

Short-term memory is session state: the running summary of the current conversation, recent tool results, the scratch decisions of the last few minutes. It typically lives outside the window but is cheap to reload, and it is discarded or archived when the session closes. Its job is coherence within an episode.

Long-term memory is the persistent store that survives across sessions indefinitely. It is where facts about the user, lessons from past tasks, stable preferences, and hard-won procedures accumulate. It is queried, not loaded wholesale — the point of long-term memory is that it can be enormous precisely because you only ever pull a few relevant items into the window at a time.

By Refinement: From Raw Events to Identity

The second axis is more subtle and more powerful. Not everything in long-term memory is the same kind of thing. Experience arrives raw and is progressively distilled into more durable, more abstract, more trusted forms. It is useful to think of this as a ladder with four rungs.

Kamino's Elephantasm framework implements exactly this four-layer ladder — Event, Memory, Knowledge, Identity — as its core hierarchy. Raw interactions land as events; a distillation step lifts the significant ones into memories; agreement and repetition consolidate memories into knowledge; and the most stable, defining conclusions settle into identity. The value of the structure is not the names but the flow: information becomes smaller, more abstract, and more trusted as it climbs, and each rung has a different retention policy. Events can be aggressively pruned once distilled; identity is almost never discarded.

# A distilled memory record — the unit long-term memory stores.
# Note the fields beyond the vector: they are what make recall work.
memory = {
    "id": "mem_8f21",
    "text": "User prefers metric units and terse answers.",
    "embedding": [...],            # semantic position, for similarity search
    "layer": "memory",             # event | memory | knowledge | identity
    "type": "semantic",            # episodic | semantic | procedural
    "importance": 0.82,            # how much this should shape behavior
    "confidence": 0.60,            # how sure we are that it is true
    "created_at": 1_714_000_000,   # for recency and decay
    "last_used": 1_714_500_000,    # bumped each time it is usefully recalled
    "source": ["session_1042"],    # provenance, for audit and revision
}

Memory Types: Episodic, Semantic, Procedural

Cutting across the hierarchy is a second, orthogonal classification borrowed from cognitive science. It answers a different question — not how refined a memory is, but what kind of thing it represents. Three types matter for agents, and conflating them is a frequent source of poor recall, because each is stored, retrieved, and applied differently.

Episodic Memory

Episodic memory records specific events situated in time: "on the 3rd of June, the user asked to reschedule the launch to Q4." It is autobiographical and time-stamped. Episodic memory is what lets an agent answer "what did we decide last week" or reconstruct the history of a decision. It tends to be high in volume and to decay fastest, because most individual episodes stop being relevant once their consequences have been absorbed into more general knowledge.

Semantic Memory

Semantic memory holds facts stripped of their episode: "the user's company operates in the EU," "metric units are preferred." It is timeless and generalized. Semantic memory is distilled from episodic memory — you observe a pattern across several episodes and promote it to a standing fact. It is what you most often want to inject into context, because it is compact and broadly applicable, and it is the natural residue of consolidation.

Procedural Memory

Procedural memory encodes how to do things: the sequence of steps that reliably deploys the service, the tool-call pattern that answers a class of question, the recovery routine for a known failure. It is the memory of skill rather than of fact. Procedural memory is the most valuable and the most underbuilt in most agent systems, because it is the layer that lets an agent get better at its job rather than merely more informed about it. A mature agent should be accumulating procedures — reusable, tested sequences — the way an experienced engineer accumulates habits.

A practical rule: tag every memory with its type, and let type participate in retrieval. When the agent is mid-task, weight procedural and semantic memory; when it is reconstructing history, weight episodic. Retrieval that ignores type will happily return a two-month-old episode when what the agent needed was the standing preference distilled from it.

Encoding and Retrieval

A memory system is only as good as its ability to surface the right memory at the right moment. That splits into two problems: encoding — how experience is turned into something searchable — and retrieval — how the relevant few are found among the many. Most teams get the first mostly right and the second badly wrong.

Encoding: Turning Experience Into Vectors

The standard encoding is the embedding: a distilled memory's text is passed through an embedding model that maps it to a point in a high-dimensional vector space, where semantic similarity corresponds to geometric proximity. "The user prefers metric" and "customer wants SI units" land close together despite sharing no words. Embeddings are what let a memory be retrieved by meaning rather than by exact keyword, and a vector index over them supports fast approximate nearest-neighbor search across millions of records.

But the embedding is not the whole record. As the schema above showed, a memory carries metadata alongside its vector: importance, confidence, timestamps, type, provenance. The embedding answers "what is this about." The metadata answers "how much should I trust it, how much does it matter, and is it still fresh." Retrieval that uses only the vector throws away most of what the system knows.

Why Cosine Similarity Alone Fails

Cosine similarity between the query vector and each memory vector is the workhorse of retrieval, and it is necessary but not sufficient. Ranking purely by similarity produces well-known pathologies:

Multi-Factor Recall

The fix is to treat similarity as one term in a composite score, not the whole score. A robust recall function blends several signals: semantic similarity, importance, confidence, and recency (with decay applied to age). Each memory's final rank is a weighted combination, and the weights themselves can shift with context — a task in flight weights procedural memory and confidence; a historical query weights recency and episodic type.

import math

def recall_score(query_vec, mem, now, w, half_life_days):
    # 1. Similarity — where the memory sits in meaning-space.
    similarity = cosine(query_vec, mem["embedding"])

    # 2. Recency via decay — old, unused memories fade unless reinforced.
    age_days = (now - mem["last_used"]) / 86_400
    recency = 0.5 ** (age_days / half_life_days)

    # 3 & 4. Importance and confidence — worth acting on, and likely true.
    importance = mem["importance"]
    confidence = mem["confidence"]

    return (
        w["sim"]        * similarity +
        w["recency"]    * recency +
        w["importance"] * importance +
        w["confidence"] * confidence
    )

Elephantasm formalizes this as four-factor recall: retrieval ranks candidates on similarity plus importance plus confidence plus recency, with decay applied so that unused memories lose pull over time while reinforced ones hold their ground. The specific weights and decay curves are tuning parameters, not universal constants — the point is the shape of the function, not any particular number. What matters is that the system asks four questions of every candidate, not one: is this relevant, does it matter, is it true, and is it still current.

Two additional refinements pay for themselves quickly. First, diversify: after scoring, apply a maximal-marginal-relevance pass so the selected set is not ten phrasings of one fact. Second, respect the token budget: retrieval must return a bounded set that fits the window with room for the task, which means ranking is only half the job — selection under a budget is the other half.

Consolidation and Forgetting

A memory system that only ever writes will fail. Left to accumulate, it fills with duplicates, contradictions, and dead episodes; retrieval quality falls as the haystack grows; storage and search costs climb. A healthy system therefore does two things continuously that have no counterpart in a naive store: it consolidates, and it forgets. Both are active processes, and both are best run in the background, away from the latency-critical path of answering the user.

Forgetting as a Feature

Forgetting is not a defect to be minimized; it is the mechanism that keeps memory useful. The goal is not to retain everything but to retain what matters and let the rest fade. The standard tool is decay: each memory carries a strength that declines over time unless reinforced. Decay is typically exponential — a memory's pull halves over some half-life — so that recent and repeatedly useful memories dominate retrieval while one-off trivia sinks quietly out of reach.

Crucially, decay should not fall uniformly. Importance and layer modulate the rate: a low-importance episode decays fast, a consolidated piece of knowledge decays slowly, and identity is nearly immune. And decay should be reinforced by use — every time a memory is usefully recalled, its strength is refreshed. This is the spacing effect in action: memories that keep proving relevant survive; memories that never resurface disappear. Forgetting, done this way, is a form of automatic, self-tuning relevance.

# Strength under decay, reinforced by use.
# strength -> 0 lets a memory be pruned; recall bumps last_used.
def strength(mem, now, half_life_days):
    age_days = (now - mem["last_used"]) / 86_400
    base = 0.5 ** (age_days / half_life_days)
    # importance and layer slow the fade
    return base * (0.5 + 0.5 * mem["importance"])

def on_recall(mem, now):
    mem["last_used"] = now          # reinforcement resets the clock
    return mem

Consolidation: Curating Memory Offline

Consolidation is the complement to forgetting: instead of letting weak memories fade, it actively improves strong ones. It is the process that takes the day's raw events and messy memories and does the work of a good archivist — merging duplicates, resolving contradictions in favor of the more recent or more confident, promoting recurring observations into stable knowledge, and pruning what has decayed below usefulness. This is expensive, reflective work, and it is exactly the kind of thing you do not want to run while a user is waiting for an answer.

The natural pattern is sleep-inspired: run consolidation as a scheduled background pass, the way biological memory is reorganized during sleep. Elephantasm names this component the Dreamer and splits it into two intensities. Light Sleep is a frequent, shallow pass — deduplicating, tidying, updating strengths, catching obvious contradictions — cheap enough to run often. Deep Sleep is an infrequent, thorough pass — clustering related memories, distilling them into higher-layer knowledge, re-evaluating importance across the whole store, and performing the larger reorganizations that are too costly to do continuously. The division mirrors the tradeoff every consolidation system faces: shallow-and-often to keep the store tidy, deep-and-rare to keep it coherent.

Spaced Repetition and Reinforcement

The last piece ties decay and consolidation together. Borrowed from how humans study, spaced repetition says that a memory's durability should increase each time it is successfully retrieved, and that the interval before it needs reinforcing should lengthen accordingly. In an agent, this falls out naturally: recall bumps a memory's strength and its last-used timestamp; consolidation notices which memories are repeatedly useful and promotes them; the useful get more durable and the unused fade. The result is a store that tunes its own retention to actual usage patterns, without anyone hand-labeling what to keep.

Practical Architecture Patterns

The concepts assemble into a small number of patterns that recur across well-built memory systems. None is exotic; the discipline is in applying them consistently rather than in any single clever trick.

Separate the Write Path from the Read Path

The read path runs on every turn and is latency-critical: embed the query, retrieve candidates, score, select under budget, inject. The write path runs after the turn and can be asynchronous: log events, distill significant ones into memories, embed and index them. Keeping these separate means the user never waits for consolidation, and the expensive reflective work — distillation, deduplication, promotion — happens off the critical path. Consolidation (the Dreamer pattern above) is the write path taken to its slowest, most deliberate extreme.

# Read path — bounded, fast, runs every turn.
def build_context(query, store, budget_tokens, w, half_life):
    q = embed(query)
    candidates = store.similarity_search(q, k=50)   # cheap wide recall
    ranked = sorted(
        candidates,
        key=lambda m: recall_score(q, m, now(), w, half_life),
        reverse=True,
    )
    selected, used = [], 0
    for m in diversify(ranked):
        cost = token_len(m["text"])
        if used + cost > budget_tokens:
            break
        selected.append(on_recall(m, now()))   # reinforce on use
        used += cost
    return format_memories(selected)

Retrieve Wide, Rank Hard, Inject Narrow

Do not try to retrieve exactly the right memories in one shot. Retrieve a generous candidate set cheaply by similarity, then apply the expensive multi-factor scoring only to that shortlist, then inject only the handful that survive ranking and fit the budget. This two-stage funnel — cheap wide recall followed by precise re-ranking — is faster and more accurate than a single-stage query, and it is where importance, confidence, and recency earn their keep.

Namespace and Scope Everything

Memory must be partitioned. A memory learned about one user must never surface for another; a fact scoped to one project must not leak into an unrelated task. Every record should carry the scope it belongs to — user, project, agent, session — and retrieval should filter on scope before ranking. Cross-scope leakage is not merely a quality problem; it is a privacy and security problem, and it is the failure mode most likely to end up in an incident report.

Keep Provenance and Make Memory Revisable

Every distilled memory should remember where it came from — which events or sessions produced it. Provenance is what makes memory auditable ("why does the agent believe this?") and revisable ("this was inferred from a misread message; correct it"). A memory system without provenance is a system whose mistakes are permanent and untraceable. Treat memories as claims with sources, not as immutable facts, and give consolidation the ability to revise or supersede them.

Failure Modes

Memory systems fail in characteristic ways. Knowing the shapes in advance is most of the defense, because each has a clear structural cause and a clear countermeasure.

Memory Poisoning

A false or malicious statement gets written as a durable memory and then influences behavior indefinitely. Because memory is trusted more than fresh input, a poisoned memory can be more dangerous than a bad prompt. The defenses are confidence scoring (do not let a single low-confidence observation become settled knowledge), provenance (so a bad memory can be traced and removed), and consolidation reconciliation (so later, better evidence can supersede an earlier error).

Retrieval Dilution and Context Flooding

Inject too many memories and you recreate the very problem persistent memory was meant to solve: a bloated, low-density window where the critical fact is lost among the merely relevant. Strict token budgets, hard ranking, and diversification are the guardrails. More retrieved memory is not more helpful memory; past a small number, it is actively harmful.

Stale Memory and Unresolved Contradiction

Without decay and reconciliation, a store accumulates superseded facts that continue to surface with full confidence — the agent insists on a preference the user reversed weeks ago. Recency-weighted retrieval and consolidation that marks memories superseded rather than co-equal are what keep the present from being outvoted by the past.

Over-Consolidation and Lost Nuance

The opposite failure: consolidation that is too aggressive flattens distinct memories into a single averaged generalization, erasing the exceptions and edge cases that mattered. Good consolidation preserves salient specifics — it promotes the general rule while keeping the notable exception, rather than smoothing everything into a bland mean. Forgetting should remove the trivial, not the distinctive.

  1. Never treat a memory as ground truth — attach confidence and provenance, and let both drive retrieval and revision.
  2. Score recall on more than similarity — blend importance, confidence, and recency, and enforce a token budget.
  3. Run consolidation and forgetting continuously in the background, not on the user's critical path.
  4. Scope and namespace every memory, and treat cross-scope leakage as a security failure, not a quality one.
  5. Make memory revisable — superseding a wrong belief should be a routine operation, not a database surgery.

Principles for Memory Systems

Strip away the specifics and a handful of durable principles remain. They hold whether you build memory from scratch or adopt a framework like Elephantasm, because they describe the problem rather than any one solution.

Memory is a layer, not a model feature. The model supplies reasoning over what is in front of it; the memory system decides what that is. Continuity is engineered around the model, never expected from it.

The hierarchy earns its keep. Raw events, distilled memories, consolidated knowledge, and stable identity have different volumes, different lifespans, and different retention policies. Flattening them into one undifferentiated store forfeits every one of those distinctions.

Retrieval is a scoring problem, not a lookup. Similarity finds what is relevant; importance, confidence, and recency decide what is worth surfacing. The composite ranking is where a memory system is won or lost.

Forgetting is a feature. Decay, reinforcement, and consolidation are not housekeeping bolted on at the end — they are what keep a memory system accurate and affordable as it grows. A store that cannot forget cannot scale.

An agent with a large context window and no memory is a brilliant amnesiac — capable in the moment, incapable of growth. Memory is what converts a sequence of impressive but disconnected performances into something that accumulates: an agent that knows you, remembers what it learned, gets better at its work, and remains recognizably itself across the long arc of use. That continuity is not a feature of the model. It is the system you build around it.