Agent memory
How an agent keeps information beyond one context window: working memory in context, plus long-term stores it writes to and retrieves from.
A language model remembers nothing between calls; it only sees what's in its Context window right now. Agent memory is how an AI agent gets a working memory for the current task and a long-term memory across tasks.
Long-term memory is usually a store the agent saves notes to and searches later — "this customer prefers email", "the staging DB is stg-2". Deciding what to save, and what to forget, is the hard part.
A detective's notebook: what's in their head fades, but the notes are there next week — as long as they wrote down the right things and crossed out the wrong ones.
Saving everything and retrieving by similarity alone. The store fills with stale, contradictory notes and the agent acts on outdated facts with full confidence.
Working memory is the context: messages, tool results, a scratchpad or todo list, managed by Context engineering (trim, compact, summarise). Long-term memory is external: files, a key-value store, SQL, or a vector index searched with Semantic search vs similarity search. Categories: episodic (what happened in past runs), semantic (facts about the user or domain), procedural (learned instructions). MemGPT (Packer et al., 2023) framed this as OS-style paging between context and external storage; Generative Agents (Park et al., 2023) ranked memories by recency, importance and relevance.
Write policy matters more than storage. Memories should be specific, attributed (who/what source), timestamped, scoped to a tenant and user, and correctable, because stale or wrong memories are retrieved with the same confidence as good ones. Memory is also an attack surface: a Prompt injection (indirect) can plant an instruction that persists across sessions. And it is a privacy store: personal data needs PII redaction, TTL & cache invalidation, Multi-tenancy isolation and real deletion for GDPR/DPDP erasure requests.
Per-tenant, per-user facts table (text, source, created_at, ttl, confidence) loaded top-10 by relevance × recency at session start; explicit remember/forget tools; writes from tool output require provenance; no instructions stored as memory; deletion API wired to account erasure.
Customer-support agent memory, per account: {fact: "Prefers contact by email, not phone", source: "ticket #88213", written: "2026-09-02", ttl: "365d"}. On a new ticket the agent loads the 10 highest-scoring facts (relevance × recency) — ≈ 600 tokens — instead of 40 past transcripts (≈ 60k). When the customer says "actually, call me", the agent updates the fact rather than appending a contradicting one.
Recommend the simplest store that fits: a per-user facts table with timestamps beats a vector DB until you have thousands of memories per user. Spend the answer on write policy, retrieval scoring, staleness and deletion.
Expect the follow-up "how do you stop memory poisoning?" — write only from trusted channels or with confirmation, store provenance, and never store instructions as facts.
Numbers worth knowing
- Memory loaded per session
- 300–1,000 tokenstop 5–15 facts; vs 10k+ for raw past transcripts
- Default fact TTL
- 90–365 daysre-confirm or expire; shorter for volatile facts
How to decide
| When | Do this |
|---|---|
| Under ~1,000 memories per user | SQL/KV table with recency and keyword filters; no vector index needed. |
| Large episodic archives (many past runs) | Vector or hybrid search with recency weighting. |
| Memory may contain personal or regulated data | TTL, redaction and a tested erasure path before launch. |
Cost maths
Loaded memory adds tokens to every call: cost = memory_tokens × calls × p_in. 800 tokens × 15 calls/session × 200k sessions/month = 2.4B tokens; at $3 / M (illustrative, Oct 2026) ≈ $7.2k/month, ≈ $0.9k if the memory block sits in the cached prefix. Replaying 10k tokens of past transcripts instead would be ≈ $90k.
Operating it
- Memories per user exceed ~1,000
- Contradicting facts for the same key in >5% of users
- Users correcting the agent about remembered facts in >2% of sessions
- Memory writes per session and share from tool output (alert on spikes)
- Contradicting-fact rate per key (alert > 5%)
- Erasure requests completed within SLA (alert on any miss)
- Memories containing imperative instructions flagged by a classifier (alert on any)
When it goes wrong
In Sept 2024 security researcher Johann Rehberger showed a ChatGPT macOS memory being poisoned so that future conversations were silently exfiltrated ("SpAIware").
Public research disclosure with a working proof of concept.
Content from an untrusted web page or document could trigger a memory write containing instructions, which then persisted into every later session.
OpenAI patched the exfiltration channel; the design lesson is that memory writes from untrusted content need provenance, user visibility and confirmation.
Related terms
Sources
- MemGPT: Towards LLMs as Operating Systems — Packer et al., 2023paper
- Generative Agents: Interactive Simulacra of Human Behavior — Park et al., 2023paper
- Spyware Injection Into Your ChatGPT's Long-Term Memory (SpAIware) — Johann Rehberger, 2024blog
Check yourself
Three questions. Every option is explained once you answer.