How do you handle memory in an agent, and what is context engineering?
- 2 min — short-term vs long-term memory; MemGPT framing
- 3 min — context engineering moves: pin, stub, summarise, retrieve
- 3 min — long-horizon example with token and success numbers
- 2 min — memory as an injection and privacy surface
- 2 min — ownership: store, retention, legal sign-off
Short-term memory is the working context: transcript, tool results, scratch notes for this task. Long-term memory persists across sessions — preferences, learned facts, past outcomes — stored outside the model (rows, files, or embeddings with metadata) and retrieved when relevant; MemGPT (Packer et al., 2023) framed this as paging between a small main context and external storage. Context engineering is deciding what occupies the window at each step: instructions, the tools needed now, retrieved facts, compressed history — and what stays out.
More context is not free or better. Every token is re-sent on every turn, and models use the middle of long contexts worse than the ends (lost in the middle, Liu et al. 2023). Keep the goal and constraints at the top, the latest state at the bottom, and compact the middle.
Long-running coding agent on a 3-hour refactor (~180 steps). Without management, raw tool output reaches 150k tokens by step 60. With context engineering: tool results older than 5 turns are replaced by one-line stubs (“grep: 14 hits, see notes”), completed sub-tasks are summarised into a NOTES.md scratchpad the agent maintains, and the system prompt pins the goal and 6 hard constraints. Context stays at 25–40k tokens; on the 30-task long-horizon eval, success rose from 47% to 70%. Long-term: a customer-support agent stores “ships to a freight forwarder in Rotterdam” as a memory with source ticket id, timestamp and 180-day TTL, retrieved only for shipping intents.
Treat memory writes as a data pipeline: provenance (which run, which source), expiry, deletion on user request (GDPR Art. 17, DPDP Act), and strict tenant scoping — a memory is a persistent injection vector, because a malicious instruction written once is replayed into every future session. Write memories only from trusted turns (the user's own statements, verified tool data), never from retrieved web pages or inbound emails. Measure: task success vs context size, memory precision (retrieved and actually used), and stale-memory incidents.
Org scope. Long-term memory is a data store with a privacy owner, not a feature flag: the platform team owns the store, retention and deletion APIs; the privacy/legal function signs off on what categories may be remembered per region; product teams decide what to write. Build vs buy: provider memory features are fine for single-user assistants; multi-tenant products need memory in your own database so deletion, residency and audit are provable. Failure mode: a compaction summary silently drops the one constraint that mattered (“don't touch the billing schema”), and step 140 violates it.
What the answer sounds like at each level
Short-term memory is the conversation; long-term memory is stored in a database and retrieved when needed.
Actively manages the window — stubs old tool output, maintains a scratchpad, pins constraints — and measures success vs context size; scopes and expires long-term memory.
Runs long-term memory as governed data with provenance, TTL, deletion and residency, restricts writes to trusted sources, and owns the build-vs-buy call for multi-tenant memory.
Follow-ups the interviewer will push
1How do you stop compaction from dropping a critical constraint?
Never compact the constraints: pin them in the system prompt or a protected block, and have the compaction prompt preserve any line tagged as a constraint. Add an eval task whose failure is exactly “constraint forgotten after compaction”.
2Now a regulated tenant in the EU asks you to erase everything about a user. What must work?
Memories keyed by user and tenant with source ids, a delete API that removes rows and vectors and invalidates caches, deletion propagated to traces and backups within the stated window, and an audit record that it happened.
3When is a 1M-token window a substitute for memory?
Within one session for read-heavy analysis, maybe. Across sessions, no: you would pay to re-send everything every turn, and retrieval of the relevant 2k tokens beats burying them in 1M.
Red flags interviewers listen for
- “Just use a bigger context window” as the memory strategy.
- Writes long-term memories from untrusted content such as web pages or inbound email.
- No deletion, TTL or tenant scoping for stored memories.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Window management | Concrete compaction tactics with measured context size and success. |
| Memory safety | Trusted-source writes, provenance, TTL, tenant scoping. |
| Privacy and region | Deletion and residency across the regions the product serves. |
| Org scope | Clear owners for the store, retention policy and write decisions. |
Sources
- MemGPT: Towards LLMs as Operating Systems — Packer et al., 2023paper
- Lost in the Middle: How Language Models Use Long Contexts — Liu et al., 2023paper