← All chapters
Memory, Context & State
Short- and long-term memory, context windows, and state management for agents.
20 terms
CheckpointingCheckpointing is the practice of periodically saving an agent's full state — task progress, accumulated memory, tool outputs, and decision trace — to durable storage so the agent can resume from that point if interrupted. It enables fault-tolerant long-running tasks, supports human review mid-task, and allows rollback to a known-good state when an agent takes an undesirable action. Checkpoints are especially important for tasks spanning many minutes or hours.Context CompressionContext compression reduces the token footprint of information that must stay in the prompt without losing its essential meaning. Techniques include extractive summarization of prior turns, LLM-driven distillation into bullet points, lossy removal of irrelevant exchanges, and selective retention of salient facts. Compression trades fidelity for capacity, allowing agents to maintain longer effective memories within a fixed context window budget.Context StuffingContext stuffing is the practice of inserting large volumes of potentially relevant text into the prompt — entire documents, extensive histories, or many retrieved passages — in hopes that the model will find what it needs. It is a blunt alternative to targeted retrieval. While sometimes effective, context stuffing inflates cost, increases latency, and can degrade model focus when the stuffed content is noisy or internally inconsistent.Context WindowThe context window is the maximum number of tokens a language model can process in a single forward pass, spanning the system prompt, conversation history, retrieved documents, tool outputs, and the model's own generated text. Everything outside this limit is invisible to the model. Larger windows enable longer reasoning traces and richer context but increase compute cost and can hurt retrieval quality as content grows.Conversation HistoryConversation history is the ordered sequence of prior turns — user messages and assistant responses — prepended to the current prompt so the model can refer to what was said before. It is the most straightforward form of short-term memory in chat-based systems. As histories grow, they consume more tokens, eventually requiring compression, summarization, or selective truncation to stay within the context window limit.Episodic MemoryEpisodic memory stores records of specific past events or interactions — such as a conversation that happened last Tuesday or a task the agent completed in a prior session. Drawn from cognitive science, the term distinguishes event-based recall from general factual knowledge. In agent systems, episodic memory lets a model say 'the last time you asked about X, we concluded Y,' grounding responses in concrete history.Long-Term MemoryLong-term memory is information persisted outside the active context window — in a vector store, relational database, or key-value cache — so that an agent can recall facts and past interactions across many sessions. Retrieval requires an explicit lookup step. Long-term memory enables continuity across conversations but introduces latency, retrieval errors, and consistency management challenges.Lost in the MiddleLost in the middle is a documented failure mode where language models recall information placed at the beginning or end of a long context more reliably than information placed in the middle. When many documents are inserted into a prompt, facts buried in the center are disproportionately ignored. Mitigation strategies include reranking retrieved passages to place the most critical content near the start or end of the context.Memory EvictionMemory eviction is the process of removing entries from a context window or memory store to free capacity for new information. In-context eviction happens automatically when older tokens fall outside the model's window; deliberate eviction strategies include least-recently-used pruning, relevance scoring, and time-decay policies. Well-designed eviction removes low-value facts while retaining the memories most critical to ongoing tasks.Memory RetrievalMemory retrieval is the process by which an agent queries its external memory store to surface facts or prior events relevant to the current task. A query — often derived from the current message or the agent's current goal — is matched against stored entries using semantic search, keyword lookup, or hybrid methods. Retrieved items are then injected into the context window for the model to use in its response.Memory StoreA memory store is any external system — vector database, relational store, document store, or key-value cache — used to persist information an agent needs across sessions or between context resets. The agent queries the store at retrieval time and writes new facts after meaningful events. Design choices around storage format, indexing strategy, and eviction policy directly affect recall accuracy and latency.ScratchpadA scratchpad is a designated region of the prompt — often marked by a special tag or turn role — where an agent writes intermediate reasoning, partial results, and notes before producing a final answer. Unlike the final response, scratchpad content is meant for the model's own use and is typically not shown to the end user. It mirrors the human practice of working through a problem on scratch paper.Semantic MemorySemantic memory holds general facts, concepts, and domain knowledge that are not tied to any single event. In an AI agent, semantic memory can be encoded in model weights through pre-training or stored externally in a vector database and retrieved on demand. Unlike episodic memory, semantic memory is about what is known, not when it was learned, making it useful for answering factual or procedural questions.Short-Term MemoryShort-term memory in an AI agent refers to information held within the active context window during a single session. It includes the conversation so far, any tool outputs, and the model's intermediate reasoning. Because it lives entirely in the prompt, short-term memory is fast to access but evaporates when the session ends or when older content is evicted to make room for new tokens.Sliding WindowThe sliding window strategy retains only the most recent N tokens or turns of a conversation, discarding older content as new content arrives. It is the simplest form of context management: fast, predictable, and free of any summarization cost. The main drawback is that information outside the window is permanently forgotten, making it unsuitable for tasks that require referencing events from early in a long session.State ManagementState management in an AI agent is the discipline of deciding what information to carry forward across loop iterations, how to serialize it into the prompt, and how to update it as the task progresses. Poorly managed state leads to context bloat, stale facts, or the agent losing track of its own prior decisions. Effective state management distinguishes ephemeral scratch work from durable task-relevant facts that must persist.Summarization MemorySummarization memory is a pattern where an agent periodically compresses older parts of the conversation into a running summary, then replaces the raw history with that summary in the prompt. This keeps the context window from overflowing while preserving a high-level record of prior exchanges. The quality of recall depends on summary fidelity; key details that fall outside the summary are permanently lost to the model.Token BudgetA token budget is an explicit allocation of tokens reserved for each component of a prompt — system instructions, retrieved documents, conversation history, the user message, and the model's output. By planning how many tokens each section may use, engineers prevent any single component from crowding out the rest. Managing a token budget is essential when operating near the context window limit or when controlling API cost.Vector MemoryVector memory is an external memory system that stores information as dense embedding vectors in a vector database. Retrieval is performed by encoding the query into the same embedding space and finding the nearest neighbors by cosine or dot-product similarity. Vector memory supports semantic recall — finding conceptually related facts even when exact words do not match — and scales to millions of stored facts efficiently.Working MemoryWorking memory refers to the subset of context the agent actively uses while reasoning through a specific step — typically the scratchpad, the most recent tool output, and the immediate task description. Drawing from cognitive science's model of human working memory, the concept highlights that even within a large context window, models attend more strongly to nearby tokens, making deliberate placement of critical information essential.