Memory and Context Management

Every agent conversation starts with an empty context window. As the loop runs, the message list grows until it approaches the model's token limit. Long-running agents and multi-session tasks need memory strategies that preserve relevant information while keeping the active context manageable.

Four types of memory

• In-context memory: the raw message history — simple but limited to the context window
• Episodic memory: summaries of past interactions stored externally and retrieved when relevant
• Semantic memory: a vector store of facts, documents, and past results the agent can search
• Procedural memory: learned patterns baked into the system prompt or fine-tuned into the model

Summarization for long contexts

When the message history grows too long, compress it. The simplest approach is a rolling summary: after every N turns, ask the model to write a concise summary of what has happened so far, then replace the earlier messages with the summary. The summary replaces completed work; recent messages stay verbatim for continuity.

def maybe_summarize(messages: list, max_tokens: int = 6000) -> list:
    total = estimate_tokens(messages)
    if total < max_tokens:
        return messages
    # Summarize everything except the system prompt and last 4 messages
    system = messages[0]
    recent = messages[-4:]
    to_summarize = messages[1:-4]
    summary_prompt = [
        system,
        *to_summarize,
        {"role": "user", "content": "Summarize the conversation above in under 200 words."},
    ]
    resp = client.chat.completions.create(model="gpt-4o-mini", messages=summary_prompt)
    summary = resp.choices[0].message.content
    return [system, {"role": "assistant", "content": f"[Summary so far]: {summary}"}, *recent]

External memory with vector search

For semantic memory across sessions, embed facts and observations into a vector store at the end of each session, then retrieve relevant entries at the start of the next one. Retrieval grounds the agent in prior context without inflating the prompt with the full history.

What to store vs. what to discard

Not everything deserves to be remembered. Store task outcomes, user preferences, decisions made, and error patterns. Discard low-signal exchanges like greetings, clarification loops that resolved, and raw tool outputs once their conclusions are captured in a summary. Treating memory as a database — with explicit writes — is more reliable than passive accumulation.