LLM Agent Memory System Design: From Short-Term Memory to Long-Term Memory

One of an agent's biggest limitations is memory. When the context window fills up, early conversation is forgotten; when a session ends, everything disappears. To build a practical agent, you have to design memory explicitly.

Four Types of Memory

Type	Characteristics	Implementation
Short-term memory	Conversation history within the current session	messages array
Summary memory	Compress long conversations into summaries	Summarization + LLM
Semantic memory	Store facts and knowledge	Vector DB
Procedural memory	Remember how to perform tasks	Store few-shot examples

Short-Term Memory: Sliding Window

Python

from collections import deque
from openai import OpenAI

class ShortTermMemory:
    def __init__(self, max_messages=20):
        self.messages = deque(maxlen=max_messages)
 self.system_prompt = " AI when."

    def add(self, role, content):
        self.messages.append({"role": role, "content": content})

    def get_context(self):
        return [{"role": "system", "content": self.system_prompt}] + list(self.messages)

client = OpenAI()
memory = ShortTermMemory(max_messages=10)

def chat(user_input):
    memory.add("user", user_input)
    response = client.chat.completions.create(
        model="gpt-4o", messages=memory.get_context()
    )
    reply = response.choices[0].message.content
    memory.add("assistant", reply)
    return reply

The problem: A sliding window deletes old information that still matters. A username mentioned in the 10th message is forgotten by the 30th.

Summary Memory: Expanding Context Through Compression

When a conversation grows long, summarize the older portion and keep that instead.

Python

import anthropic

client = anthropic.Anthropic()

class SummaryMemory:
    def __init__(self, window_size=10, summary_threshold=8):
        self.recent_messages = []
        self.summary = ""
        self.window_size = window_size
        self.summary_threshold = summary_threshold

    def _summarize(self):
        to_summarize = self.recent_messages[:self.summary_threshold]
 existing = f" : {self.summary}\n\n" if self.summary else ""
        conversation = "\n".join(f"{m['role']}: {m['content']}" for m in to_summarize)

        response = client.messages.create(
            model="claude-haiku-4-5-20251001",
            max_tokens=300,
            messages=[{
                "role": "user",
 "content": f"{existing} 3-4 :\n\n{conversation}"
            }]
        )
        return response.content[0].text

    def add(self, role, content):
        self.recent_messages.append({"role": role, "content": content})
        if len(self.recent_messages) > self.window_size:
            self.summary = self._summarize()
            self.recent_messages = self.recent_messages[self.summary_threshold:]

    def get_context(self):
        messages = []
        if self.summary:
            messages += [
 {"role": "user", "content": f"[ ]\n{self.summary}"},
 {"role": "assistant", "content": " ."}
            ]
        return messages + self.recent_messages

Long-Term Memory: Vector DB

To persist information across sessions, you need an external store.

Python

from openai import OpenAI
import json
from datetime import datetime

client = OpenAI()

class LongTermMemory:
    def __init__(self, user_id, supabase_client):
        self.user_id = user_id
        self.sb = supabase_client

    def _embed(self, text):
        response = client.embeddings.create(
            model="text-embedding-3-small", input=text
        )
        return response.data[0].embedding

    def save(self, content, memory_type="fact"):
        embedding = self._embed(content)
        self.sb.table("agent_memories").insert({
            "user_id": self.user_id,
            "content": content,
            "memory_type": memory_type,
            "embedding": embedding,
            "created_at": datetime.utcnow().isoformat()
        }).execute()

    def recall(self, query, top_k=3):
        query_embedding = self._embed(query)
        result = self.sb.rpc("match_memories", {
            "query_embedding": query_embedding,
            "user_id": self.user_id,
            "match_count": top_k
        }).execute()
        return [r["content"] for r in (result.data or [])]

Integrating Memory Layers

Python

class AgentWithMemory:
    def __init__(self, user_id, supabase_client):
        self.short_term = SummaryMemory(window_size=10)
        self.long_term = LongTermMemory(user_id=user_id, supabase_client=supabase_client)
        self.client = OpenAI()

    def chat(self, user_input):
 # 1.
        relevant_memories = self.long_term.recall(user_input)

 system_content = " AI when."
        if relevant_memories:
 system_content += "\n\n[ ]\n" + "\n".join(f"- {m}" for m in relevant_memories)

 # 2. +
        messages = [{"role": "system", "content": system_content}]
        messages.extend(self.short_term.get_context())
        messages.append({"role": "user", "content": user_input})

        response = self.client.chat.completions.create(model="gpt-4o", messages=messages)
        reply = response.choices[0].message.content

        self.short_term.add("user", user_input)
        self.short_term.add("assistant", reply)

 # 3. save
 self.long_term.save(f"User: {user_input}\nwhen: {reply}")

        return reply

What to Store

Do store: User preferences, project names and goals, recurring patterns, explicit requests ("remember this")
Don't store: One-off questions, general facts (search can replace these), error messages

In the next post, we'll cover evaluation and debugging for the agent you've built.

Code verification scope (2026-09-30)

Python code block syntax was checked with ast.parse on Python 3.14.6. External SDK calls, service connections, and performance claims were outside this check. Verify dependencies and model availability in the deployment environment.

Official references: OpenAI model reference, Anthropic API documentation.

#LLM #Agents #Architecture #LangChain

✦ ✦ ✦

Sources and verification

Sources and checks vary by article. Only articles with a recorded check date, scope and sources below were checked, and only within that scope. For other articles, verify against the current docs for your version before applying anything, and email us about errors. Editorial policy · email us

This article has no recorded human check (date and scope) yet.

Published · May 20, 2026

Topic-related official docs (auto-linked · not per-claim sources)LangChain documentation ↗pgvector repository ↗

How we write and review →Editorial policy →

Was this helpful?

Comments

Be the first to comment.