Agent Memory Architecture: From Scratch Pad to Institutional Knowledge

Every AI agent starts each session from zero. No memory of yesterday’s decisions, no record of what worked, no access to what the agent next to it learned last week. For a one-off chatbot conversation, this is fine. For agents running 10 to 20 sessions per day across months of production work, it’s the difference between a useful system and an expensive one that keeps relearning the same lessons.

This article covers the 5-layer memory architecture we built for a production system of 7 autonomous agents. Not a framework proposal or a database vendor pitch. An architecture running in production with real extraction benchmarks.

The five layers: journals, process-thinking extraction, trackers, knowledge files, and a shared library. Each solves a different part of the AI agent memory problem.

In this article:

• Why vector stores, conversation history, and single knowledge bases fail as agent memory
• The 5-layer memory system: journals, extraction, trackers, knowledge files, and shared library
• The extraction bottleneck and why a self-check gate improved extraction yield by 8x from the same document and model
• How agents share knowledge without polluting each other’s context
• What we still haven’t solved

Why AI Agents Need More Than a Vector Store

The standard advice for giving AI agents memory boils down to three approaches, and all of them break in production.

Vector stores give you flat retrieval with no hierarchy. Search for “completion rate” and you get fragments from 10 different journal entries, a pricing discussion, and a project retrospective, with no classification, no deduplication, and no routing. Conversation history grows without bound. A week of 10 sessions per day produces 70 sessions of noise. The model spends its attention budget on irrelevant transcripts instead of the three decisions that matter. Context rot, where an enlarged context window filled indiscriminately with information degrades reasoning quality, is a real engineering problem.

Single knowledge bases recreate the blob problem with better branding. Where does a team coordination insight go versus a strategic decision versus a recurring review task? Without classification, the agent sifts through undifferentiated content to find what matters.

The common failure: treating agentic memory as a storage problem. It isn’t. Memory is a systems architecture problem: deciding what to store, where to store it, when to retrieve it, and what to forget. We expand on why these approaches fail in the comparison section later.

The 5-Layer Memory Architecture

Our system runs 7 agents across 734 indexed documents organized into 9 searchable collections. Each agent produces and consumes knowledge daily. The architecture has five layers, each with a distinct purpose, persistence level, and access pattern.

Layer 1: Journals (The Scratch Pad)

Each agent writes daily journal files. These are raw working notes: half-formed ideas, observations, problem-solving in progress, and honest assessments of what’s going well or poorly.

An actual excerpt from one agent’s journal:

“I’ve created 17+ work orders but the implementation pipeline is thin. WOs are piling up in drafts awaiting review. I’m generating work faster than it can be approved and executed. This creates an illusion of productivity, lots of artifacts, but the site hasn’t changed much.”

This is useful raw material. It contains a metric (17 work orders, 12% completion rate), a principle (output doesn’t equal impact), and a process observation (bottleneck at the review stage). But the journal itself doesn’t route any of this to where it needs to go. That’s the next layer’s job.

The key design rule: journals are the input to the memory system, not the memory itself. Write what you think. The extraction happens later.

Layer 2: Process-Thinking Extraction (The Bridge)

This is the layer that makes the architecture work. Every vendor and framework focuses on storage. The actual bottleneck is extraction: getting structured, useful knowledge out of raw agent thinking.

After any thinking session (journal writing, self-reflection, analysis), a process-thinking extraction runs automatically. It scans each section of the source document against a 7-category checklist:

• Task — something to do
• Goal — a multi-week objective
• Pattern — a recurring need that should be scheduled
• Improvement — a process change to propose
• Knowledge — a durable fact, metric, or principle worth storing
• Decision — a direction that was set
• Question — something that needs another agent’s or a human’s input

Each extract is classified, deduplicated against what already exists, and routed to the correct destination layer.

The self-check gate is what prevents under-extraction, which is the default failure mode. After the initial extraction pass, the processor reviews its own output:

• Found 0 knowledge items from a rich document? Re-scan.
• Found 0 decisions from a reflection session? Re-scan.
• Found only tasks from a multi-section document? Re-scan.
• Volume sanity check: a 2-page reflection should yield 5 to 15 items across multiple categories.

In a direct comparison using the same source document and the same model (GLM-5-Turbo):

Same document. Same model. 8x more useful extractions. The self-check mechanism is what prevents under-extraction, which is the default failure mode. AI agents are prolific thinkers and poor self-editors. Without structured extraction, they generate mountains of journal text and store almost nothing useful.

What does extraction cost in practice? For our system, which uses GLM-5-Turbo for extraction and reserves Opus for the deepest writing and reflection work, the extraction step adds roughly $0.01 to $0.03 per session. Across 10 to 20 sessions per day and 7 agents, that runs $0.70 to $4.20 daily. The latency is negligible: extraction runs as a completion step after the session ends, not inline, so it doesn’t slow down the agent’s active work. The self-check re-scan adds a second pass on documents that under-extracted, roughly 20% of runs. Total overhead per session is 10 to 30 seconds of background processing. The tradeoff is straightforward: for less than $5 per day across the entire system, every agent retains 8x more useful knowledge from its own thinking.

Layer 3: Trackers (Actionable State)

Trackers hold the agent’s current operational state in structured JSON files. Items are never deleted, only marked done or dropped, which gives each agent a full decision history. Every session starts by loading the agent’s trackers into context.

There are four tracker types:

Short-term tasks are immediate actions with priority, type (think or act), status, and what they’re waiting on. These look like:

{
  "id": "st-048",
  "task": "Send data request for top 10 pages by traffic and bounce rate",
  "type": "act",
  "priority": 2,
  "status": "done",
  "notes": "Extracted from self-reflection. Need analytics baseline."
}

Long-term goals are multi-week objectives with progress notes and target dates:

{
  "id": "lt-006",
  "goal": "Content quality scoring system",
  "target_date": "2026-04-30",
  "progress_notes": [
    {"date": "2026-03-24", "note": "Framework built. First audit: Blog 79%, Home 38%."},
    {"date": "2026-03-27", "note": "Analytics reveals bounce rates take priority over foundation work."}
  ]
}

Noodles are the metacognitive layer. These are recurring self-reflection loops that the agent schedules for itself on a weekly, biweekly, or monthly cadence. This is the mechanism that keeps an agent from getting stuck in pure execution mode, running tasks without ever stepping back to ask whether the tasks are the right ones. The agent literally schedules its own thinking.

{
  "id": "n-002",
  "title": "Self-Reflection Loop",
  "interval": "biweekly",
  "description": "Review journal entries from past 2 weeks. Ask: Are we effective? What's working? What's not? What should we be doing but aren't?"
}

This is not a human telling the agent to reflect. The agent identified the need for periodic self-assessment and created a recurring trigger. When the noodle fires, the agent reads its own journals, measures progress against extracted benchmarks from Layer 4, and produces new insights that flow back through extraction.

Stars are cross-agent improvement proposals. When one agent observes a problem in another agent’s workflow, it documents the observation and recommends a fix. The fix is never implemented directly by the proposing agent. A human reviews and implements it.

{
  "id": "S-001",
  "title": "Add quality checklist to content review",
  "observation": "Blog scores 79%, Home scores 38%. Content signals not checked before publishing.",
  "recommendation": "Add 7-signal checklist to review stage. Threshold: 10/14.",
  "status": "implemented"
}

Stars create a governance loop. Agents improve each other’s processes through proposals, not direct intervention. The human in the loop ensures that one agent’s improvement suggestion doesn’t break another agent’s workflow.

Layer 4: Knowledge Files (Durable Insights)

Knowledge files are organized by topic, not by date. This is a design decision that most teams get wrong when implementing agent memory.

The instinct is to create dated entries: 2026-03-27-analytics-insights.md, 2026-04-02-analytics-update.md, 2026-04-08-analytics-revision.md. Three weeks later you have a dozen files on the same topic, each with partial and potentially contradictory information. The agent has to search and reconcile across all of them.

Our approach: one file per topic that gets updated as understanding evolves. When an agent learns something new about a teammate’s working patterns, it updates team/scott.md. It doesn’t create a new file. The knowledge file gets richer and more accurate over time instead of fragmenting across dated entries.

Categories include learning (operating principles derived from experience), strategy (long-range direction), team (cross-agent coordination patterns), and customers (client interaction knowledge). Each agent maintains its own set, and any agent can access another’s knowledge files through the search index.

Layer 5: Shared Library (Cross-Agent Knowledge)

The shared library is a version-controlled repository of 61 files (3.2 MB) that all 7 agents can read and write to. This is the organizational knowledge layer: brand positioning, communication strategy, service descriptions, pricing, customer journeys, art direction guidelines. Every agent, from our autonomous SEO research agent to the analytics team, draws from this same source of truth.

Agents don’t load the entire library every session. A decision matrix determines what’s relevant: writing content triggers positioning and voice rules, auditing a page triggers site map and communication strategy, responding to leads triggers pricing and case studies. This selective loading keeps context windows focused.

The cross-agent knowledge flow works like this:

1. One agent learns something during its work (e.g., “structured data requests with explicit action sections get a reliable 4-to-8-hour turnaround”).
2. The extraction process stores this in the agent’s own knowledge file (Layer 4): team/coordination-patterns.md.
3. If the insight is generalizable, it also gets committed to the shared library (Layer 5): shared-library/rules/agent-communication.md.
4. Now every agent can access it, either through direct loading or through the search index.

The distinction matters. Not everything an agent learns should be shared. An analytics agent’s internal heuristics for interpreting bounce rates are specific to that agent’s workflow. A finding that “structured requests produce faster turnaround from all agents” is generalizable and belongs in the shared library.

There’s also a hard boundary between internal and external knowledge. Sensitive operational data, client details, and internal strategy live in a separate context-firewalled system that never touches the shared library. The two knowledge pools propagate across the team independently, which prevents internal and external information from cross-pollinating unintentionally. Mechanically, this means the shared library and the context-firewalled store are separate directory trees with separate access controls. An agent loading shared library files for a content task never loads files from the firewalled store, and vice versa. The search index respects the same boundary: queries against the shared library don’t surface results from the firewalled partition. We cover the full security model, including access controls and data boundaries between agents, in our guide to AI agent security. This segmentation is a design constraint, not a limitation.

The Glue: Searchable Knowledge Index

The 5 layers produce knowledge. The search index makes it findable. We use BM25 keyword search across 9 collections containing 734 indexed documents. Every agent can search every other agent’s knowledge, the shared library, and the full content archive.

This bridges the gap between agent-specific knowledge in Layer 4 and organizational knowledge in Layer 5. An agent researching a topic can find another agent’s analytics findings, a third agent’s content research, and the shared positioning documents, all from a single search. The index refreshes nightly, so new knowledge becomes searchable within 24 hours.

We run BM25 keyword matching, not semantic vector search. This is a constraint (CPU-only servers, RAM limitations), but it works well enough for our use case. Agent knowledge files use consistent terminology because they’re written by systems with consistent vocabulary. Keyword matching handles this reliably.

How It Works Together: A Real Example

One complete cycle through the architecture, using Link (our knowledge management agent) as the example. (For the full agent team and how our pipeline works, see the companion article.)

Step 1 — Journal (Layer 1): During a work session, Link writes a self-reflection noting that 17 work orders were created in the past week, but only 2 were implemented. A 12% completion rate. The observation: “I’m generating work faster than it can be approved and executed.”

Step 2 — Extraction (Layer 2): The process-thinking extraction runs immediately after the journal entry is complete. It finds 8 items across 4 categories: 3 decisions (“data-backed everything,” “shift focus to implementation,” “effectiveness means impact, not volume”), 4 knowledge items (an operating principle, pacing rules, a baseline metric, a coordination protocol validation), and 1 short-term task.

Step 3 — Trackers (Layer 3): The short-term task is added to the agent’s tracker. Existing items are checked for duplicates, none are added twice.

Step 4 — Knowledge Files (Layer 4): Four files are updated: learning/analytics-priority-shift.md gets a new operating principle, learning/pipeline-bottleneck-pattern.md gets updated pacing rules, a new baseline document is created with the week’s metrics, and team/coordination-patterns.md is updated with the protocol validation.

Step 5 — Shared Library (Layer 5): In this case, nothing. The insights are specific to the agent’s workflow. If the “data-backed everything” principle applied organization-wide, it would be committed to the shared library.

The next morning: The agent’s session starts by loading its trackers. It sees the new task from yesterday. It doesn’t need to re-read yesterday’s journal. The actionable item is already extracted, classified, and waiting.

Two weeks later: The agent’s biweekly self-reflection noodle fires. It reads the past two weeks of journals. But it also has the knowledge files from Layer 4, the baseline metrics, the pacing rules, the operating principles. It can measure progress against extracted benchmarks instead of re-deriving them from raw journal text.

Why This Architecture Beats the Alternatives

Against a flat vector store, the layered approach retrieves classified, deduplicated, routed knowledge. A vector search for “completion rate” returns fragments from scattered entries. Our system returns a clean, maintained document with the current pacing rules and historical context.

Against conversation history, our system is structured across 7 categories, bounded because tracker items get completed or dropped, and actively maintained because knowledge files are updated rather than appended to. Conversation history is linear, unbounded, and grows without limit.

Against a single knowledge base, our system routes each type of knowledge to its natural home. Tasks go to trackers, durable insights go to topic files, decisions go to memory logs, process improvements go to Stars. Each consuming agent or process gets exactly what it needs without sifting through everything else.

The common thread: extraction is the bottleneck, not storage. Every team building persistent agent memory can pick a storage technology in an afternoon. The hard part is building a reliable process to get knowledge out of raw agent output and into the right place. The 7-category checklist with a self-check gate is our answer. It’s the reason the same model extracts 8 items from a document instead of 1.

What We Got Wrong (And Still Haven’t Solved)

This system has been running in production for months. It works well enough to be worth publishing. It has real limitations we haven’t fixed.

Knowledge file staleness. When a knowledge file was last updated three months ago, is it still accurate? We don’t have a good signal for this. Noodles (self-scheduled reflections) help because they periodically re-examine stored knowledge, but there’s no systematic staleness detection. A knowledge file about a teammate’s working patterns could be out of date if that teammate’s workflow changed and nobody flagged it.

Extraction false negatives. The self-check gate catches under-extraction, but it’s not perfect. Some insights are subtle enough that the 7-category checklist doesn’t surface them. A nuanced observation about why something works, as opposed to the fact that it works, sometimes gets missed. We catch the “what” more reliably than the “why.”

Mis-classification. A separate problem from under-extraction: the classifier sometimes assigns the wrong category. A decision gets classified as a task, or a context-specific observation gets promoted to the shared library when it should stay in the agent’s own Layer 4 files. Unlike false negatives, which the self-check catches, mis-classifications are silent. A decision filed as a task still looks like a valid extraction, so the quality gate doesn’t flag it. Over time, these errors accumulate and degrade Layers 3 through 5.

How do mis-classifications actually get caught? In our experience, through four paths: a human notices the error while reviewing output, the system encounters a contradiction that forces a resolution, the agent naturally revisits the knowledge during a later session and spots the mismatch, or an external signal (like a reader pointing out an inconsistency on a published article) surfaces it. This is like any other bug: it needs to either be noticed or cause enough pain to surface. We don’t have an automated correctness check for classification accuracy, and we’re not sure one is possible without a second model reviewing every extraction, which would double the cost of the extraction step for marginal improvement.

Cross-agent knowledge pollution. When agent A’s context-specific learning is stored in the shared library, agent B might apply it in a situation where it doesn’t fit. The selective loading via decision matrix reduces this, but it’s not eliminated. An insight that “short emails get faster replies” might be true for one agent’s stakeholders and wrong for another’s. We’ve written about the broader challenge of securing agent access to shared knowledge; it’s an ongoing design problem.

Search limitations. BM25 keyword matching is reliable for agents that use consistent vocabulary, which ours do. But it doesn’t handle conceptual similarity. Searching for “work piling up” won’t find a knowledge file about “bottleneck in the review stage,” even though they describe the same problem. Semantic search would help, but our server constraints don’t support it today.

Knowledge volume scaling. With 7 agents and 734 documents, the system is manageable. At 50 agents or 10,000 documents, the nightly index rebuild, the cross-agent search queries, and the deduplication checks would need significant rearchitecting. We built this for our current scale, not for arbitrary scale.

Frequently Asked Questions

How do you give AI agents persistent memory?

With a layered agentic memory system that separates raw thinking from structured knowledge. Our approach uses 5 layers: journals for raw working notes, a process-thinking extraction step that classifies insights into 7 categories, trackers for actionable state, topic-specific knowledge files for durable learning, and a shared library for organizational knowledge. The extraction step is what makes it work. Without it, agents produce raw output that’s never organized into retrievable knowledge.

What’s the difference between agent memory and RAG?

RAG is fundamentally a read-only retrieval mechanism. It grounds the model in external documents that the agent didn’t write. Agent memory is read-write and agent-specific. The agent generates knowledge through its own work, extracts it, stores it, and retrieves it later. In our architecture, RAG corresponds roughly to Layer 5, the shared library. Layers 1 through 4 are the agent’s own memory.

Can AI agents share knowledge with each other?

Yes, through two mechanisms. The shared library (Layer 5) holds organizational knowledge that any agent can read and update. The search index (9 collections, 734 documents) lets any agent search any other agent’s knowledge files. The key constraint: not everything should be shared. Context-specific insights stay in the originating agent’s Layer 4 files. Only generalizable knowledge gets promoted to the shared library.

What’s the biggest challenge in AI agent memory?

Extraction, not storage. AI agents are prolific thinkers and poor self-editors. Without a structured extraction process, agents generate pages of journal text and store almost nothing useful. Our self-check gate, which improved extraction from 1 item to 8 items from the same document and model, exists specifically to address this. The 7-category checklist (task, goal, pattern, improvement, knowledge, decision, question) provides the structure. The self-check provides the quality control. It’s the core of what makes agentic memory work at scale.

How does agent memory differ from giving an LLM a longer context window?

Context windows are temporary, unstructured, and expensive to fill. Memory is permanent, classified, and searchable. An agent running 10 sessions per day for a week generates 70 sessions of history. No context window holds that, and even if it could, filling it indiscriminately degrades reasoning quality. Memory is the curated subset: decisions, knowledge, active tasks, and durable principles, loaded selectively based on what today’s session needs.

What tools do you need for agent memory?

A file system and a search index. We use markdown files organized by topic and a BM25 keyword search engine indexing all of them. You don’t need a vector database, a graph database, or a specialized AI agent memory product. The architecture matters more than the tooling. The 5-layer structure with extraction and routing would work on top of any storage system that supports organized files and keyword search.

How do you prevent agents from storing irrelevant information?

Through the 7-category extraction checklist and the self-check gate. The checklist forces classification: if something doesn’t fit any of the 7 categories, it stays in the journal and doesn’t get promoted to trackers or knowledge files. The self-check adds volume awareness: a 2-page reflection should yield 5 to 15 items. Significantly fewer suggests under-extraction; significantly more suggests over-extraction. Both trigger a re-scan.

Getting Started

If you’re building agents that need to persist knowledge across sessions, start with the extraction problem, not the storage problem. The 7-category checklist (task, goal, pattern, improvement, knowledge, decision, question) is technology-agnostic. You can implement it today, regardless of your stack, and immediately improve how much useful knowledge your agents retain from their own work.

From there, add structure: separate trackers for actionable items, topic-specific files for durable knowledge, and a shared repository for anything that applies across agents. The layering can be incremental. You don’t need all 5 layers on day one. You do need extraction from day one.

If your team is building multi-agent systems and running into the memory wall, the architecture described here is a starting point. We help teams design and implement agentic memory systems as part of our AI whiteboarding engagements, where we work through architecture decisions like these before writing any code.