Agent Context Management: From Chat History to Structured Memory Assets

Agent context management has evolved significantly. What started as "keep the last N messages" has grown into a discipline of its own — deciding what enters the context window, how it's organized, and how to retrieve exactly what's needed without overflow.

This article traces the evolution and explains why structured memory assets matter.

Stage 1: Raw Chat History

The earliest approach: stuff all conversation history into the prompt. Every turn accumulates. Simple, but doesn't scale.

• Turn 1: 500 tokens
• Turn 10: 5,000 tokens
• Turn 50: 25,000+ tokens

Cost grows linearly. Eventually you hit the model's context window limit and either truncate (losing information) or start a new session (losing everything).

Stage 2: Summarization + Vector Retrieval

The next step: summarize old conversations, store summaries as embeddings, and retrieve relevant ones via vector search.

This is what solutions like Mem0 and Letta do. It works for single-agent scenarios but has limitations:

• Summaries lose precision — a version number or error string may not survive summarization
• Memory is a flat list of summaries — no structure, no ownership, no access control
• Can't handle non-conversation information: documents, code structure, reusable workflows

Stage 3: Structured Memory Assets

TencentDB Agent Memory takes a different approach: instead of one undifferentiated "memory" store, it recognizes that agents reuse different types of information and manages each as a distinct asset type.

Four Memory Asset Types

Four-Layer Architecture Within Chat Memory

Even within Chat Memory, information is layered by granularity:

Retrieval: L2/L3 for fast guidance; BM25 + vector search + RRF for specific facts; hard limits on entry count, character budget, and timeout.

Why This Matters for Context Management

The core insight: context management isn't about compression — it's about relevance.

Summarization compresses everything equally. Layered memory with structured assets lets you retrieve exactly what the current task needs:

• Starting a new feature? Load L2 scenario block for the project + CodeGraph for the relevant module
• User mentions a preference? It's in L1 atoms, retrieved by vector match
• Need to verify what was said exactly? Fall back to L0 raw conversation
• Onboarding a new agent? Equip it with L3 persona + relevant Skills

All within a character budget — memory enriches context without overflowing it.

Framework-Agnostic by Design

Memory assets aren't locked into one agent framework. A Proxy sits between agents and the memory server — point any agent's base URL to it, done.

Supported: DeepSeek Harness · Claude Code · Codex · CodeBuddy · WorkBuddy · Hermes · OpenClaw

Switch frameworks? Your memory follows.

Benchmark

PersonaMem: 48% → 76% (+59%) with structured memory enabled.

Quick Start

git clone https://github.com/TencentCloud/TencentDB-Agent-Memory.git
cd TencentDB-Agent-Memory/deploy/global-images
./start-all.sh

Panel: http://localhost:8125

Repo: https://github.com/TencentCloud/TencentDB-Agent-Memory