LLMs are stateless: every request starts from zero, and “memory” is just whatever text you resend. Three layers build real systems — short-term context inside a 128k-token context window, the conversation history you manage and trim, and long-term memory via summarization and retrieval-based memory. This topic covers token limitations, context budgeting and the patterns that keep a two-week incident thread coherent.
LLMs are stateless: every request starts from zero, and “memory” is just whatever text you resend. Three layers make up real systems. Short-term context = the current conversation inside the context window — 128,000 tokens on GPT-4o-class models (roughly 300 pages), 200k on Claude. Conversation history = the messages array you append to and resend each turn. Long-term memory = anything that survives the session: a vector store of past facts retrieved per query (retrieval-based memory), or structured memory — user preferences, open tickets, ruled-out causes — persisted in a database and injected into the prompt.
A troubleshooting assistant without memory re-asks for the device name every message; without context management it silently forgets the first half of a long incident. Token limits are money and quality at once: a 128k window lets you brute-force “keep everything”, but prompt cost scales linearly with history, and models attend less reliably to the middle of very long contexts (“lost in the middle”). Knowing when to trim, when to summarize and when to retrieve is the difference between a demo and an assistant that survives a two-week incident thread.
Manage the window in layers. Recent turns stay verbatim (last 5-10 messages). Older history gets summarized — a cheap model compresses 50 messages into a 200-token running summary that replaces them. Relevant knowledge arrives by retrieval: per query, fetch top-k chunks from the vector store instead of carrying whole documents. Fixed context (system prompt, user profile, current device facts) sits at the top where attention is strongest. Budget it: e.g. 1,000 tokens system + 500 summary + 2,500 retrieved + 2,000 recent turns ≈ 6k of a 128k window — headroom beats a full window on both cost and accuracy.
┌─────── Context Window (128k tokens) ───────┐
│ System prompt + user profile (fixed) │
│ Running summary of older turns │
│ Retrieved chunks (vector search) │
│ Last 5-10 messages, verbatim │
└────────────────────────────────────────────┘
↓
Stateless LLM → reply → history +1 messageAn incident thread that outlives the window: Monday the agent triages 0x800705B4, Wednesday it reviews new IME logs, Friday the admin returns asking “where were we?”. Verbatim history would be 60k tokens by then. Instead the system keeps a running summary (“LAP-042, Autopilot ESP stall, GPO filter ruled out, next: check ESP timeout config”), retrieves relevant KB chunks per question, and holds a facts table — device, error, ruled-out causes — in the database. This is the same pattern Copilot-style assistants use to feel like they “remember” across sessions.
Store the messages array (or its source) keyed by user + session in your database. The API is stateless; forgetting to persist means your bot has amnesia across page reloads.
Keep the system prompt + last 5-10 messages verbatim. Count tokens — with tiktoken or the usage field in API responses — never guess. A “roughly fine” window is how cost spikes and truncation surprises happen.
When history exceeds the budget, run a cheap-model pass: “Summarize this conversation in under 200 tokens, preserving error codes, device names and decisions.” Replace old messages with the summary; keep codes and names verbatim inside it.
Embed durable facts and past incidents into your vector store. Per query, retrieve top-k and inject as context — the user gets long-term memory without paying for it on every turn. Topic 13 built exactly this index.
Keep structured state — device, error code, ruled-out causes, next step — in a database and re-inject the current state each turn. Prompts drift; databases do not.
Simulate 40-turn sessions and check whether the assistant still knows the device name and the original error at turn 40. Memory bugs only appear in long conversations — short demos hide them.
npx tiktoken-cli --encoding o200k_base --file chat.json -- or in code: const n = enc.encode(JSON.stringify(messages)).length; // keep n under your budgetMemory is a data-retention decision, not just a UX one. Conversation histories and extracted facts contain whatever users typed — treat them like logs: encrypt at rest, scope by tenant/user, honour deletion requests (GDPR), and set retention limits. Beware cross-session leakage: a fact extracted from one user’s session must never be retrieved for another. And sanitize before summarizing or embedding — secrets pasted into chat (“the BitLocker key is…”) should be redacted before they enter any store.
Only because the application resends context: conversation history for the short term, a running summary for older turns, and retrieved facts from a vector store for long-term memory. Structured state — device, incident, decisions — lives in a database and is re-injected each turn. The model contributes reasoning; the architecture provides memory.
The maximum tokens a model can consider in one request — 128k on GPT-4o-class models. Using it all costs linearly per turn and degrades quality: models attend less reliably to the middle of very long contexts. Curated context — summary + retrieval + recent turns — outperforms a stuffed window on both accuracy and cost.
Summarization compresses this conversation — it preserves narrative continuity (“we already ruled out GPO”) cheaply. Retrieval brings back relevant knowledge from outside the conversation — past incidents, documentation — and is better for facts you need selectively. Most production assistants run both: summary for the thread, retrieval for the corpus.
Extend your topic-15 chatbot: persist history per session, keep the last 8 messages verbatim, summarize anything older into a ≤200-token running summary, and store three facts per session (device, error, status) that are re-injected every turn. Then run a 30-turn scripted conversation and verify the bot still knows the device name at the end.
About 4 characters or ¾ of an English word; Hindi (Devanagari) typically costs more tokens per word — roughly 1.5-2x. A 128k window is roughly a 300-page book, but the per-turn bill scales with what you actually send, which is why trimming matters.
No — they are product layers doing exactly what this topic describes: extracting durable facts, storing them server-side, and injecting them into future prompts. The underlying API call remains stateless, with the full context resent every time.
Depending on the API: the request errors, or the platform silently truncates — often dropping your older messages or, worse, part of the system prompt. Never rely on implicit truncation; measure tokens yourself and manage the window explicitly.