Users expect chatbots to remember recent context. Naively stuffing full transcripts into every request breaks token budgets and latency SLAs. LangChain memory modules offer patterns from sliding windows to vector-backed long-term recall. Choosing the wrong memory pattern wastes tokens on every request or loses context users expect — match the pattern to conversation length and privacy requirements.
Buffer and window memory
ConversationBufferWindow keeps the last K turns — sufficient for many support bots and onboarding flows. Pair with token trimmers so verbose users cannot exhaust context windows in a single session. Set K based on your median conversation depth — support bots often need six to eight turns; transactional bots may need only three. Log turn count distributions to right-size K after two weeks of production traffic.
Window memory is predictable for cost forecasting. Use it when conversations are short and recent context dominates answer quality. Trim by token count rather than turn count when users paste long messages — a single turn can exceed half your context window. LangChain's trim_messages helper respects role boundaries when cutting history.
Summary memory for long sessions
Periodic LLM compression summarizes older turns into a compact state object. Costs extra on write but saves on every subsequent read — ideal for coaching, setup wizards, and complex B2B configuration chats. Store the summary as structured JSON — user preferences, open tickets, partial form state — not prose blobs that are expensive to parse on read. Version summaries so you can detect when compression lost critical facts.
Summarize on a schedule or when token count crosses a threshold. Store summaries with timestamps so you can audit what the bot believes it knows about the user. Trigger re-summarization when new facts contradict the summary rather than on a fixed schedule alone — stale summaries cause worse errors than no summary. Compare summary token count to full transcript savings monthly.
Vector store memory across sessions
Extract facts — preferences, account details, resolved issues — and index them for retrieval at query time. Index entities and preferences, not raw chat logs, to reduce privacy risk and retrieval noise. Use entity extraction on each turn to populate the vector memory index — raw transcript embeddings retrieve irrelevant chit-chat alongside useful facts. Namespace vector memory per user_id and thread_id in metadata filters.
Combine window memory for immediacy with vector memory for multi-session continuity. Always enforce tenant isolation in the memory index — memory leaks are trust killers. Set TTL on memory entries and honor user delete requests across both relational session stores and vector indexes — GDPR and DPDP both apply to chat memory. Never share memory indexes across tenants even for internal analytics.
What to do next
Start with one measurable outcome — latency, accuracy, adoption, or revenue — and design your stack around that constraint. GenOrbit ships production AI systems across healthcare, astrology, fintech, and SaaS; if you want a second pair of eyes on architecture or delivery, we are happy to talk.
Related: /services/llm-rag-agent-development · /blog/streaming-llm-responses-react