LLM chats get expensive when every turn replays the full transcript. Bound the window, then summarize older turns on purpose.
Turn 22 asked me to apply “that” to this application. “That” was a hiring rule from turn 5: five years of Java to leave phone screen. The assistant still had every greeting, every tool dump, and twenty retrieved resumes in the prompt. The model remembered. The bill remembered louder. I do not want a recruiter assistant that forgets the opening. I also do not want one that resends the opening on every later turn. Efficient summarization is the boring middle: keep the last complete turns as they were said, and replace the older ones with a short rolling summary before they leave the prompt. The leak I keep shipping I keep a Java lab for this. It is the Spring AI 2.0 support assistant in learning-with-me/tokens-reduction . It looks like something a team would ship. The waste is ordinary. One of the five findings the scanner emits is MEMORY-UNBOUNDED-HISTORY , severity High. The recommendation in the analyzer is the sentence this article is about: use bounded message or token memory, and summarize before eviction. The source shape it matches is an append-only list named like a transcript. The fixture compiles against Spring AI BOM 2.0.0 : groovy · examples/spring-ai-support-assistant/build.gradle implementation platform('org.springframework.ai:spring-ai-bom:2.0.0') implementation 'org.springframework.ai:spring-ai-client-chat' implementation 'org.springframework.ai:spring-ai-vector-store' ChatModel in the lab is an offline test double. Your app still needs a model starter. The leak is the list, not the provider. java · SupportAssistantService, the fixture package com.example.support; import java.util.ArrayList; import java.util.List; import org.springframework.ai.chat.client.ChatClient; import org.springframework.ai.chat.messages.AssistantMessage; import org.springframework.ai.chat.messages.Message; import org.springframework.ai.chat.messages.UserMessage; import org.springframework.ai.document.Document; import org.springframework.ai.vectorstore.SearchRequest; import org.springframework.ai.vectorstore.VectorStore; public class SupportAssistantService { private final ChatClient assistant; private final VectorStore vectorStore; /** Append-only transcript with no window or summarization; replayed on every turn. */ private final List<Message> conversation = new ArrayList<>(); public SupportAssistantService(ChatClient assistant, VectorStore vectorStore) { this.assistant = assistant; this.vectorStore = vectorStore; } public String answer(String question, String customerId) { List<Document> hits = vectorStore.similaritySearch( SearchRequest.builder().query(question).topK(20).build()); StringBuilder context = new StringBuilder(); if (hits != null) { for (Document hit : hits) { context.append(hit.getText()).append("\n\n"); } } for (Message message : conversation) { context.append(message.getMessageType()).append(": ") .append(message.getText()).append('\n'); } String userMessage = "Customer " + customerId + " asks: " + question + "\n\nUse the following context:\n" + context; String answer = assistant.prompt() .user(userMessage) .call() .content(); conversation.add(new UserMessage(question)); conversation.add(new AssistantMessage(answer == null ? "" : answer)); return answer; } public List<Message> transcript() { return List.copyOf(conversation); } } Retrieval topK(20) is a separate finding, RAG-EXCESSIVE-TOP-K . The memory bug is the list that never shrinks, stuffed into the same user message as the chunks. Each turn pays for every earlier turn, plus a fresh RAG dump, plus a system prompt that already tried to cover every hiring stage in one evergreen block. The lab fixture does the same thing with billing, SSO, and seat management. When a recruiter escalates, the coordinator forwards the entire transcript to the specialist. That is a second finding, AGENT-OVERSIZED-HANDOFF . Compacting memory does not excuse handing a child agent the parent’s whole context. Pass a brief: the opening, the rule, the evidence for this application. java · EscalationCoordinator.handle, the oversized handoff import java.util.stream.Collectors; public String handle(String question, String customerId) { String draft = assistant.answer(question, customerId); if (!needsSpecialist(question, draft)) { return draft; } String entireConversation = assistant.transcript().stream() .map(message -> message.getMessageType() + ": " + message.getText()) .collect(Collectors.joining("\n")); return specialist.handoff(entireConversation); } assistant is SupportAssistantService . specialist is a second ChatClient wrapper. The specialist should get a brief, not transcript() . Memory is not history LangChain4j’s chat-memory tutorial draws a line I wish more Spring apps would draw. History is what the user sees in the UI. It should stay intact. Memory is what you send the model so it behaves as if it remembers. Memory is allowed to drop, compress, and rewrite. If you need a complete record, store it yourself. Do not make the prompt be the archive. Spring AI’s own Chat Memory page says the same thing in different words. ChatMemory is for contextual awareness. It is a poor store for the full conversation. I keep history in the product database. I keep memory in a window, and when the window is no longer enough I summarize the part that is about to leave. History is what the user sees. Memory is what you are willing to pay to send the model. Three policies I only use three. Mixing them in one prompt without naming the policy is how the list grows again. Policy What the model sees What you pay What you lose Unbounded replay Every stored message, every turn Grows without bound Nothing, until the context window or the bill stops you Sliding window The last N messages or tokens Capped, no extra model call Anything older than the cut, including “that” from turn 5 Summarize-and-keep-recent A rolling summary plus the last complete turns Capped, plus one cheaper model call when the trigger fires Wording and tool JSON you did not copy into the summary A window that only evicts is cheap. It is also amnesiac. Summarization is for sessions that still need older facts after those turns no longer fit. what the next prompt should contain system instructions (stable, short) │ ▼ rolling summary of older turns │ ▼ last complete turns, verbatim │ ▼ this user message │ ▼ retrieved docs for this question, not last week's Do not splice last turn’s chunks into memory. Retrieval belongs to this question. Memory belongs to the session. First fix: a message window Spring AI 2.0 already auto-configures a ChatMemory bean as MessageWindowChatMemory over an in-memory repository. The default window is 20 messages. Older ones leave. SystemMessage stays. Eviction snaps to a turn boundary, so the kept window starts at a UserMessage instead of cutting through a tool call. Size maxMessages large enough to hold at least one representative turn. A multi-step tool exchange can be bigger than a naive 8. Wire it through MessageChatMemoryAdvisor . Pass a conversation id you minted on the server from the signed-in user, not a constant shared across customers. ChatModel still comes from the model starter you already use. Memory types live in the client-chat module. Gradle, Spring AI 2.0 ChatMemory dependencies { implementation platform('org.springframework.ai:spring-ai-bom:2.0.0') implementation 'org.springframework.ai:spring-ai-client-chat' // Optional JDBC store. It silently drops tool calls: // spring-ai-starter-model-chat-memory-repository-jdbc } MessageWindowChatMemory + MessageChatMemoryAdvis...