Seven RAG failure modes in Java: chunking, embedding mismatch, stale indexes, missing filters, context stuffing, no eval, dropped metadata. pgvector samples.
The demo that always works: you point a notebook at a vector index, ask a question, and it answers. Three weeks later the same system cites a document that is not in the store, and occasionally stuffs another tenant's row into the prompt. I have watched more RAG projects die from unmeasured drift than from any modeling choice. I ship this in Java: Spring Boot, PostgreSQL, pgvector , a leather inventory assistant that has to answer zapatos cafe talla 41 without inventing a pair we do not stock. Retrieval-augmented generation (RAG) is not plumbing. Embed the query, find nearest neighbors, stuff them into a prompt: that sentence hides seven hops that can each lie politely. Java has to refuse. One question. Five neighbors. No size 41. Vector search always returns a neighbor. That is not an answer. Seven hops that lie Seven pitfalls. Chunking like bread. Query and index embedded by different models. An index that never updates. Plausible-but-wrong context and cross-tenant leakage. Stuffing the context window. Shipping with no eval. Flattening metadata until you cannot filter, cite, or audit. The posture is portable: make retrieval explicit and measurable. Pin one embedding model. Scope the candidate set before approximate nearest neighbor (ANN) runs. Refuse weak matches. Rerank and trim. Keep a golden set. Treat RAG as plumbing and these are the default outcome. Not exotic incidents. Chunking like bread The lazy default is a fixed character window with zero overlap, applied to API pages, contracts, and product rows. Two shapes show up. Chunks too big: the embedding averages six unrelated topics and cosine goes mushy. Chunks too small: you split mid-sentence, the retriever returns the maximum is , and the model invents the rest. The worst version cuts a table so the header lands in chunk 7 and the values land in chunk 8. For a catalog I do not slice SKUs. One parent document. One variant document. Size stays with the row. Structure first, size second. Match the embedding model's real token window so you do not truncate silently. When an answer is wrong, read the retrieved chunks first. Half the time the model did its job and the chunk was garbage. There is no universal chunk size. There is a universal debugging move: open the rows. public record EmbeddingDocument( String entityType, // PARENT or SKU UUID entityId, String textEs, Map<String, Object> metadata) {} List<EmbeddingDocument> documents = new ArrayList<>(); documents.addAll(loadParentDocuments(importId)); // family, color, parent_code documents.addAll(loadSkuDocuments(importId)); // item_code, talla, stock // Do not run RecursiveCharacterTextSplitter across a SKU row. Docs and markdown still want heading-aware splitters and a modest overlap. Tables and code blocks stay intact. Catalog RAG is a different shape. Do not copy a blog-chunker onto an Excel ingest. Two embedding brains This one is subtle because nothing errors. You indexed in June with text-embedding-3-small . Last sprint someone wrote a query path and reached for another 1536-d model because it was top of mind. Dimensions match. similarity_search succeeds. The results are quietly nonsense: two different geometries. The cousin: you re-embed half the corpus after a model bump. Recall craters. No exception to point at. Stop letting the embedding model be an implicit choice scattered across the codebase. One port. Index and query both call it. Store model_id on the row. Assert it at query time. A model change is a new index, not a mixed table. 1024 dimensions is not compatibility. Pin the name and the version. public interface EmbeddingGateway { boolean isAvailable(); String modelId(); // e.g. openai/text-embedding-3-small int dimension(); // 1536 float[] embed(String text); List<float[]> embedAll(List<String> texts); } float[] queryVector = embeddingGateway.embed(normalizedQuery); if (!embeddingGateway.modelId().equals(indexModelId)) { throw new IllegalStateException( "Embedding model mismatch. Reindex the corpus."); } You get that guarantee only if every call goes through one gateway. Two RestClient beans with “the same dim” is how this ships to production. The index that never updates Symptom: you edit the Excel, the relational tables clearly have the new text, ingest reports success, and the retriever still serves last week's answer. The endpoint is healthy. Nothing is red. It is just stale. The usual root cause is a pipeline nobody called. Embeddings are a second write. Relational ingest is not the vector index. After importUpload completes I have to call indexImport(importId) . If the gateway is down, I log and skip. Retrieval then falls back to trigram. That is honest. Serving last week's vectors as if they were live is not. UUID importId = inventoryImportService.importUpload(file); int indexed = productEmbeddingIndexer.indexImport(importId); if (indexed == 0) { log.warn("Import {} has no embeddings. Retrieval will use trigram only.", importId); } // You call this. It does not run because the row exists. Scope vectors by import_id so an old snapshot cannot leak into a new chat. Count rows after index. If indexed_row_count never moves, you did not re-embed. Do not debug the prompt first. Always a neighbor. No floor. Ask about a product you do not sell and you still get five neighbors, ranked, looking authoritative. The model grounds on them and produces a fluent, specific, completely wrong reply. People blame the LLM. The retriever handed it garbage with a straight face. The dangerous version is multi-tenant. Nearest-neighbor search does not care about ownership. Tenant B's contract is semantically close to tenant A's question. That is not a quality bug. That is a leak. I have seen the filter sit on the backlog until after launch. Two fronts. Always filter by the metadata that scopes the request: snapshot, tenant, document type, recency. Derive that id from the authenticated session, server-side. A tenant filter the user can override is not a tenant filter. Parameterized SQL only. Then set a similarity floor. If the best match is below it, treat it as no relevant context and say so. ANN does not know ownership. Honest empty beats a fluent lie. SELECT pe.entity_id, 1 - (pe.embedding <=> CAST(:queryVector AS public.vector)) AS score, sv.item_code FROM syj_rag.product_embedding pe JOIN syj_rag.sku_variant sv ON sv.id = pe.entity_id WHERE pe.import_id = :importId -- session, not request JSON AND pe.entity_type = 'SKU' AND 1 - (pe.embedding <=> CAST(:queryVector AS public.vector)) >= :minScore ORDER BY pe.embedding <=> CAST(:queryVector AS public.vector) LIMIT :limit I pin min-score at 0.70 in config. The number is a product choice. The refusal is not. Empty after the floor is NO_CONTEXT . The chat says we do not have a document that answers that. Fluent is not a success metric. Stuffing the window Bigger context windows tempted a bad habit: retrieval is fuzzy, so send top-20 and let the model sort it out. You pay for thousands of tokens of mostly irrelevant text. You walk into lost-in-the-middle: models attend to the start and end and skim the middle, so the one chunk that answered the question, sitting at position 11, gets ignored. The right answer was in the prompt. The model never read it. More retrieved chunks is not more knowledge. Retrieve a wider candidate set if you like. Then fuse, rerank, and trim to a tight few. Order so the strongest land where the model actually looks. Delimit chunks. Cite item_code . int fetchLimit = Math.max(variantTopK * 2, parentTopK * 3); List<RetrievalCandidate> vectorHits = repository.searchByVector(importId, queryVector, fetchLimit, minScore, "SKU"); List<RetrievalCandidate> trigramHits = repository.searchByTextTrigram(importId, query, fetchLimit); List<RetrievalCandidate> fused = fuse(vectorHits, trigramHits); List<RetrievalCandidate> ranked = Retriev...