RAG isn't just one thing. There are actually several flavors of it, each suited for different use cases, and I've been deep in the weeds building them with LangChain4j. Here's what I learned along the way
RAG broke my assumptions. I thought it was just "put docs in a vector store, search at query time, done." That was the version I shipped on the internal knowledge base tool we built for the support team. It worked okay, I guess, but "okay" in production with real users asking real questions is basically failure in slow motion. So I went deeper and turns out there are several meaningfully different patterns that all get called RAG, and they're not interchangeable. Which one you reach for depends on what your data looks like, how fresh it needs to be, and what kind of questions you're trying to answer. I've been building with LangChain4j for a few months now (the Java ecosystem's answer to LangChain/LlamaIndex), and I want to share what I've learned, because the docs, while decent, don't always make these distinctions obvious. What We Mean When We Say RAG Retrieval-Augmented Generation is the idea that instead of relying purely on what a language model memorized during training, you fetch relevant context at inference time and include it in the prompt. The model then answers based on that context. Simple concept. The complexity is entirely in the retrieval part. LangChain4j gives you the building blocks: embedding models, vector stores, document loaders, retrievers, and an AiServices abstraction that wires it all together. I'm currently on 1.15.0 . The Maven dependency to get started: <dependency> <groupId>dev.langchain4j</groupId> <artifactId>langchain4j</artifactId> <version>1.15.0</version> </dependency> You'll add provider-specific modules on top of that, things like langchain4j-open-ai or langchain4j-cohere , depending on what you're connecting to. Naive RAG (The Starting Point Everyone Uses) You chunk your documents, embed them, store the embeddings in a vector store, and at query time you embed the user's question, find the top-k nearest chunks, stuff them into the prompt, and ask the LLM to answer. In LangChain4j 1.15.0, that flow looks something like this: EmbeddingStore<TextSegment> embeddingStore = new InMemoryEmbeddingStore<>(); EmbeddingModel embeddingModel = OpenAiEmbeddingModel.builder() .apiKey(System.getenv("OPENAI_API_KEY")) .modelName("text-embedding-3-small") .build(); EmbeddingStoreIngestor ingestor = EmbeddingStoreIngestor.builder() .documentSplitter(DocumentSplitters.recursive(500, 50)) .embeddingModel(embeddingModel) .embeddingStore(embeddingStore) .build(); ingestor.ingest(documents); // List<Document> loaded from files, URLs, etc. ContentRetriever retriever = EmbeddingStoreContentRetriever.builder() .embeddingStore(embeddingStore) .embeddingModel(embeddingModel) .maxResults(5) .minScore(0.75) .build(); Then you wire retriever into an AiServices interface and you're off. It's genuinely fast to set up. Like, embarrassingly fast for what it does. interface SupportAssistant { String answer(String question); } SupportAssistant assistant = AiServices.builder(SupportAssistant.class) .chatLanguageModel(chatModel) .contentRetriever(retriever) .build(); The problem is chunk quality. Split documents at 500 tokens with 50-token overlap and you will absolutely get splits that cut across a logical idea. The retrieved chunk ends up confusing or incomplete. I hit this hard with our policy documents, where a sentence in chunk N referenced a definition sitting back in chunk N-2. The model had no idea what it was looking at. Not great. But it's still the right starting point, especially for prototyping. Advanced RAG: Query Rewriting and Re-ranking The first upgrade worth making is query transformation before retrieval. Users don't ask questions the way documents are written. They use different vocabulary, they're vague, they cram multi-part things into one sentence. LangChain4j has a QueryTransformer interface for this. A simple version uses the LLM itself to rewrite the query: ChatLanguageModel chatModel = OpenAiChatModel.builder() .apiKey(System.getenv("OPENAI_API_KEY")) .modelName("gpt-4o-mini") .build(); QueryTransformer queryTransformer = new CompressingQueryTransformer(chatModel); RetrievalAugmentor augmentor = DefaultRetrievalAugmentor.builder() .queryTransformer(queryTransformer) .contentRetriever(retriever) .build(); SupportAssistant assistant = AiServices.builder(SupportAssistant.class) .chatLanguageModel(chatModel) .retrievalAugmentor(augmentor) .chatMemory(MessageWindowChatMemory.withMaxMessages(10)) .build(); CompressingQueryTransformer takes the conversation history into account and rewrites the current message into a standalone, context-aware query. So if a user asks "what about the exceptions?" after asking about a refund policy, it expands that to something like "what are the exceptions to the refund policy?" before hitting the vector store. This alone made a visible difference in retrieval quality on our support tool. One change, measurably better results. The second upgrade is re-ranking. You retrieve a broad set (say, top 15) and then a re-ranker scores each result for relevance to the actual question, keeping only the top 3 or 4. Cohere's re-rank API is what I've used in production. The improvement is real, but it adds latency, somewhere around 400-600ms in my testing. ScoringModel scoringModel = CohereScoringModel.builder() .apiKey(System.getenv("COHERE_API_KEY")) .modelName("rerank-english-v3.0") .build(); ContentAggregator contentAggregator = ReRankingContentAggregator.builder() .scoringModel(scoringModel) .maxResults(4) .build(); ContentRetriever broadRetriever = EmbeddingStoreContentRetriever.builder() .embeddingStore(embeddingStore) .embeddingModel(embeddingModel) .maxResults(15) .build(); RetrievalAugmentor augmentor = DefaultRetrievalAugmentor.builder() .queryTransformer(queryTransformer) .contentRetriever(broadRetriever) .contentAggregator(contentAggregator) .build(); Worth it for most production use cases. Not worth it if your retrieval corpus is small and already well-structured. Use your judgment. Modular RAG: Multiple Retrievers This one took me a while to appreciate. Sometimes your data lives in different places with genuinely different access patterns. On the claims processing service rewrite last fall, we had structured claim data in Postgres and unstructured policy documents sitting in S3. Embedding the Postgres rows made no sense. You want SQL for structured queries and vector search for the unstructured text. Two different tools for two different problems. DefaultRetrievalAugmentor accepts a list of ContentRetriever instances. Each one gets queried and the results are merged. ContentRetriever sqlRetriever = new ClaimsDataRetriever(dataSource); // custom impl ContentRetriever docRetriever = EmbeddingStoreContentRetriever.builder() .embeddingStore(embeddingStore) .embeddingModel(embeddingModel) .maxResults(5) .build(); RetrievalAugmentor augmentor = DefaultRetrievalAugmentor.builder() .contentRetrievers(List.of(sqlRetriever, docRetriever)) .contentAggregator(contentAggregator) .build(); ClaimsDataRetriever just implements ContentRetriever , which is a single-method interface. Pretty clean to write once you understand the contract: public class ClaimsDataRetriever implements ContentRetriever { private final DataSource dataSource; public ClaimsDataRetriever(DataSource dataSource) { this.dataSource = dataSource; } @Override public List<Content> retrieve(Query query) { String sql = "SELECT summary FROM claims WHERE to_tsvector(summary) @@ plainto_tsquery(?)"; try (Connection conn = dataSource.getConnection(); PreparedStatement ps = conn.prepareStatement(sql)) { ps.setString(1, query.text()); ResultSet rs = ps.executeQuery(); List<Content> results = new ArrayList<>(); while (rs.next())...