Most RAG tutorials assume you live in Python. I don't, and I'd bet a lot of you don't either. Here's how I built a fully functional RAG pipeline using LlamaIndex concepts, but with Java and Spring Boot.
Can Spring AI Replace LlamaIndex for Java RAG Applications? Last year I spent a few weeks building a document Q&A feature for an internal tool at a fintech company I was contracting with. The idea was straightforward enough: ingest a pile of regulatory PDFs, let compliance people ask questions in plain English, and return answers with citations. Classic RAG stuff. So I started digging into what the Java ecosystem actually had for this kind of work, besides LangChain4j. Turns out, more than I expected. The Java AI Ecosystem Is Not What It Was A Year Ago Spring AI has become the obvious starting point for Java teams that want to build LLM-powered applications without leaving the Spring ecosystem. For this kind of project today, I would target Spring AI 1.1.x with Spring Boot 4.0.x. There is newer milestone work happening toward the 2.x line, but for production systems I would stay on the stable branch unless I had a very specific reason to experiment. The easiest way to think about Spring AI is this: it is the Spring-idiomatic way to wire up LLMs, embedding models, vector stores, advisors, and document ingestion pipelines. It is not a direct port of LangChain or LlamaIndex. That is probably a good thing. It feels like a Spring project, not like a Python framework translated into Java. When I first heard “Spring AI,” I assumed it was going to be a thin wrapper around the OpenAI REST API with a few @Bean annotations sprinkled on top. It is more thoughtful than that. The abstractions are familiar if you have spent years in Spring Boot: configuration through properties, dependency injection, starters, templates, clients, and integration points that fit into the rest of the application. That matters more than people think. In production, the framework with the most features is not always the best fit. Sometimes the best fit is the one your team can actually operate. What LlamaIndex Gives You Before comparing anything, it helps to be clear about what we are replacing. In a Python project, LlamaIndex gives you a lot out of the box. It handles document loading, chunking, embedding generation, vector store indexing, retrieval, response generation, reranking, query transformations, response synthesis, and evaluation. Spring AI covers the core RAG flow, but the vocabulary is different. A Document represents text plus metadata. TokenTextSplitter handles chunking. VectorStore abstracts the vector database. QuestionAnswerAdvisor handles the retrieval step before context is sent to the model. So the mental model maps pretty cleanly: PDF → Documents → Chunks → Embeddings → Vector Store → Retrieval → LLM Answer That is the core of RAG, regardless of whether you are using Python or Java. The difference is that Spring AI lets you keep that flow inside a normal Spring Boot application. No separate Python runtime. No extra service. No second deployment model. No “who owns this FastAPI sidecar?” conversation six months later. For the fintech team, that mattered. Setting Up The Project For a current Spring AI setup, I would use the newer starter names rather than the older 1.0-style dependencies. The dependency management block looks like this: <dependencyManagement> <dependencies> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-bom</artifactId> <version>1.1.7</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> And the dependencies: <dependencies> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-starter-model-openai</artifactId> </dependency> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-starter-vector-store-pgvector</artifactId> </dependency> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-pdf-document-reader</artifactId> </dependency> </dependencies> I am using PgVector here because that is what we ran in production for the ATS. You could use another vector store, but PgVector or MariaDB is a very practical option for Java teams already running PostgreSQL. It keeps the architecture simple. One database. One backup strategy. One operational model. For a lot of internal tools, that is not a compromise. That is a feature. The properties are straightforward: spring.ai.openai.api-key=${OPENAI_API_KEY} spring.ai.openai.embedding.options.model=text-embedding-3-small spring.ai.openai.chat.options.model=gpt-4o spring.ai.vectorstore.pgvector.dimensions=1536 spring.ai.vectorstore.pgvector.distance-type=COSINE_DISTANCE The dimensions setting is worth calling out. text-embedding-3-small produces 1536-dimensional embeddings, so PgVector needs to be configured with the same value. That is one of those tiny details that can waste an afternoon if you get it wrong. The Ingestion Pipeline The ingestion pipeline took me longer to get right than the query side. The regulatory PDFs were dense, inconsistently formatted, and some were scanned. Scanned PDFs are basically useless for text extraction unless you run OCR first, which is a separate problem. For normal text-based PDFs, Spring AI gives you a pretty clean path. Here is the simplified version of the ingestion service: import org.springframework.ai.document.Document; import org.springframework.ai.reader.ExtractedTextFormatter; import org.springframework.ai.reader.pdf.PagePdfDocumentReader; import org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig; import org.springframework.ai.transformer.splitter.TokenTextSplitter; import org.springframework.ai.vectorstore.VectorStore; import org.springframework.core.io.Resource; import org.springframework.stereotype.Service; import java.time.Instant; import java.util.List; @Service public class DocumentIngestionService { private final VectorStore vectorStore; public DocumentIngestionService(VectorStore vectorStore) { this.vectorStore = vectorStore; } public void ingest(Resource pdfResource) { var reader = new PagePdfDocumentReader( pdfResource, PdfDocumentReaderConfig.builder() .withPageTopMargin(0) .withPageExtractedTextFormatter( ExtractedTextFormatter.builder() .withNumberOfTopPagesToSkipBeforeDelete(0) .build() ) .withPagesPerDocument(1) .build() ); var splitter = new TokenTextSplitter(512, 64, 5, 10000, true); List<Document> documents = splitter.apply(reader.get()); documents.forEach(document -> { document.getMetadata().put("source", pdfResource.getFilename()); document.getMetadata().put("ingested_at", Instant.now().toString()); }); vectorStore.add(documents); } } The most important part here is not the PDF reader. It is the chunking. This line matters: var splitter = new TokenTextSplitter(512, 64, 5, 10000, true); That gives us 512-token chunks with 64 tokens of overlap. The overlap is the thing people forget. Without overlap, an important sentence or concept can land across a chunk boundary. Then neither chunk has enough context to be useful. Retrieval quality drops, and the model starts giving answers that look confident but feel slightly wrong. I spent an embarrassingly long afternoon debugging bad answers before realizing the chunks were too clean. No overlap. That was the whole problem. Metadata Is Part Of The Retrieval Strategy One thing I wish I had done from day one was attach richer metadata during ingestion. At minimum, you want the source filename and ingestion timestamp: documents.forEach(document -> { document.getMetadata().put("source", pdfResource.getFilename()); document.getMetadata().put("ingested_at", Instant.now().toString()); }); In a real syst...