Most RAG pipelines feel solid until you actually put a number on them. That's where RAGAS comes in, and honestly, the results might surprise you. I wrote up how to evaluate your RAG properly, because vibes are not a metric.
I was working on an internal knowledge assistant for a client, basically a RAG system sitting on top of about 40,000 documents from their engineering wiki and support tickets. The thing felt good. Responses seemed coherent, the team was happy during demos, and I was starting to get a little too confident. Then someone asked it a question about our incident response policy, and it answered with complete confidence using information from a completely unrelated document about deployment pipelines. The answer was fluent, well-structured, and wrong. Nobody caught it during the demo because it sounded right. That's when I realized I had no real measurement in place. Vibes aren't a metric. I'd heard about RAGAS (Retrieval Augmented Generation Assessment) from Jonathan Sanchez , who'd used it on a customer support bot project the previous quarter. He kept saying "you need to actually score this thing" and I kept nodding and not doing it. I finally sat down with it properly, and I want to share what I found, because the gap between "it feels like it's working" and "I have numbers that tell me it's working" is bigger than I expected. One quick note before we get into the code: RAGAS is a Python library, full stop. There's no official Java SDK but if your team is running a Java backend (Spring Boot, Quarkus, whatever), you still have good options. I'll show you how to call the RAGAS evaluation service from Java using a small Python sidecar that exposes a REST endpoint, which is exactly how we wired it up on this project. What RAGAS Actually Is RAGAS is an open-source evaluation framework specifically built for RAG pipelines. It's been around since late 2023 and has matured quite a bit through 2024. The current version (0.1.x) has a reasonably clean API and integrates with LangChain and LlamaIndex with minimal friction. The core idea is that you give RAGAS three things: a question, the answer your system produced, and the context chunks your retriever actually fetched. From those inputs it calculates several metrics automatically, using an LLM under the hood to judge quality. Yes, you're using an LLM to evaluate an LLM. I know how that sounds. But in practice, with GPT-4 as the judge, the scores correlate surprisingly well with human ratings, at least in my experience. The Metrics You Actually Need to Understand RAGAS ships with several metrics. I'm going to focus on the four I actually use and can explain clearly, because some of the others feel half-baked to me honestly. Faithfulness Measures whether the answer is grounded in the retrieved context. Not whether it's correct in some absolute sense, just whether every factual claim in the answer can be traced back to something in the retrieved chunks. This is the metric that would have caught my incident response problem. The answer my system gave was confidently stated but unsupported by the retrieved documents. Faithfulness would have been low. Score range is 0 to 1. Anything below 0.7 in my experience means your system is hallucinating too much to be trusted in production. Answer Relevancy Measures whether the generated answer is actually relevant to the question asked. Sounds obvious, but this catches a subtle failure mode where the system retrieves the right documents and then generates a response that technically references those documents but doesn't answer what was asked. This one surprised me because I assumed relevancy would always be high if faithfulness was high. Not true. I've seen cases where the retrieved context was perfect and the LLM still went off on a tangent. Context Precision This is about your retriever. Of all the context chunks that were retrieved, how many were actually useful for answering the question? If your retriever is pulling in 5 chunks and only 1 is relevant, your context precision score is going to be bad. Low context precision usually means you're adding noise to the LLM prompt, which hurts answer quality and wastes tokens. Both bad. Context Recall The flip side. Did your retriever actually get the chunks it needed? This one requires a ground-truth answer to compute, so it's slightly harder to calculate, but it's important for catching cases where your retriever is too narrow. Low recall means the information existed in your knowledge base but your retriever missed it. That's a chunking or embedding problem, usually. The Architecture: Python Sidecar + Java Client Since RAGAS is Python-only, the cleanest approach for a Java shop is to wrap it in a small Flask (or FastAPI) service and call it over HTTP from your Java application. We ran this as a sidecar container in our Kubernetes setup, deployed alongside the main Spring Boot service. It added maybe 200ms of overhead per evaluation call, which was fine since we were running evaluations asynchronously in a background job anyway. Here's the Python sidecar. Keep it small: # eval_service.py from flask import Flask, request, jsonify from ragas import evaluate from ragas.metrics import ( faithfulness, answer_relevancy, context_precision, context_recall, ) from datasets import Dataset app = Flask(__name__) @app.route("/evaluate", methods=["POST"]) def run_evaluation(): payload = request.json data = { "question": payload["questions"], "answer": payload["answers"], "contexts": payload["contexts"], "ground_truth": payload.get("groundTruths", [""] * len(payload["questions"])), } dataset = Dataset.from_dict(data) result = evaluate( dataset, metrics=[faithfulness, answer_relevancy, context_precision, context_recall], ) return jsonify(result) if __name__ == "__main__": app.run(host="0.0.0.0", port=8081) Run it: pip install ragas flask datasets langchain-openai python eval_service.py Now the Java side. I'm using Spring Boot with RestTemplate here, but you could just as easily use WebClient if you're on a reactive stack. First, the request and response models: // RagasEvaluationRequest.java import com.fasterxml.jackson.annotation.JsonProperty; import java.util.List; public class RagasEvaluationRequest { private List<String> questions; private List<String> answers; private List<List<String>> contexts; @JsonProperty("groundTruths") private List<String> groundTruths; // constructors, getters, setters public RagasEvaluationRequest( List<String> questions, List<String> answers, List<List<String>> contexts, List<String> groundTruths) { this.questions = questions; this.answers = answers; this.contexts = contexts; this.groundTruths = groundTruths; } public List<String> getQuestions() { return questions; } public List<String> getAnswers() { return answers; } public List<List<String>> getContexts() { return contexts; } public List<String> getGroundTruths() { return groundTruths; } } // RagasEvaluationResult.java import com.fasterxml.jackson.annotation.JsonProperty; public class RagasEvaluationResult { @JsonProperty("faithfulness") private double faithfulness; @JsonProperty("answer_relevancy") private double answerRelevancy; @JsonProperty("context_precision") private double contextPrecision; @JsonProperty("context_recall") private double contextRecall; // getters public double getFaithfulness() { return faithfulness; } public double getAnswerRelevancy() { return answerRelevancy; } public double getContextPrecision() { return contextPrecision; } public double getContextRecall() { return contextRecall; } } And the service that calls the sidecar: // RagasEvaluationService.java import org.springframework.stereotype.Service; import org.springframework.web.client.RestTemplate; import org.springframework.http.*; import java.util.List; @Service public class RagasEvaluationService { private final RestTemplate restTemplate; private final String ragasServiceU...