Fact-Checking LLM Outputs Programmatically: Building a Verification Layer That Catches Hallucinations
Last month I was building a feature for an applicant tracking system. The idea was straightforward: feed candidate resumes and job descriptions into an LLM, get back a structured summary with relevant experience, skill matches, years in role, that kind of thing. Recruiters would use these summaries to triage a high-volume pipeline without reading every resume cold. The model produced clean, well-structured output. Specific tenure figures, named skills, match percentages. It looked exactly like what we needed. Except it wasn't. On one candidate, the model reported six years of Python experience. The actual resume said two. On another, it cited a certification the candidate never mentioned anywhere. The model just filled in what seemed plausible. Calmly. With complete confidence. No hedging, no "I'm inferring this", nothing. In a hiring context, that's not just a data quality problem. That's a liability. A candidate gets screened out because the LLM invented a skill gap. Or someone gets advanced because the model hallucinated a qualification. Neither outcome is acceptable, and "the model seemed confident" is not a defense you want to be making to an HR team. That was it for me. I stopped trusting LLM output on anything factual without a verification layer sitting in front of it. Why Hallucinations Are Worse Than You Think The frustrating thing isn't that LLMs are sometimes wrong. It's that they're wrong in the most convincing possible way. Bad outputs from a rule-based system look broken. A garbled JSON response, a null pointer, a stack trace, something that makes you stop and say "okay, clearly this failed." Bad outputs from an LLM look polished. A hallucinated tenure figure arrives in a grammatically perfect sentence, embedded in context that sounds like careful analysis. It doesn't trigger your "something's off" instinct the way a 500 error would. Your brain just... accepts it. And that's the real problem for production systems. If you're building anything where LLM output touches real decisions, real documents, or real users, you can't just eyeball it. You need a programmatic layer that checks the output before it goes anywhere. The approach I landed on has three loosely connected parts: claim extraction, grounding verification, and confidence scoring. None of them are magic. Together they catch probably 80% of the stuff that would have slipped through before. Okay, I'm estimating at 80%, but it's in that neighborhood. Step 1: Extract Claims From the Output The first thing you need is a way to pull discrete, verifiable claims out of whatever the model generated. Sounds simple. It's actually the hardest part to get right. I started by treating this as another LLM task (yes, using an LLM to check an LLM, I know). The key is that the extraction prompt is much simpler and less prone to hallucination than the original generation task. You're not asking it to reason or synthesize, just to identify. That distinction matters more than you'd expect. In the ATS context, the claims I care most about are numeric (years of experience, employment dates, team sizes), attributions (this person worked at Company X, holds Certification Y), and stated outcomes (led a project that reduced costs by 30%). Spring AI makes this reasonably clean to wire up because it handles prompt templating and structured output binding in one place, which cuts down on a lot of the boilerplate you'd otherwise write yourself. Here's how I set up the claim extraction step using Spring AI's ChatClient with a structured output converter: // Claim types we care about in the ATS context public record Claim( String claim, ClaimType claimType, String sourceHint, double confidence ) {} public enum ClaimType { NUMERIC, ATTRIBUTION, EVENT } public record ClaimExtractionResult(List<Claim> claims) {} @Service public class ClaimExtractionService { private final ChatClient chatClient; private static final String EXTRACTION_PROMPT = """ You are a fact extraction tool. Given the following text, extract every specific, verifiable factual claim. A verifiable claim contains: - A specific number, date, percentage, or quantity - An attribution to a named person, company, or organization - A stated skill, credential, certification, or role title - A stated event or outcome For each claim, identify: - claim: the exact claim as a string - claimType: one of NUMERIC, ATTRIBUTION, EVENT - sourceHint: where in the text this appears (e.g. "work history", "skills section") - confidence: your confidence this is a verifiable claim (0.0 to 1.0) TEXT: {resumeSummary} """; public ClaimExtractionService(ChatClient.Builder builder) { this.chatClient = builder.build(); } public List<Claim> extractClaims(String resumeSummary) { ClaimExtractionResult result = chatClient.prompt() .user(u -> u.text(EXTRACTION_PROMPT) .param("resumeSummary", resumeSummary)) .call() .entity(ClaimExtractionResult.class); return result != null ? result.claims() : List.of(); } } Spring AI's .entity() method handles the structured output binding automatically. Under the hood it generates a JSON schema from the record class and instructs the model to conform to it, which is the equivalent of setting response_format manually. You get type-safe deserialization without writing a custom parser. Worth it. The sourceHint field is something I added specifically for this ATS use case. When a claim fails verification later, knowing it supposedly came from the "skills section" versus the "work history" helps a recruiter find the discrepancy quickly. Small thing, but it saved a lot of back-and-forth on my team. One thing I learned the hard way: filter on claimType before sending anything to verification. Attribution claims are usually verifiable with a direct lookup against the resume text. Numeric claims need a different path. Event claims are harder, and sometimes not worth the effort depending on your use case. Step 2: Ground Each Claim Against a Source Once you have a list of claims, you need something to check them against. For an ATS, the source of truth is the resume itself, plus any structured data already parsed from it: employment dates, listed skills, education records. The LLM summary should be consistent with the source document. If it isn't, that's a hallucination. For numeric claims (tenure, dates, years of experience), I do a structured lookup against a parsed resume record and compare values within a tolerance band: @Service public class NumericClaimVerifier { private static final double TENURE_TOLERANCE = 0.10; // 10% for rounding like "5 years" vs 4.8 public VerificationResult verify(Claim claim, ResumeRecord resumeRecord) { ParsedClaimEntity parsed = claimEntityParser.parse(claim.claim()); if (parsed == null) { return VerificationResult.unverifiable("could not parse entity"); } Optional<Double> actualValue = resumeRecord.lookup( parsed.entity(), parsed.metric(), parsed.period() ); if (actualValue.isEmpty()) { return VerificationResult.unverifiable("no source data found"); } double actual = actualValue.get(); double claimed = parsed.value(); double diff = Math.abs(actual - claimed) / (actual != 0 ? actual : 1.0); if (diff <= TENURE_TOLERANCE) { return VerificationResult.verified(actual, claimed, diff); } else { return VerificationResult.failed( actual, claimed, diff, String.format("value differs by %.1f%%", diff * 100) ); } } } I bumped the tolerance to 10% specifically for tenure. People round constantly. "Five yea...