Testing AI behavior in Java has always felt like testing nothing, no assertions that hold, no output you can trust twice. I dug into Dokimos to see if spec-based testing can finally give us something solid to grab onto. Sharing my notes on what worked and where it still falls short.
Dokimos is a Java testing framework built specifically for evaluating LLM outputs, prompts, and AI-driven behavior, the kind of thing that used to live entirely in Python land with tools like promptfoo or DeepEval. If you've ever tried to bolt a Python eval script onto a Spring Boot service just to sanity-check an LLM call, you know how awkward that gets. Different runtime, different CI pipeline, and suddenly someone on the team who "knows Python enough" becomes the bottleneck. What I like about Dokimos, and this is really the whole pitch, is that it treats prompt and model testing as a first-class JUnit citizen. You write assertions, but instead of checking equality you're checking things like semantic similarity, structural conformance, or whether a response satisfies a rubric scored by another model. Assertions for fuzzy outputs. That's the gap it fills. I'll be honest: my first reaction was skepticism. I've seen plenty of "AI testing frameworks" that are really a thin wrapper around calling an LLM twice and hoping for the best. Dokimos isn't that, but it's also not magic. You still have to think carefully about what you're actually testing. Why This Matters More Than It Looks Like It Does On the ITJobOpportunities backend, our Job Fit Check feature scores a candidate's resume against a job posting and returns matched skills, missing skills, and a short LLM-written summary. That summary hits real candidates on a public page. If the model starts hallucinating a skill that isn't in the resume, or the tone drifts somewhere condescending (it happened once, briefly, during a prompt tweak I made too late in the evening), that's a trust problem, not a bug ticket. Traditional unit tests verify the pipeline: did we call the resume parser, did we get a score between 0 and 100, did we save the candidate record only when an email was found. All deterministic, all testable the normal way. But the actual content quality of the LLM summary lives in a gray zone that regular JUnit assertions can't reach. This is where I think a lot of teams get lazy. They test the plumbing and skip testing the water quality. Then months later someone notices an AI feature has quietly degraded because a model provider changed something upstream, and nobody caught it because nothing was watching for it. Setting It Up Dokimos plugs into an existing JUnit 5 setup, which is honestly the main reason I gave it a real shot instead of just bookmarking the repo. Here's the actual dependency setup I'm running in the jobs-posting test module (check the version tags yourself, this ecosystem moves fast and I don't want you copy-pasting a stale one): // build.gradle repositories { mavenCentral() maven { url 'https://repo.dokimos.io/releases' } } dependencies { testImplementation platform('org.junit:junit-bom:5.10.2') testImplementation 'org.junit.jupiter:junit-jupiter' testImplementation 'io.dokimos:dokimos-junit5:0.9.2' testImplementation 'io.dokimos:dokimos-assertions:0.9.2' testImplementation 'io.dokimos:dokimos-rubric:0.9.2' testRuntimeOnly 'org.junit.platform:junit-platform-launcher' } tasks.named('test') { useJUnitPlatform { excludeTags 'llm-judge-nightly' } systemProperty 'dokimos.provider.deepseek.apiKey', System.getenv('DEEPSEEK_API_KEY') } That excludeTags line matters more than it looks like it does. I don't want the judge-model tests running on every PR, so I tag them separately and give them their own Gradle task for the nightly pipeline. More on that in a minute. Before I show the test itself, it's worth showing what senior-java-dev-resume.txt actually contains, since every test in this article loads it. When I first built this fixture I wrote a fictional candidate from scratch, and it worked fine, but then I realized there was a better option sitting right in front of me: my own resume, genericized. It isn't anyone's PII, and because it's my own career, I already know exactly what's true and what isn't. If the model invents a skill I don't have or fabricates an employer, I catch it in about two seconds instead of squinting at a stranger's fake job history trying to remember what I made up: Jose A. Aleman Senior Java Developer / Technical Lead / Architect SUMMARY Software engineer with 25+ years building Java and Spring Boot backends, distributed systems, and cloud-native platforms across FinTech, healthcare, and public-sector programs. Strong background in microservices decomposition, event-driven integration, and API design for high-availability systems. EXPERIENCE Founder | Job platform for technical hiring | 2024 - Present - Built resume parsing, skills extraction, and AI-assisted job matching for a recruitment platform serving candidates and recruiters across Costa Rica and LATAM. - Designed a Spring Boot API serving two separate React frontends through a single backend with runtime configuration injection. Senior Java Developer | Disclosure platform program | 2023 - 2025 - Maintained Java services supporting regulatory filing workflows for enterprise clients. - Contributed to a parallel cost-reporting integration for a separate client program. Senior Java Developer | Hospital microservices rewrite | 2022 - 2023 - Decomposed legacy healthcare services into AWS-hosted microservices using Kinesis for event streaming and Istio for service mesh traffic management. Senior Java Developer | FinTech and public-sector platform | 2017 - 2022 - Built Kafka-based event pipelines with Debezium change-data-capture feeding downstream services on EKS. - Integrated Keycloak for centralized authentication across a multi-team platform. Technical Lead | Lending microservices program | 2017 - Led a small team building lending microservices deployed on AWS and Rancher. SKILLS Java, Spring Boot, Spring Cloud, Kafka, Kinesis, PostgreSQL, AWS, Kubernetes/EKS, Docker, Istio, Keycloak, Liquibase, REST API design, distributed systems, technical leadership, mentoring EDUCATION (none listed) I left the education section blank on purpose. That's actually accurate to my real profile, and it's a small but useful edge case: the summary generator should never invent a degree just because a template expects one. A basic test, closer to what actually sits in our repo today: package com.josalero.posting.resume; import io.dokimos.junit5.DokimosExtension; import io.dokimos.assertions.DokimosAssertions; import io.dokimos.rubric.Rubric; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.extension.ExtendWith; import org.junit.jupiter.api.Tag; @ExtendWith(DokimosExtension.class) class ResumeSummaryQualityTest { private ResumeService resumeService; private String seniorJavaResume; @BeforeEach void setUp() { resumeService = new ResumeService(new OpenRouterClient(), new SkillExtractor()); seniorJavaResume = TestFixtures.load("senior-java-dev-resume.txt"); } @Test void resumeSummaryShouldMentionSeniorityLevel() { String summary = resumeService.generateSummary(seniorJavaResume); DokimosAssertions.assertThat(summary) .semanticallyContains("25+ years of experience") .doesNotContain("junior") .doesNotContain("entry-level") .isBetween(40, 120, WordCounter::countWords); } @Test @Tag("llm-judge-nightly") void resumeSummaryShouldPassToneRubric() { String summary = resumeService.generateSummary(seniorJavaResume); Rubric toneRubric = Rubric.builder() .criterion("Tone is professional and neutral, not exaggerated") .criterion("No fabricated job titles, employers, or degrees not present in the source resume") .criterion("Does not use superlatives like 'best', 'world-class', or 'guru'") .minScore(0.8) .build(); DokimosAssertions.assertThat(summary) .satisfiesRubric(tone...