Model-independent evaluation harness

Build a model-independent eval harness for Java coding agents with Spring Boot: same tasks, same commit, deterministic Gradle or Maven tests, and a per-task model-selection policy as the output.

Every few weeks a new model tops a coding leaderboard, and someone on the team asks if we should switch. I used to answer with an opinion. Now I think the question itself is wrong. "Which model is best at coding" is a leaderboard question. The question a team actually has is narrower and more useful: which model should handle which kind of task in our repos, at what cost, with how much cleanup afterward, and without shipping something unsafe? No public benchmark answers that. So the experiment for this week, starting today, November 16, 2026, is to build the thing that does: a model-independent evaluation harness for Java coding agents. A Spring Boot service submits the same tasks to five current models, validates every result with deterministic Gradle or Maven tests, and produces a model-selection policy. Not a winner. A policy. Why public benchmarks do not transfer to my repos Public coding benchmarks are useful signals. They are not decisions. Language mix. The best-known agent benchmarks lean heavily on Python repositories. My work is Java 25 and 26, Spring Boot, Gradle and Maven, JPA, and PostgreSQL. A model that is great at patching a Python library can still stumble on a Spring context that refuses to start. Repo shape. Real services have conventions, custom starters, Testcontainers setups, and build plugins that fail the build for reasons unrelated to the change. Benchmarks rarely include that friction. Definition of solved. "Tests pass" is necessary. It is not sufficient. A diff that passes but needs twenty minutes of human cleanup is a different outcome than one that merges as is. Contamination. Popular public repos and their fixes may already sit in training data. Private fixtures do not. A leaderboard tells you who won someone else's exam. Your repo is the exam that matters. The experiment in one paragraph Pick 30 to 50 tasks from real Java work. Freeze each one at a specific commit. Give every model the same prompt, the same tools, the same budget, and the same starting repository state. Let each model produce a diff. Then throw away everything the agent says about its own work and run the build. A shared validator, outside the agent's sandbox, decides pass or fail. Repeat each task a few times per model. Collect seven metrics. Turn the results into a routing table per task type. The model writes the diff. The build decides if it worked. The lineup, as of November 16, 2026 The model list will be stale in a month. That is the point of building a harness instead of a ranking. For this run, I picked five models that cover the choices a Java team actually has today: two price tiers from one vendor, the default model of a major coding CLI, a strong long-context option, and one model I can run on my own hardware. Model Vendor Why it is in the run Claude Opus 5.5 Anthropic The long, hard-run option. It costs about twice as much per token as Sonnet 5.5, so it has to earn that on hard tasks Claude Sonnet 5.5 Anthropic The mid-price default many teams already use for agentic coding GPT-6.1 Sol OpenAI The default model in Codex CLI; a common choice for test-driven fixes Gemini 4 Carbon Google Google's strongest model for long-running coding and agent work, released October 28 Qwen3.8-27B Alibaba (open weights, Apache 2.0) Fits on one GPU. The only lane for code that cannot leave our network I left Claude Fable 5.2 (released November 9) out of the first run on purpose. It is the most expensive model on the list, and a harness that starts with the priciest option never answers the cost question. It enters the second run only if no eligible model clears the bar on hard refactors. The harness The harness is a plain Spring Boot service. Nothing exotic. It has five parts: Task catalog. One manifest per task: fixture repo, base commit, build tool, prompt, visible tests, hidden tests, protected paths, time and tool budgets. Workspace factory. For every run, a fresh checkout of the fixture at the pinned commit. One run never sees another run's files. Model port. One interface, one adapter per model or agent runtime. The orchestrator does not know which provider is behind it. Validator. A separate container with a pinned JDK image, a warmed dependency cache, and no network. It applies the diff and runs ./gradlew test or ./mvnw verify . Results store. PostgreSQL rows per run: outcome, metrics, the diff, the build log, and the agent trace. flowchart LR T[Task catalog] --> O[Spring Boot orchestrator] O --> W[Fresh workspace at pinned commit] W --> A[Claude Opus 5.5] W --> B[Claude Sonnet 5.5] W --> C[GPT-6.1 Sol] W --> G[Gemini 4 Carbon] W --> Q[Qwen3.8-27B self-hosted] A --> D[Diff and run trace] B --> D C --> D G --> D Q --> D D --> V[Validator container, no network] V --> S[Security scan and diff guards] S --> R[(Results store)] R --> H[Blind human review] R --> P[Selection policy] H --> P The model port is the piece that keeps the harness model-independent. Each adapter hides the provider SDK, the agent loop, and the tool wiring. The orchestrator only sees what came back. public interface CodingAgent { String modelId(); AgentRun solve(TaskPrompt prompt, Workspace workspace, RunBudget budget); } public record RunBudget(Duration timeout, int maxToolCalls, long maxOutputTokens) {} public record AgentRun( String diff, int toolCalls, int failedToolCalls, long inputTokens, long outputTokens, Duration wallClock, StopReason stopReason) {} public enum StopReason { COMPLETED, TIMEOUT, TOOL_BUDGET_EXHAUSTED, REFUSED, ERROR } Spring AI and LangChain4j both put chat models behind one interface, which makes a direct-model adapter short. If you are evaluating full agent runtimes or CLIs instead of raw models, the adapter shells out to them and parses their trace. Either way, the contract is the same: a diff, a trace, and counters. The run loop fans out tasks × models × repeats on virtual threads, with a semaphore so the harness does not open fifty sandboxes at once or hit provider rate limits. public List<TaskResult> runAll(List<EvalTask> tasks, List<CodingAgent> agents, int repeats) throws InterruptedException { Semaphore permits = new Semaphore(maxConcurrentRuns); try (ExecutorService executor = Executors.newVirtualThreadPerTaskExecutor()) { List<Callable<TaskResult>> runs = new ArrayList<>(); for (EvalTask task : tasks) { for (CodingAgent agent : agents) { for (int attempt = 1; attempt <= repeats; attempt++) { int runNumber = attempt; runs.add(() -> { permits.acquire(); try { return runner.run(task, agent, runNumber); } finally { permits.release(); } }); } } } return executor.invokeAll(runs).stream().map(this::resultOrFailure).toList(); } } The validator is the part I trust most, because it does not involve a model at all. public ValidationResult validate(Workspace workspace, EvalTask task) throws IOException, InterruptedException { List<String> buildCommand = switch (task.buildTool()) { case GRADLE -> List.of("./gradlew", "--offline", "--no-daemon", "clean", "test"); case MAVEN -> List.of("./mvnw", "-o", "-B", "clean", "verify"); }; List<String> command = new ArrayList<>(List.of( "docker", "run", "--rm", "--network", "none", "-v", workspace.path() + ":/work", "-w", "/work", validatorImage)); command.addAll(buildCommand); Process process = new ProcessBuilder(command) .redirectErrorStream(true) .redirectOutput(workspace.buildLog().toFile()) .start(); if (!process.waitFor(task.validationTimeout().toMillis(), TimeUnit.MILLISECOND...