Most teams rush to add AI to their projects without thinking about structure. I learned that the hard way. Here's exactly how I set up AI Harness Engineering in my projects, what worked, and what I'd do differently
I'd heard the term "AI harness engineering" thrown around but hadn't taken it seriously. I thought it was one of those things consultants put in slide decks. Turned out it was exactly what we needed, and setting it up properly changed how I think about building any feature that touches a model. What "AI Harness" Actually Means in Practice There's no single canonical definition, so let me tell you what it means on our team: a harness is the scaffolding you build around your AI calls so you can test, observe, evaluate, and iterate on them without flying blind. It's the difference between "I think this prompt is better" and "this prompt scores 12% higher on factual accuracy across our eval set." Not glamorous. But genuinely important work. The harness typically covers four things: Input/output capture: logging what goes in and what comes out, with enough metadata to reproduce any call Evaluation: some mechanism, automated or semi-automated, for scoring outputs Prompt versioning: treating prompts like code (because they are) Regression testing: knowing when a model update or prompt change breaks something I'll walk through how we set each of these up, with the actual tools and code patterns we use. We run a Java shop, so everything here is Spring AI on Java 25. Step 1: Capture Everything The first thing I did on our billing summarization service was wrap every model call in a structured logger. Before that, we were just printing the response to stdout like amateurs (okay, I'm exaggerating a bit, but not by much). We use Spring AI 1.x with the OpenAI integration, and our calls go through a thin service wrapper that handles logging, error tracking, and token metadata. Java 25's records make the call record structure clean and immutable: // AICallRecord.java package com.example.harness; import java.time.Instant; import java.util.List; import java.util.UUID; public record AICallRecord( String callId, String promptVersion, String model, List<MessageRecord> inputMessages, String rawOutput, double latencyMs, int promptTokens, int completionTokens, String error, String contextId ) { public static AICallRecord empty(String promptVersion, String model, String contextId) { return new AICallRecord( UUID.randomUUID().toString(), promptVersion, model, List.of(), null, 0.0, 0, 0, null, contextId ); } public record MessageRecord(String role, String content) {} } // HarnessService.java package com.example.harness; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.ai.chat.client.ChatClient; import org.springframework.ai.chat.messages.SystemMessage; import org.springframework.ai.chat.messages.UserMessage; import org.springframework.ai.chat.model.ChatResponse; import org.springframework.ai.chat.prompt.Prompt; import org.springframework.stereotype.Service; import java.time.Instant; import java.util.List; @Service public class HarnessService { private static final Logger log = LoggerFactory.getLogger(HarnessService.class); private final ChatClient chatClient; private final CallRecordRepository repository; public HarnessService(ChatClient.Builder builder, CallRecordRepository repository) { this.chatClient = builder.build(); this.repository = repository; } public AICallRecord callWithHarness( String systemPrompt, String userContent, String promptVersion, String contextId ) { var messages = List.of( new SystemMessage(systemPrompt), new UserMessage(userContent) ); var inputRecords = List.of( new AICallRecord.MessageRecord("system", systemPrompt), new AICallRecord.MessageRecord("user", userContent) ); long start = System.currentTimeMillis(); String rawOutput = null; String error = null; int promptTokens = 0; int completionTokens = 0; try { ChatResponse response = chatClient .prompt(new Prompt(messages)) .call() .chatResponse(); rawOutput = response.getResult().getOutput().getContent(); var usage = response.getMetadata().getUsage(); promptTokens = usage.getPromptTokens().intValue(); completionTokens = usage.getGenerationTokens().intValue(); } catch (Exception e) { error = e.getMessage(); log.error("AI call failed: contextId={}, error={}", contextId, e.getMessage()); } double latencyMs = System.currentTimeMillis() - start; var record = new AICallRecord( java.util.UUID.randomUUID().toString(), promptVersion, "gpt-4o-mini", inputRecords, rawOutput, latencyMs, promptTokens, completionTokens, error, contextId ); log.info("AI call completed: callId={}, version={}, latencyMs={}, contextId={}", record.callId(), promptVersion, latencyMs, contextId); repository.saveAsync(record); return record; } } We persist these records to a Postgres table via a background task, nothing fancy, but having a searchable history of every call, with the exact input, output, model version, and latency, is invaluable when something goes sideways in production. The contextId field, tied to the originating support ticket ID, was something I added later and wish I'd included from day one. Being able to pull up every AI call associated with a specific ticket completely changed how we debug issues. Small addition, huge quality-of-life improvement. Storing every prompt input and output does raise data retention questions. We anonymize ticket content before it hits the log table. Worth thinking about early, not as an afterthought. Step 2: Version Your Prompts Like Code This one sounds obvious. It isn't obvious in practice. For the first two months, we had prompts living as string constants scattered across service classes. One person would edit them, push, and the rest of the team had no idea what changed. No rollback, no history, no accountability. A mess, honestly. Now we store prompts in a prompts/ directory with versioned YAML files, loaded at startup by a PromptRegistry bean: # prompts/ticket_summary_v3.yaml version: "3" name: ticket_summary created_at: "2024-11-14" author: "dana@company.com" description: "Tightened extraction rules for priority field, added JSON schema enforcement" system: | You are a support operations assistant. Given a customer support ticket, extract the following fields and return them as valid JSON: - summary: a one-sentence description of the issue (max 25 words) - priority: one of [low, medium, high, critical] - category: one of [billing, technical, account, other] - requires_escalation: boolean Return only the JSON object. No explanation, no markdown fences. user_template: | Ticket content: {ticket_content} // PromptRegistry.java package com.example.harness; import org.springframework.core.io.Resource; import org.springframework.core.io.support.PathMatchingResourcePatternResolver; import org.springframework.stereotype.Component; import org.yaml.snakeyaml.Yaml; import jakarta.annotation.PostConstruct; import java.io.InputStream; import java.util.Map; import java.util.concurrent.ConcurrentHashMap; @Component public class PromptRegistry { // key: "name:version", e.g. "ticket_summary:3" private final Map<String, Map<String, Object>> registry = new ConcurrentHashMap<>(); @PostConstruct public void load() throws Exception { var resolver = new PathMatchingResourcePatternResolver(); var yaml = new Yaml(); for (Resource r : resolver.getResources("classpath:prompts...