What I've Learned Building Systems That Actually Have to Be Accountable
The ATS story is one I bring up a lot when I'm trying to convince engineering leaders to take this seriously earlier. Several major employers faced legal scrutiny over AI-driven applicant tracking systems that filtered out candidates in ways that correlated with protected characteristics. The vendors had built capable systems. They just hadn't built accountable ones. That distinction ended up costing some of those companies millions in settlements and forced product redesigns that would have been a fraction of the price to build correctly from the start. So when I talk about governance, I'm not talking about something abstract or theoretical. I'm talking about the difference between a product that can be defended and one that can't. Governance isn't a compliance checkbox. It's an engineering problem. I think a lot of teams still approach AI safety and governance the way they approach cookie consent banners. Something legal asked for, something you implement minimally, something that lives in a PDF somewhere. That framing is going to cause real damage as AI gets embedded deeper into hiring, lending, healthcare, and anywhere else consequential decisions get made. The actual work of governance is technical. It's about auditability, drift detection, access control, output validation, and decision traceability. All of that has to be designed in. You can't retrofit it into a system that was built without it in mind, or at least you can, but it'll cost you those eight weeks I mentioned. Maybe more. What I've settled on, after a few projects, is thinking about governance across three layers: what goes into the model, what the model does at inference, and what happens after the model makes a decision. Layer 1: What goes in Data governance is where a lot of AI safety problems actually start, even though we tend to blame the model. If your training data has a skewed representation of some demographic group, no amount of post-hoc fairness patching fixes that cleanly. This is exactly where the ATS failures began. The training sets used to rank candidates were often built from historical hiring data, which meant they encoded whatever biases existed in previous hiring decisions. The model wasn't doing anything mysterious. It learned that candidates who looked like past hires ranked higher, and past hires skewed heavily in ways that correlated with protected characteristics. Then it reproduced that pattern at scale, automatically, across every application. The tools I reach for during dataset construction: Great Expectations for automated data quality checks on incoming training batches. The ability to define expectations declaratively and fail a pipeline run when they're violated is genuinely useful. Evidently AI for tracking feature distributions over time. When the distribution of your input features drifts from what the model was trained on, you want to know before users do. Demographic parity audits during dataset construction, not after training. This one gets skipped constantly, and I get why, it's not glamorous work, but it matters. For an ATS pipeline specifically, the data quality checks need to account for the fact that resume text is messy and that certain proxy features (school names, zip codes, employment gaps) can act as protected characteristic proxies even when no explicitly protected fields are present. Here's a simplified version of what that looks like using Great Expectations: import great_expectations as ge import pandas as pd # Load a batch of parsed resume records destined for training df = ge.read_csv("resume_training_batch_2024_q3.csv") # Basic field completeness df.expect_column_values_to_not_be_null("candidate_id") df.expect_column_values_to_not_be_null("years_of_experience") df.expect_column_values_to_not_be_null("highest_degree") # Years of experience should be in a plausible range df.expect_column_values_to_be_between("years_of_experience", min_value=0, max_value=50) # Reject records where protected-class proxies have leaked into features. # Employment gap in months: present but shouldn't be used as a ranking signal # Flag if it's been one-hot encoded or otherwise featurized df.expect_column_to_not_exist("employment_gap_months_encoded") df.expect_column_to_not_exist("graduation_year") # age proxy # Verify that outcome labels (historical hire decisions) aren't # perfectly correlated with a single institution name, which would # indicate the training set encodes school-based bias results = df.validate() if not results["success"]: raise ValueError( "Training data failed quality checks. " "Review flagged columns before proceeding. Aborting pipeline." ) It's basic stuff, but "basic" is doing a lot of work in a domain where most teams skip it entirely. Layer 2: What happens at inference This is where things get more interesting and, honestly, more contested. There's a real tension between model capability and output control. You want the model to be useful, which usually means giving it some latitude. But in regulated domains, that latitude can produce outputs that are inconsistent, hallucinated, or legally problematic. In an ATS context, that means ranking rationales that either can't be explained in terms of lawful selection criteria or that a plaintiff's attorney could use to demonstrate disparate impact. Here's a simplified version of how I'd structure the inference layer for a candidate screening tool, using Spring AI since a lot of enterprise ATS vendors are running Java shops: import org.springframework.ai.chat.client.ChatClient; import org.springframework.ai.converter.BeanOutputConverter; import org.springframework.stereotype.Service; @Service public class CandidateScreeningService { private final ChatClient chatClient; public CandidateScreeningService(ChatClient.Builder builder) { this.chatClient = builder.build(); } public CandidateScreeningDecision evaluateCandidate(CandidateSummary candidate) { var outputConverter = new BeanOutputConverter<>(CandidateScreeningDecision.class); String systemPrompt = """ You are a candidate pre-screening assistant. Based on the applicant data provided, return a structured screening decision. Valid decisions are: ADVANCE, DECLINE, HOLD. You must include a plain-language rationale that references only job-relevant qualifications: skills match, years of relevant experience, and required certifications. Do not reference institutions, location, employment gaps, or graduation dates. {format} """; String userPrompt = """ Role applied for: {roleTitle} Required skills: {requiredSkills} Candidate's listed skills: {candidateSkills} Years of relevant experience: {yearsExperience} Required certifications: {requiredCerts} Candidate certifications held: {candidateCerts} """; String response = chatClient.prompt() .system(s -> s.text(systemPrompt) .param("format", outputConverter.getFormat())) .user(u -> u.text(userPrompt) .param("roleTitle", candidate.roleTitle()) .param("requiredSkills", candidate.requiredSkills()) .param("candidateSkills", candidate.candidateSkills()) .param("yearsExperience", candidate.yearsOfRelevantExperience()) .param("requiredCerts", candidate.requiredCertifications()) .param("candidateCerts", candidate.heldCertifications())) .call() .content(); CandidateScreeningDecision decision = outputConverter.convert(response); validateDecision(decision); return decision; } private void validateDecision(CandidateScreeningDecision decision) { if (decision.outcome() == null) { throw new IllegalStateException("Model returned null scr...