Bolting an API call onto your app is not AI engineering. There is a real gap between calling a model and building systems that reason, retrieve, and hold up in production. I break down what actually separates AI integration from AI engineering, and why that distinction matters for every team shipping with LLMs right n...
Last year I bolted an LLM call onto a resume parser and called it a feature. It worked, in the sense that it returned JSON most of the time and nobody complained during the demo. Six weeks later it was silently returning malformed skill lists for maybe one in twenty resumes, and I only found out because a candidate emailed us confused about why their Kubernetes experience showed up as "Kubernets" with a typo baked right into the extracted data. That's when it hit me: I hadn't built an AI feature. I'd wired a prompt to an API and hoped. That distinction — "I called an LLM API" versus "I engineered a system that happens to use an LLM" — is basically the whole conversation right now among backend teams. AI integration is what most of us did in 2023 and early 2024: grab a provider key, write a prompt, parse the response, ship it. AI engineering is what happens when that thing has to survive production traffic, fail gracefully, and not blow your token budget in a bad week. I've been through that transition more than once on ITJobOpportunities, once building the resume improvement pipeline and once reworking a candidate-assessment flow that kept timing out under load. Both times the lesson was the same: the model call is the easy 10%. What "integration" actually looked like Early on, my Job Fit feature — the "check your fit against this posting" card on the job detail page — was basically this: @PostMapping("/job-fit/check") public JobFitResult checkFit(@RequestParam MultipartFile resume, @PathVariable Long jobId) { String resumeText = pdfExtractor.extractText(resume); String prompt = buildPrompt(resumeText, jobId); String response = openRouterClient.complete(prompt); return objectMapper.readValue(response, JobFitResult.class); } Looks fine, reads fine, it's also a synchronous, single-provider, no-retry, no-timeout-handling problem waiting to happen. If the provider had a slow minute (and it did, more than once), the request just hung until the client gave up. If the model returned something that wasn't quite valid JSON, readValue threw and the candidate got a 500 with zero explanation. No fallback provider, no way to tell whether a bad result came from a bad prompt or a bad model day. This is what most "we added AI" announcements actually ship, and there's nothing wrong with starting here — you need to prove the idea works before investing in resilience. But if that code is still in production a year later, you've got a liability, not a feature. The shift: treating the model like an unreliable dependency The mental model that finally clicked for me: an LLM call is not a function call. It's closer to calling a flaky third-party gateway that sometimes takes forty seconds to respond and sometimes just gives you something wrong. Once I started treating it that way, the architecture decisions got obvious. For Job Fit, that meant moving to an async job with SSE as the primary transport and polling as a fallback for clients that can't hold a stream open. The endpoint that kicks off the check returns immediately with a job handle: @PostMapping("/job-fit/check") public ResponseEntity<JobFitJobHandle> submitFitCheck( @RequestParam MultipartFile resume, @PathVariable Long jobId) { String jobHandle = jobFitService.enqueue(resume, jobId); return ResponseEntity.accepted().body(new JobFitJobHandle(jobHandle)); } The scoring runs on a separate thread pool, writes progress to a small state table, and the frontend either subscribes to a stream or falls back to polling status every couple of seconds. That one change — decoupling the HTTP request from the model call — fixed most of my reliability headaches. Candidates stopped seeing timeouts, and when a provider had a rough afternoon, the job just took longer instead of failing outright. Fallback providers aren't optional anymore Something I didn't expect: model providers degrade more often than most backend APIs I've depended on in 25 years of doing this. Not constantly, but often enough that a single point of failure is a real risk once AI sits on your critical path. On ITJobOpportunities, resume skill extraction hits OpenRouter first and falls back to DeepSeek if the primary call fails or times out — a simple provider abstraction: public interface ContentAnalysisProvider { SkillExtractionResult extractSkills(String resumeText); } @Service public class ResilientSkillExtractor { private final List<ContentAnalysisProvider> providers; public SkillExtractionResult extract(String resumeText) { for (ContentAnalysisProvider provider : providers) { try { return provider.extractSkills(resumeText); } catch (ProviderUnavailableException e) { log.warn("Provider {} failed, trying next", provider.getClass().getSimpleName()); } } throw new AllProvidersExhaustedException(); } } Nothing fancy. But it turned "the resume upload feature is broken" into "the resume upload feature is a little slower today," which is a completely different support conversation. Evals: the part nobody wants to build I resisted this the longest. Writing evals for prompt outputs felt like busywork compared to shipping the next feature. I was wrong, and it cost me some embarrassing bugs before I admitted it. The candidate AI assessment feature in the ATS console — the one recruiters trigger manually on an application to get a client-ready summary — went through several prompt rewrites before it was usable. Each time, I'd tweak the prompt, eyeball a couple of outputs, and ship it. Then a recruiter flagged that an assessment for a strong candidate came back lukewarm because the resume mentioned "Kafka" in a context the model read as "familiar with" rather than "built production pipelines with." Small nuance, big difference in how a recruiter reads it. Now there's a small eval set — resume-job pairs with known expected outcomes (strong match, weak match, missing key skill) — and every prompt change gets run against that set before it ships. It's not fancy, just a script that calls the endpoint and diffs outputs against expected categories. It turned "does this feel better" into "did this regress the case we already know about." That's the difference between vibes and engineering, and it's a genuinely uncomfortable amount of extra work for a feature that otherwise felt done. Where I draw the line between fit check and assessment This is a product decision as much as a technical one, and worth calling out because it's easy to blur. Job Fit (public, on the job detail page) is instant, low-stakes, and candidate-facing — score, matched skills, missing skills, a short summary. We only save the candidate record if there's an email on the resume, and recruiters never see fit-check-only leads. It's meant to help candidates self-select, not to feed the pipeline. AI assessment (ATS, recruiter-triggered) is manual, async, and written for a hiring manager who will actually read it and might forward it to a client. It has to be evidence-backed — skills grounded in the actual resume text, not vibes. Recruiters trigger it deliberately, one candidate at a time, because it's meant to support a decision, not automate one. Keeping those two flows completely separate — different endpoints, different data retention, different tone in the model output — was one of the better architecture calls I made on this platform. Mixing them would've been faster to build and much worse to operate. A candidate doing a casual fit check on a Tuesday night shouldn't end up in a recruiter's pipeline as a "lead." That's a trust problem more than a technical one, but it shapes the code. Prompt engineering is real engineering now, whether you like the term or not I used to roll my eyes a bit at "prompt engineering" as a title. I've softened on that, mostly because I've seen how much a badly structured prompt costs in retries, token spend, and garbage output. A few t...