Schedulers still execute. Agents reason. What actually changes when you put a decision loop on top of ETL, ELT, and EtLT without ripping out your DAGs.
I spent a good chunk of my early career writing shell scripts that kicked off SQL jobs that fed into scheduling tools somebody else babysat overnight. Nobody on that pipeline needed to understand why the finance team wanted a rolled-up revenue number by 6 a.m. They just needed the job to finish before the dashboard refreshed. That was the deal for twenty-plus years. Engineers did the thinking, wrote the steps, and the system just executed them, reliably, dumbly, on schedule. That deal is starting to break down, and I don't think enough people in data engineering have sat with what that actually means yet. Three contracts under the same morning deadline Finance still wants a rolled-up number by 6 a.m. What changes is who is allowed to decide the next step when the input is messy. ETL / ELT / EtLT A person compiles the plan. The scheduler executes it. Predictable, auditable, weak on surprises. Agent-augmented Keep the deterministic backbone. Add reasoning only where fixed logic keeps failing. Evals required. Fully agent-designed Interesting to prototype. Not what I would put unsupervised in front of a compliance-sensitive flow. The old contract: humans think, systems execute ETL, ELT, and the hybrid EtLT that showed up a few years back all share one assumption: a person understands the business context, breaks the requirement into discrete steps, and a scheduler (Airflow, Control-M, cron if you're feeling nostalgic) runs those steps on a clock or a trigger. The difference between the three patterns is really just where in the pipeline the human's transform logic gets executed. Classic ETL looks like this: pull the rows out of the source, transform them in application code before they ever touch the warehouse, then load the finished result. The three patterns only disagree about where that transform sits. Figure · where the transform lives Green blocks are where the business logic runs. The scheduler still does not understand it. // Classic ETL: the transform happens before the data touches the warehouse public void runNightlyRevenueJob() { List<OrderRecord> rawOrders = sourceDb.extract( "SELECT order_id, customer_id, amount, currency, created_at " + "FROM orders WHERE created_at >= ?", yesterday() ); Map<String, BigDecimal> revenueByRegion = rawOrders.stream() .map(this::normalizeCurrency) .map(this::applyRegionalTaxRules) .collect(groupingBy(OrderRecord::getRegion, mapping(OrderRecord::getAmountUsd, reducing(BigDecimal.ZERO, BigDecimal::add)))); warehouse.load("fact_daily_revenue", toRevenueRows(revenueByRegion)); } ELT flips the order. Once storage and compute got cheap, it stopped making sense to transform on the way in. You dump the raw rows into the warehouse first and let a tool like dbt do the transform where the data already lives, with tests attached. -- ELT: load raw first, transform lives in the warehouse as a dbt model -- models/staging/stg_orders.sql with source as ( select * from raw.orders ) select order_id, customer_id, case when currency = 'EUR' then amount * 1.08 else amount end as amount_usd, date_trunc('day', created_at) as order_date from source where created_at is not null # models/staging/schema.yml models: - name: stg_orders columns: - name: order_id tests: [unique, not_null] - name: amount_usd tests: [not_null] EtLT sits in between the two. Sometimes you cannot afford to load everything raw, masking a customer's email or coercing a broken timestamp type has to happen before the row lands anywhere, so you do a light transform on the way in and leave the heavier logic for a dbt model after load. // EtLT: a light transform before load, heavier transform after public void loadOrdersToWarehouse(List<OrderRecord> rawOrders) { List<OrderRecord> prepped = rawOrders.stream() .map(order -> order.withCustomerEmail(maskEmail(order.getCustomerEmail()))) .map(this::coerceTimestampType) .toList(); warehouse.copyInto("raw.orders", prepped); // currency normalization and tax rules happen later, in dbt } None of these three patterns ever ask the system to understand anything. They just move where in the pipeline the "understanding" gets baked in. The scheduler doesn't know what "monthly active users" means and doesn't need to. It runs task A, then task B, then task C, and pages someone if task B times out. This worked because the hard part, the actual business logic, got compiled down into code and SQL ahead of time by a human who understood the domain. The system's job was execution, not comprehension. That separation made things predictable. You could reason about failure modes, write a runbook, and know that a 3 a.m. break was either schema drift, a network blip, or someone dropped a column in the source system again. Where the cracks started showing I noticed the strain first on the transform side, not the orchestration side. dbt's dependency graph and testing model genuinely made ELT transformations more maintainable than a hand-rolled DAG of SQL scripts. I still recommend it. But even dbt assumes someone writes the model. A human decides what a staging table should look like and writes the not_null and unique tests you saw above. The tool is smarter about dependency resolution than a stack of cron jobs, but it's still executing a plan a person authored in advance. EtLT tried to patch a real gap: sometimes you need that light transform before load because loading everything raw and dealing with it later isn't practical, whether that's sensitive-field masking, deduping, or basic type coercion. Fine. Still the same contract, though. Someone wrote the rule. The system just runs it. What agents actually change Here's the part that's genuinely different, and I want to be precise about it because "AI agents in data engineering" has become one of those phrases that gets thrown around without anyone defining what changed underneath. An agent, in the sense I mean it, doesn't execute a pipeline you wrote. It reasons about a goal you gave it, decides which steps to take, and adjusts when something doesn't go as expected. The scheduler used to be dumb by design. Now there's a layer that can look at a broken step, read the error, and decide (not always correctly) what to try next. I ran into this building the resume and Job Fit pipelines on ITJobOpportunities. Resume parsing is basically a mini ETL problem: extract text from a document, transform it into structured skills and experience, load it into the candidate record. My first pass at this was a classic pipeline, extract text, send it to an LLM with a fixed prompt template, parse the response, write it to Postgres. Deterministic in structure, even though the model call itself was probabilistic. Real resumes are chaos, though. Two-column layouts, skills buried in a "Projects" section instead of a "Skills" section, exports that scramble the text. A fixed pipeline handles the well-behaved cases fine and quietly mangles the rest. What changed my thinking wasn't rewriting the extraction logic for every new edge case. It was giving that step a small amount of room to decide its own next move: if the first extraction attempt comes back thin on skills, try a different chunking or reclassification strategy before giving up, or fall back to a secondary provider instead of failing outright, which is close to what already happens under the hood when a primary LLM provider call doesn't return usable output. That's a contained example of agent behavior. Nothing dramatic. But the step stopped being "run this fixed prompt" and started being "pursue this goal, and here are a couple of fallbacks you're allowed to try." // simplified illustration, not production code CandidateProfile extractProfile(String resumeText) { CandidateProfile result = llmExtract(resumeText, Strategy.STRUCTURED_SECTIONS); if (result.getSkills...