Every pipeline works fine until production sends it something it wasn't built for. I put together the patterns I actually rely on to keep data flowing when things break, not just the theory that looks good in a design doc
Pipelines are deceptively simple on a whiteboard. Pull data, reshape it, put it somewhere. Three boxes and two arrows. Then you ship it, and reality shows up: schemas drift, upstream systems go down for maintenance nobody announced, volume triples during a marketing push, some record has a null where your code assumed a value would always exist. I've built and rebuilt enough of these over 25 years, mostly on the JVM, Kafka-based pipelines, Spring Batch jobs, event-driven consumers feeding PostgreSQL, to have opinions about what actually holds up. This isn't a theory piece. It's the stuff that kept me from getting paged, and a few things I learned the hard way. Assume the Schema Will Change, Because It Will Early in my career I treated schemas as fixed contracts. Nice idea, wrong almost every time in practice, especially when the upstream system is owned by another team or an external vendor you don't have a Slack channel with. On a healthcare microservices program, we had event producers publishing to Kafka topics that fed downstream reporting pipelines. One team added an optional field to their Avro schema. Should have been a non-event. But a downstream consumer had strict schema validation with no default-value handling, and the whole pipeline stalled until someone manually bumped the schema registry compatibility mode. The fix wasn't complicated, just disciplined: Use backward-compatible schema evolution as a rule, not an afterthought. With Avro or Protobuf, new fields get defaults, and you never remove a field without a deprecation window. Enforce this at the schema registry level , Confluent Schema Registry or AWS Glue Schema Registry, not just in code review. Code review misses things. A registry with BACKWARD or FULL compatibility checks configured rejects the bad change outright. If you're consuming from a source you don't control, a third-party API, a partner feed, validate incoming payloads against a schema you define, not one they define. Fail loud on unexpected shapes rather than silently coercing nulls. Here's the pattern I use in Spring Boot for validating incoming records before they hit the transform stage, with Jakarta Bean Validation on a record type: public record InboundRecord( @NotBlank String userId, @NotBlank String eventType, Double amount ) {} @Component public class RecordParser { private final ObjectMapper objectMapper; private final Validator validator; public RecordParser(ObjectMapper objectMapper, Validator validator) { this.objectMapper = objectMapper; this.validator = validator; } public Optional<InboundRecord> parse(Map<String, Object> raw) { try { InboundRecord record = objectMapper.convertValue(raw, InboundRecord.class); Set<ConstraintViolation<InboundRecord>> violations = validator.validate(record); if (!violations.isEmpty()) { logSchemaViolation(raw, violations); return Optional.empty(); } return Optional.of(record); } catch (IllegalArgumentException ex) { logSchemaViolation(raw, ex); return Optional.empty(); } } } It returns Optional.empty() instead of throwing. That distinction matters more than people think. More on that in the dead-letter section. Idempotency Is Not Optional I used to think idempotency was a "nice to have if you have time" concern. Then a retry storm during a Kinesis throttling event caused a consumer to reprocess the same batch of records four times, and we ended up with duplicate charge records in a billing table. That was a fun one to explain. Here's the thing about distributed pipelines: retries happen. Networks blip, consumers crash mid-batch, brokers rebalance partitions. If your write operations aren't idempotent, every one of those normal, boring failure modes turns into a data quality incident. The pattern I default to now: every record carries, or can derive, a deterministic key, and every downstream write is an upsert, not a blind insert. In a Spring Boot service that usually means wrapping the SQL in a repository or JdbcTemplate call rather than letting an ORM's default save behavior decide for you: @Repository public class OrderWriter { private final JdbcTemplate jdbcTemplate; public OrderWriter(JdbcTemplate jdbcTemplate) { this.jdbcTemplate = jdbcTemplate; } public void upsert(String orderId, String status, Instant updatedAt) { jdbcTemplate.update(""" INSERT INTO orders (order_id, status, updated_at) VALUES (?, ?, ?) ON CONFLICT (order_id) DO UPDATE SET status = EXCLUDED.status, updated_at = EXCLUDED.updated_at WHERE orders.updated_at < EXCLUDED.updated_at """, orderId, status, Timestamp.from(updatedAt)); } } That WHERE clause matters too. Without it, you can get out-of-order writes clobbering newer data with stale data, its own quiet nightmare. I got burned by that exact thing on a Kafka consumer that processed partitions out of strict order during a rebalance. Nothing crashed. A dashboard just showed wrong numbers for about a day before anyone noticed. Dead-Letter Queues Aren't a Nice-to-Have Early pipelines I built let a bad record throw an exception and kill the batch. One malformed record took down processing for everything behind it in the queue. The fix is boring and effective: route anything that fails validation, parsing, or transformation into a dead-letter queue instead of failing the whole job. With Kafka this is usually a separate topic. With SQS it's a built-in DLQ configuration on the queue itself. With Spring Kafka, the wiring is a few lines: @KafkaListener(topics = "orders-inbound") public void consume(ConsumerRecord<String, String> record) { try { OrderEvent event = objectMapper.readValue(record.value(), OrderEvent.class); processOrder(event); } catch (Exception ex) { kafkaTemplate.send("orders-inbound-dlq", record.key(), record.value()); log.warn("Routed record to DLQ, offset={}", record.offset(), ex); } } A few things I'd add, and this is where opinions come in: Don't just dump and forget. A DLQ with 40,000 unread messages sitting there for three weeks is worse than no DLQ, because it gives false confidence that "we handle failures gracefully" when really you've just moved the pile somewhere less visible. Alert on DLQ depth , not just on pipeline failures. Depth growing steadily tells you something upstream is degrading before it becomes a full outage. Build a lightweight replay tool. Even a small Spring Boot batch job or a @Scheduled admin endpoint that reads the DLQ and reprocesses records after a fix ships is enough. You don't need a fancy UI on day one. Backpressure and Volume Spikes Volume spikes rarely look like a failure. Everything's green on the dashboard, then three hours later you notice consumer lag on a Kafka topic has climbed to millions of messages because upstream had a burst you weren't ready for. I ran into this on a lending platform's microservices setup, where a marketing campaign drove a multiple-times spike in application submissions over about 40 minutes. Our consumer group was fine on paper, correct partition count, reasonable consumer instances, but the downstream service it called for scoring had a rate limit we hadn't accounted for. The queue backed up not because Kafka couldn't keep up, but because the thing on the other end of our call couldn't. What actually helped: Decouple ingestion from processing speed. Kafka is good at this by design; you can accept messages fast even if your consumers process slowly, as long as you're honest about lag and alerting on it. Add backpressure-aware consumers . With Spring Cloud Stream or Reactor, you can control concurrency and prefetch explicitly instead of letting the framework grab everything at once: spring: cloud: stream: b...