What if the best way to build resilient systems is to break them yourself? I spent time intentionally injecting failures into Spring Boot apps using Chaos Monkey, and the lessons were humbling. Here's everything I learned from breaking things on purpose.
Last year we were three weeks out from a major cutover on our order processing service, the one handling about 40k transactions a day, and our tech lead dropped a question in the team standup that made everyone go quiet: "How do we actually know this thing recovers when a dependency goes down?" We had unit tests. We had integration tests running against Testcontainers. We had a staging environment that was, generously speaking, about 60% representative of production. But none of that answered his question. We knew our happy path cold. The failure paths? Not so much. That's when I started properly looking at Chaos Monkey for Spring Boot (the library, not Netflix's original tool, though they share the same spirit). I'd heard the name before but always assumed it was overkill for anything short of a Netflix-scale distributed system. I was wrong about that. What the Library Actually Does Chaos Monkey for Spring Boot is a library from Codecentric. It plugs into your Spring Boot app and deliberately injects faults into your Spring-managed beans at runtime. The idea is borrowed from Netflix's original chaos engineering work, but instead of terminating EC2 instances, it attacks your service's internals: it can make methods throw exceptions, add artificial latency, or kill the JVM thread. The library works through Spring AOP. You add the dependency, enable it via a property flag, and then configure which types of beans it targets. It calls those targets "Watchers" and the things it does to them "Assaults." That naming convention is a bit dramatic, but it sticks. <dependency> <groupId>de.codecentric</groupId> <artifactId>chaos-monkey-spring-boot</artifactId> <version>3.1.0</version> </dependency> You'll also need Spring Boot Actuator, because the library exposes its control API through Actuator endpoints. spring: profiles: active: chaos-monkey chaos: monkey: enabled: true watcher: service: true rest-controller: true repository: true assaults: level: 5 latency-active: true latency-range-start: 2000 latency-range-end: 8000 exceptions-active: true exception: type: java.io.IOException arguments: - className: java.lang.String value: "Chaos Monkey: Simulated IO failure" That level property (1-10) controls how frequently assaults trigger. Level 5 means roughly every fifth request to a watched bean gets hit. I'd start at level 2 or 3 in a shared environment unless you enjoy explaining yourself to your team. The Watchers: Picking Your Targets By default the library can watch @Service , @Controller , @RestController , @Repository , and @Component beans. You can also target @Bean methods in configuration classes, which is useful when you have infrastructure beans like RestTemplate or WebClient instances you want to test around. On the order service project, we enabled watchers for @Service and @Repository only. The reasoning was pretty straightforward: the service layer is where all our business logic lives, and the repository layer is where we'd feel a database connectivity issue first. Attacking the REST controller layer felt too surface-level for what we were actually trying to learn. One thing I'd flag early: be careful with @Repository watchers if you're using Spring Data JPA and your transaction management is doing something unusual. We had one repository method that was supposed to be read-only, annotated with @Transactional(readOnly = true) , and when Chaos Monkey threw an IOException mid-transaction, we got rollback behavior we weren't expecting. Not a Chaos Monkey bug. Just something our code wasn't handling correctly. Worth it to find that in staging rather than at 2am on a Tuesday. Assaults: What You Can Actually Throw at Your App There are four main assault types. I'll be honest, I use two of them constantly and barely touch the other two. Latency assaults are my go-to. You configure a range (say, 2000ms to 8000ms) and the library adds a random sleep in that window before the method executes. This is fantastic for testing timeout configurations, circuit breaker thresholds, and whether your thread pools handle slow dependencies gracefully. We found two places in the order service where we were using a shared RestTemplate with no timeout set at all. Two. In a service that had been running for 18 months. Don't ask me how nobody caught that sooner. Exception assaults throw a configured exception from the targeted method. You pick the exception type and message. The default is RuntimeException , but you can configure anything throwable. I prefer to use something realistic, like org.springframework.dao.DataAccessException for repository-level attacks, because throwing a generic RuntimeException from a repository doesn't tell you much about how your app handles actual infrastructure failures. The specificity matters. KillApplication assaults call System.exit(1) . Yes, really. It just kills the process. I've only used this in a dedicated chaos testing environment where we were explicitly testing Kubernetes pod restart behavior and our readiness probe configuration. Don't enable this casually. Seriously. Memory assaults try to exhaust heap space by creating large objects. I've used this exactly once. The feedback loop is slow and the failure mode is pretty blunt (OutOfMemoryError is not that interesting to observe in isolation). Skip it unless you're specifically testing memory pressure scenarios. The Actuator API: Runtime Control This is the part I didn't appreciate until I started writing proper chaos test scenarios. The library exposes endpoints under /actuator/chaosmonkey that let you enable, disable, and reconfigure everything at runtime without restarting the application. # Check current status curl http://localhost:8080/actuator/chaosmonkey/status # Enable chaos monkey at runtime curl -X POST http://localhost:8080/actuator/chaosmonkey/enable # Update assault configuration curl -X POST http://localhost:8080/actuator/chaosmonkey/assaults \ -H 'Content-Type: application/json' \ -d '{ "level": 3, "latencyActive": true, "latencyRangeStart": 1000, "latencyRangeEnd": 5000, "exceptionsActive": false }' # Check watcher configuration curl http://localhost:8080/actuator/chaosmonkey/watchers Being able to flip this on and off via API means you can build it into your test automation. We wrote a small test harness that would spin up the service, hit the chaos endpoint to activate latency assaults, run a suite of integration scenarios, observe the circuit breaker metrics via Micrometer, then disable chaos and verify the service recovered cleanly. The whole thing ran in the CI pipeline against a staging deployment. Not in unit tests, to be clear. That's an important distinction, and I've seen people try to blur it. Real Scenario: Testing Resilience4j Circuit Breakers This is the scenario that made the whole thing click for me. We were using Resilience4j's circuit breaker on calls to an external inventory service, and our setup looked like this: @Service public class InventoryClient { @CircuitBreaker(name = "inventoryService", fallbackMethod = "fallbackInventoryCheck") public InventoryStatus checkAvailability(String productId) { return inventoryApi.getStatus(productId); } private InventoryStatus fallbackInventoryCheck(String productId, Exception ex) { log.warn("Inventory service unavailable for product {}, using cached data", productId); return inventoryCache.getLastKnownStatus(productId); } } The Resilience4j config in application.yml said the circuit should open after a 50% failure rate over 10 calls, with a 30-second wait before attempting to half-open. Fine on paper. But we'd never actually seen the circuit open under real conditions. We'd mocked it in unit tests, sure. Our integration tests always talked to a running inventory stub that never failed. So...