Running Chaos Monkey pod-by-pod does not scale. I built a command relay POC that fans out assault config over RabbitMQ, applies it via loopback actuator, and aggregates per-replica status — here's the pattern and what still belongs in test.
I had the assault config right. I knew which method I wanted to fault. I even had a JSON payload ready. Then I spent twenty minutes port-forwarding into three pods, curling /actuator/chaosmonkey/assaults on each one, and hoping I had not missed a replica hiding behind a rolling deploy. That is not chaos engineering. That is operational toil with extra steps. So I built a proof-of-concept chaos command relay : one API call fans out to every replica of a target service, each pod applies Chaos Monkey locally, and the relay tells you whether the command actually landed. This post is the blog-sized version of that work — the mental model, the patterns that held up, and the shortcuts you should not copy into production. You can find the code in this repo: Chaos Command Relay POC The Problem: Per-Pod Actuator Calls Do Not Scale Chaos engineering on a microservice platform has a few hard requirements: Configure assaults in a repeatable way Target a specific application and environment Know whether every pod applied the command Observe downstream behavior while faults are active SSH, port-forward, and curl to actuator endpoints fail all four once you have more than a handful of replicas. Embedding a full operator UI in every service couples chaos concerns to application code. Neither scales. The fix is separation of concerns across three layers: Control plane ( chaos-command-relay ): Validate commands, publish to the bus, aggregate results, expose operator APIs Data plane listener ( chaos-listener-lib ): Embeddable library on each target pod; filter messages; apply via loopback actuator Verification surface ( chaos-poc-ui ): Read runtime Chaos Monkey state, exercise APIs, chart telemetry The relay submits commands. The listener applies them. The verify UI observes — it does not send commands. Control and observation stay separate, which matters when you are debugging whether a failure is in the bus, the listener, or the assault itself. Architecture at a Glance Chaos Command Relay architecture — component zones and numbered command flow Project Structure The POC is a Maven multi-module repo (Java 17, Spring Boot 3.0.9, Spring Cloud 2022.0.4, Chaos Monkey 3.1.0): chaos-poc/ ├── chaos-listener-lib/ # Embeddable listener + shared message DTOs │ └── src/main/java/com/samba/chaos/listener/ │ ├── message/ # ChaosCommandMessage, ChaosCommandResult, ChaosAssaultConfig │ ├── ChaosCommandListener.java │ ├── ChaosActuatorClient.java │ └── ChaosListenerAutoConfiguration.java ├── chaos-command-relay/ # Control plane (REST API + operator console) │ └── src/main/java/com/samba/chaos/relay/ │ ├── ChaosCommandController.java │ ├── ChaosCommandService.java │ ├── ChaosCommandStore.java │ └── console/ # Thymeleaf operator UI ├── chaos-poc-demo/ # Primary assault target (order flow + Feign gateways) ├── chaos-poc-downstream/ # Mock inventory/auth — second listener target ├── chaos-poc-ui/ # React verify surface (Vite + Tailwind) ├── scenarios/ # JSON command presets (latency, exceptions, multi-service) ├── scripts/ # verify-scenarios.sh and helpers ├── docker-compose.yml └── Dockerfile # Multi-stage; target selects runtime image Modules chaos-listener-lib : Fanout listener, loopback actuator client, result publisher, shared JSON contract chaos-command-relay : Validate → publish → aggregate → status API + Thymeleaf console chaos-poc-demo : Real Chaos Monkey on an order service; embeds the listener chaos-poc-downstream : Downstream dependency mock for multi-service scenarios chaos-poc-ui : Read-only verify UI — CM monitor, API playground, live telemetry Diagram Mental Model: Fanout, Filter, Loopback, Aggregate Four design choices make the POC work: 1. Fanout, not point-to-point routing. The relay publishes once to a RabbitMQ fanout exchange. Every listener binds an anonymous exclusive queue. Add a replica, and it self-registers — no relay reconfiguration. 2. Application-level filtering on the consumer. Each pod ignores messages where environment or targetApplication does not match its local config. One publish, many consumers, no brittle routing keys per service. 3. Loopback actuator only. The listener calls http://127.0.0.1:{port}/actuator/chaosmonkey/* inside the pod network namespace. No cross-pod actuator traffic. No actuator credentials on the relay. 4. Shared message contract in a library. ChaosCommandMessage , ChaosCommandResult , and ChaosAssaultConfig live in chaos-listener-lib so relay and targets serialize identical JSON. Here is the lifecycle in one pass: Operator → POST /internal/v1/chaos/commands → relay validates, stores PENDING → Rabbit fanout → each matching pod: dedupe by commandId → POST assaults/enable (loopback) → ChaosCommandResult → results queue → relay merges outcomes Operator ← GET /internal/v1/chaos/commands/{id} → APPLIED | FAILED | PARTIAL | TIMED_OUT Poll until you hit a terminal status. PARTIAL usually means expectedInstances does not match how many pods actually reported — a common POC gotcha when replica counting is still static. What a Command Looks Like The relay accepts structured JSON. A typical assault configures Chaos Monkey to throw on a specific service method: { "environment": "test", "targetApplication": "chaos-poc-demo", "action": "CONFIGURE_AND_ENABLE", "issuedBy": "poc-operator", "correlationId": "scenario-bean-interceptor", "expiresAt": "2026-12-31T23:59:59Z", "assault": { "level": 1, "deterministic": true, "exceptionsActive": true, "watchedCustomServices": [ "com.chaos.demo.service.OrderService.placeOrder" ], "exception": { "type": "java.lang.RuntimeException", "method": "<init>", "arguments": [ { "type": "java.lang.String", "value": "Chaos Monkey - RuntimeException" } ] } } } Two details burned me early: watchedCustomServices needs the fully qualified class name. OrderService.placeOrder will not match. For Feign clients, wrap the client in a @Service gateway bean so Chaos Monkey has a concrete Spring bean to intercept. Each pod publishes a result with outcome ( SUCCESS or ACTUATOR_ERROR ), podName , and the failing actuator step if something went wrong. The relay rolls those up into command-level status. The Listener Library: Drop-In, Gated, Boring on Purpose chaos-listener-lib is Spring Boot auto-configuration. It activates only when: Profile includes test or chaos-monkey samba.chaos.command-listener.enabled=true @AutoConfiguration @Profile({"test", "chaos-monkey"}) @ConditionalOnProperty(prefix = "samba.chaos.command-listener", name = "enabled", havingValue = "true") Each pod creates a non-durable, exclusive, auto-delete queue bound to the fanout exchange — classic competing-consumer fanout, except every consumer gets every message and filters locally. Inside the listener, the flow is deliberately dull: receive message → dedupe by commandId → call loopback actuator → publish result. Dull is good. You want the interesting failures to be in your application under assault, not in the chaos plumbing. Production services would keep the listener behind a dedicated chaos profile so assault machinery never wakes up in a production runtime profile by accident. Try It Locally The POC is a Maven multi-module project (Java 17, Spring Boot 3.0.9, Chaos Monkey 3.1.0) with Docker Compose wiring relay, RabbitMQ, demo app, downstream mock, and a React verify UI. services: rabbitmq: image: rabbitmq:3.13-management ports: - "5672:5672" - "15672:15672" environment: RABBITMQ_DEFAULT_USER: chaos RABBITMQ_DEFAULT_PASS: chaos healthcheck: test: ["CMD", "rabbitmq-diagnostics", "-q", "ping"] interval: 10s timeout: 5s retries: 12 start_period: 20s chaos-poc-downstream: build:...