The Hidden Bottlenecks That Break Microservices in Production

The Hidden Bottlenecks That Break Microservices in Production

The Connection Pool Trap This one gets people constantly, and it got us. Each service had its own database connections. Reasonable, right? But when the order service started receiving a surge, it opened connections faster than the pool could handle, and the downstream inventory service started queuing requests internally. The inventory service's connection pool to its own Postgres instance hit its limit. Requests started timing out. And because the order service wasn't handling those timeouts gracefully (it was waiting, not failing fast), threads piled up. Within about four minutes we had a thread exhaustion problem in a service that had nothing to do with the original bottleneck. The fix wasn't just "increase the pool size", though that was part of it. The deeper fix was understanding what our actual concurrency budget was per service and being deliberate about it. Spring Boot uses HikariCP by default, and the default max pool size is 10. Ten. For a service handling hundreds of concurrent requests, that's almost always wrong. Here's roughly what we landed on after some tuning: spring: datasource: hikari: maximum-pool-size: 30 minimum-idle: 10 connection-timeout: 3000 idle-timeout: 600000 max-lifetime: 1800000 But the number isn't the point. The point is you need to calculate it based on your actual query latency and throughput targets, not just bump it until things stop breaking. The formula I keep coming back to is roughly: pool size = (core count * 2) + effective spindle count. For most OLTP workloads that's a reasonable starting point, and you tune from there. One thing worth knowing if you're on Spring Boot 3.2 or later: virtual threads change this equation a bit. With Project Loom enabled, the platform threads that used to sit blocked waiting on a connection are now cheap virtual threads, so the cost of a saturated pool is lower than it used to be. You enable it with one line: spring: threads: virtual: enabled: true That said, virtual threads don't fix a misconfigured pool. They just make the failure mode a little less catastrophic. You still want to tune HikariCP explicitly. Retry Storms Oh, this one. This one is my nemesis. When a downstream service is slow or returning errors, every well-intentioned retry policy becomes a multiplier. Say you have five upstream services all retrying against one struggling downstream service, each with three retries and an exponential backoff. Under load, that "helpful" retry logic can quadruple the traffic hitting an already overwhelmed service. I've watched this turn a recoverable blip into a full cascade. Not great. The naive Spring approach is to just call the service and retry on exception. The problem isn't the retry itself, it's that every caller wakes up at roughly the same time after the backoff and hammers the downstream service in a synchronized wave. The fix is jitter, and Spring Retry makes this straightforward: <dependency> <groupId>org.springframework.retry</groupId> <artifactId>spring-retry</artifactId> </dependency> @Configuration @EnableRetry public class RetryConfig { @Bean public RetryTemplate retryTemplate() { ExponentialRandomBackOffPolicy backoff = new ExponentialRandomBackOffPolicy(); backoff.setInitialInterval(500); backoff.setMultiplier(2.0); backoff.setMaxInterval(10000); SimpleRetryPolicy retryPolicy = new SimpleRetryPolicy(); retryPolicy.setMaxAttempts(3); RetryTemplate template = new RetryTemplate(); template.setBackOffPolicy(backoff); template.setRetryPolicy(retryPolicy); return template; } } ExponentialRandomBackOffPolicy adds jitter automatically, which desynchronizes the retry waves. Small detail, big difference under load. But jitter alone isn't enough. You need a circuit breaker sitting in front of the retry logic. Resilience4j is the standard choice here, and it integrates cleanly with Spring Boot 3.x through the resilience4j-spring-boot3 starter: @Component public class InventoryClient { private final CircuitBreaker circuitBreaker; private final RestClient restClient; public InventoryClient(CircuitBreakerRegistry registry, RestClient.Builder builder) { this.circuitBreaker = registry.circuitBreaker("inventoryService"); this.restClient = builder.baseUrl("http://inventory-service").build(); } public StockResponse getStock(String itemId) { return circuitBreaker.executeSupplier(() -> restClient.get() .uri("/stock/{id}", itemId) .retrieve() .body(StockResponse.class) ); } } resilience4j: circuitbreaker: instances: inventoryService: failure-rate-threshold: 50 wait-duration-in-open-state: 10s sliding-window-size: 10 permitted-number-of-calls-in-half-open-state: 3 When the circuit opens, you stop hammering the failing service and give it room to recover. Non-negotiable, as far as I'm concerned. And notice I'm using RestClient here, not RestTemplate . Spring Boot 3.2 deprecated RestTemplate in favor of RestClient , which has a cleaner fluent API and better support for virtual threads. Synchronous Chains Are Time Bombs During the Q3 load-testing sprint last year, we traced a single user-facing API call and found it was making seven synchronous HTTP calls before returning a response. Seven. Each one had a 500ms timeout. In the worst case, that's 3.5 seconds of latency just from the chaining, before any of our own logic ran. Under normal load this was fine. Under high load, when each of those seven services was slightly slower due to resource contention, the tail latency exploded. P99 went from around 800ms to over 6 seconds. Users noticed fast. The fix isn't always to go fully async. Sometimes that's the right call, but it adds a lot of complexity. What we actually did first was identify which of the seven calls were truly required for the response and which ones were "nice to have" data enrichment. Three of them were enrichment. We moved those to parallel async calls using CompletableFuture and accepted that we'd sometimes return a slightly less complete response: @Service public class ProductService { private final UserServiceClient userClient; private final RecommendationClient recClient; public ProductResponse getProductPage(String userId, String productId) { CompletableFuture<UserProfile> profileFuture = CompletableFuture.supplyAsync(() -> userClient.getProfile(userId)); CompletableFuture<List<Recommendation>> recsFuture = CompletableFuture.supplyAsync(() -> recClient.getRecommendations(userId)); UserProfile profile = profileFuture.join(); List<Recommendation> recs = recsFuture.join(); return ProductResponse.of(productId, profile, recs); } } With virtual threads enabled in Spring Boot 3.2+, the supplyAsync calls run on virtual threads from the common pool, so you're not burning platform threads while waiting on I/O. That alone cut our average latency by about 40% on that endpoint. Not because we removed load, just because we stopped waiting on things serially that didn't need to be serial. The Observability Gap Nobody Talks About Dashboards lie. Not maliciously, just by omission. Average latency is almost meaningless. I know everyone knows this, but I still see teams building alerting on p50 latency and wondering why users are complaining. Look at p99 and p99.9. A service can have a perfectly healthy average while 1% of requests are timing out, and at scale that 1% is a lot of real people having a bad time. The other thing I consistently see missing is distributed tracing with actual context propagation. Logs per-service are fine, but when something goes wrong under load you need to follow a single request across service boundaries. We set up Grafana Tempo with Micrometer Tracing...