A million requests a minute is 16,667 per second. One imagined system built as a monolith and as microservices: internal traffic, database budgets, caching, timeouts, retries, load shedding, the outbox, scaling, tracing, load tests, and cost, with Spring Boot 4 code.
Someone asks me how to handle a million requests. My first answer is a question: per what? The second is another question: through a monolith or through microservices? They sound like dodges. They're the two most useful things I can say, because the same number describes very different systems, and the architecture changes where the limits are, not whether you hit them. Per what? One million per Average rate What it usually takes Day about 12 per second One small instance and a database with an index Hour about 278 per second A few instances, a connection pool you sized on purpose Minute about 16,667 per second Caching, load shedding, a database budget, a queue for writes Second 1,000,000 per second Sharding, edge caching, a platform team, and a very different bill A million a day and a million a second are not the same system. They're not even the same team. The average also lies. Traffic doesn't arrive evenly across 86,400 seconds. It comes in peaks: a business-hours curve, a marketing email, a retry storm after a deploy. I size for the peak I measured, not for the daily total divided by the clock. For the rest of this piece I'll use one million per minute, about 16,667 requests per second at peak. It's large enough that every layer hits a limit and small enough that one team can still own it. The system I'm imagining, built twice Imagine a job marketplace. Users search jobs, open a job, look at their profile, and apply. Every application triggers a notification. I'll assume this mix at peak: Edge request Share Per second Search jobs 70% 11,667 Job details 20% 3,333 Profile 8% 1,333 Apply 2% 333 A search shows a fit score for each result, so it needs job summaries and the user's profile. An application checks that the job is open and snapshots the profile. Built as a monolith, that's one deployable with search, jobs, profiles, applications, and notifications as modules, sharing one PostgreSQL database. Built as microservices, it's a gateway, a search service, a job service, a profile service, an application service, and a notification service. Each service has its own PostgreSQL database, and Kafka carries events between them. flowchart LR subgraph mono["Monolith"] medge["Edge"] --> mapp["App: search, jobs, profiles, applications"] mapp --> mdb["One PostgreSQL"] end subgraph micro["Microservices"] gw["Gateway"] --> search["search-service"] gw --> jobs["job-service"] gw --> profiles["profile-service"] gw --> apps["application-service"] search --> jobs search --> profiles apps --> jobs apps --> profiles apps --> kafka["Kafka"] kafka --> notify["notification-service"] end Same features, same 16,667 requests per second at the edge. Very different numbers inside. Little's Law, and the traffic you didn't count Little's Law says the number of requests in flight equals the arrival rate times the time each request spends in the system. In the monolith, a search calls the job and profile modules as method calls. At 16,667 requests per second and 200 ms per request, that's 16,667 × 0.2 = about 3,333 requests in flight across the fleet. In the microservices version, those method calls become HTTP calls, and each one is a request some other service has to serve: Service Called by Requests per second search-service Gateway 11,667 job-service Gateway, search, applications 15,333 profile-service Gateway, search, applications 13,333 application-service Gateway 333 Total about 40,667 16,667 requests per second at the edge become about 40,667 inside. The 24,000 extra are calls between services, and every one of them pays for a network hop, serialization on both ends, a thread, and a timeout I have to choose. On a thread-per-request server, each request in flight holds a thread, and Tomcat in Spring Boot 4 defaults to 200. Java 21 virtual threads remove that ceiling in either architecture with one property: spring.threads.virtual.enabled=true Virtual threads remove the thread ceiling. They don't add database connections. That's the next hop, and it's where the two architectures differ most. The database budget: shared or split Assume reads take 20 ms, search queries take 50 ms, writes take 10 ms, and caching takes 70% of searches, 95% of job lookups, and 90% of profile lookups off the database. Connections busy at once are queries per second times query time: Workload Database queries per second Busy connections Search 3,500 175 Job lookups 767 about 15 Profile lookups 1,333 about 27 Applications 333 about 3 In the monolith, all of that lands on one database: about 220 busy connections. PostgreSQL ships with max_connections set to 100. In the microservices version, the same work is split across four databases. Three of them are comfortable. The search database still needs 175 connections on its own. Microservices don't reduce database work. They isolate it. In the monolith, a slow search query competes with applications for the same connections. In microservices, the application database doesn't notice a search spike. That isolation is real, and it's often the honest reason to split. The monolith can get most of that isolation without a network hop. I send search reads to a read replica with its own pool, and keep writes on the primary with a separate, smaller pool. Two pools, two budgets, one deployable. Pool size per instance is where microservices quietly break the budget. HikariCP, Spring Boot's default pool, defaults to a maximum of 10 connections per instance. Assume a load test showed one instance handles about 1,000 requests per second. The job service then runs 16 instances, and 16 × 10 = 160 open connections against a database that allows 100, even though only 15 are busy at peak. The pool is a per-database budget divided by the instance count, not a per-instance default. For the job service, that's a pool of 3 or 4. For many instances against one database, a connection pooler like PgBouncer in front of PostgreSQL keeps the server-side count flat while instances scale. The HikariCP pool sizing guide's starting point is (core_count × 2) + effective_spindle_count connections per database, then measured. I also shorten how long a request waits for one. Hikari's default connectionTimeout is 30 seconds: spring: datasource: hikari: maximum-pool-size: 4 connection-timeout: 2s A request that can't get a connection in 2 seconds fails fast, and the failure shows up on a dashboard instead of in a timeout three services upstream. The cache is the multiplier Caching is the cheapest throughput there is, and the math is unforgiving. At 11,667 searches per second, a 70% hit rate sends 3,500 to the database. A 90% hit rate sends 1,167, and the search database drops from 175 busy connections to about 58. The failure mode is the stampede. A hot key expires, and every request that arrives in the next few milliseconds misses at the same time and goes to the database for the same row. In Spring, sync = true makes one thread per key load the value while the others wait: @Cacheable(cacheNames = "jobSummaries", key = "#id", sync = true) public JobSummary findSummary(long id) { return jobRepository.findSummary(id); } The architectures differ in where the cache can live. In the monolith, an in-process cache serves every module. Search reads job summaries from memory, with no serialization and no hop. In microservices, the job service caches its own data, but search still pays an HTTP call to reach that cache. So the caller often caches too, which means two caches, two TTLs, and a staleness window that's the sum of both. That's fine for job summaries. It isn't fine for anything that has to be current, like whether a job is still open. I add jitter to TTLs in both, so keys written together don't expire together. Latency and timeouts: one budget or a chain In the monolith, a search is one request with one timeout at the edge, plus a statement timeout on the database. Module calls cost nanoseconds and don't...