Microservices don't fail because of the pattern. They fail because of the decisions nobody wrote down until production made them expensive. After 25 years shipping distributed systems, I keep seeing the same 7 choices separate the teams that scale from the teams that drown in their own service graph. Here's what I've...
A few years back, on the healthcare microservices rewrite I worked on, we split a hospital scheduling monolith into what someone on the team proudly called "eleven clean services." Six months later we had eleven services, sure, but also a distributed transaction nightmare, three different ways of handling retries, and an on-call rotation that dreaded Tuesdays (deploy day, for reasons nobody could fully explain anymore). The services were small. The system was not simple. That gap between "small services" and "simple system" is where most microservices programs actually live or die, and it's almost never about the diagram you draw on day one. I've been building distributed systems since around 2011, first at Bank of America's digital marketing stack, then through lending platforms, document pipelines, and eventually the hospital rewrite I just mentioned. Now I'm building ITJobOpportunities from scratch as a founder, which means I get to make every one of these decisions myself instead of inheriting someone else's. That's a different kind of pressure, honestly. So here are seven decisions I've seen determine whether a microservices architecture actually works in production, versus just looking good in a slide deck. 1. Where you draw service boundaries (and why "one service per table" is a trap) This is the one everyone gets wrong first. Teams look at their domain, see a User table, an Order table, a Payment table, and think "great, three services." That's not domain modeling, that's just table-splitting with extra network hops. The better question isn't "what data do we have" but "what changes together, and what fails independently." On a FinTech platform I worked on at Encora, we had a lending service that touched loans, payments, and disbursements. It would've been tempting to split those into three services from day one. We didn't, because they shared a transaction boundary that mattered: a loan approval and its initial disbursement had to be consistent, and forcing that across a network call would've meant building a distributed saga for something that used to be a single database commit. Boundaries should follow business capabilities, not entities. Ask yourself: does Team A ever need to deploy without waiting on Team B? If the answer is "not really," you probably don't need two services yet. You can always split later. Merging two services back together after a bad split is way more painful than most people admit. 2. Sync REST versus async events (and the middle ground nobody talks about) Everyone frames this as a binary choice. It isn't. I default to synchronous REST for anything the caller needs a real-time answer to, think authentication checks, or a "check your fit" style scoring call where the user is staring at a loading spinner. For everything else, especially anything involving side effects across services, I reach for events. On ITJobOpportunities, the Job Fit Check feature is a good example of mixing both intentionally. A candidate uploads a resume against a job posting, and we kick off an async scoring job (LLM-backed skill matching, relevance scoring against the job description). The frontend doesn't sit there blocking on an HTTP call for 20 seconds. It opens an SSE connection and falls back to polling if the browser or proxy doesn't cooperate: @GetMapping(value = "/api/job-fit/{jobId}/stream/{sessionId}", produces = MediaType.TEXT_EVENT_STREAM_VALUE) public SseEmitter streamJobFitResult(@PathVariable Long jobId, @PathVariable String sessionId) { SseEmitter emitter = new SseEmitter(60_000L); jobFitEmitterRegistry.register(sessionId, emitter); return emitter; } That single decision, async job plus SSE plus polling fallback, saved us from a much worse alternative: making every candidate wait on a synchronous LLM call that might take anywhere from a few seconds to well over ten, depending on provider load. We tried the synchronous version first, actually. It worked fine with light traffic and started to strain the moment we had more than a handful of concurrent requests hitting the LLM provider. The middle ground I'd recommend: use REST for the request that kicks off work, use events (or a lightweight async job table, which is honestly underrated) for the actual processing, and give the client a way to observe progress. Don't force everything into pure pub/sub just because it feels more "microservices." 3. Database per service, or shared database with discipline I know the textbook answer here: one database per service, no exceptions, full stop. In practice, I've broken that rule more than once, and I'd do it again. On a public-sector integration project at Encora, we ran Kafka with Debezium doing change-data-capture off a shared PostgreSQL instance, feeding multiple downstream services. Was it textbook-pure? No. Did it let us avoid building five separate eventual-consistency pipelines for data that genuinely belonged together? Yes. Here's my actual rule of thumb: If two services need strong consistency on shared data, and splitting the database would force you into distributed transactions, keep them on the same schema, at least for now. Sagas are powerful but they're also a maintenance tax you pay forever. If a service owns a clear, bounded piece of data that nothing else needs to touch transactionally, give it its own schema (even within the same PostgreSQL instance, that's often enough isolation to start). Physical database-per-service (separate instances, separate backups, separate scaling) should be reserved for services with genuinely different scaling or compliance needs. On ITJobOpportunities, the jobs-posting backend runs everything under a single job schema in PostgreSQL 16, with Liquibase managing the changelog. One backend, one schema, two frontends (the public portal and the ATS console) hitting it through different routes. People will tell you that's not "real" microservices. Fine by me. It's shipped, it's fast to develop against, and I haven't had a single 2 AM page about eventual consistency between candidates and applications. When I eventually split out the AI pipeline (resume parsing, skill extraction) into its own deployable, it'll be because of scaling or team ownership, not because a blog post told me to. 4. How you handle failure (circuit breakers aren't optional decoration) I used to think of Resilience4j as something you add once things are "stable enough." That's backwards. The first time a downstream dependency goes slow instead of down, you'll understand why. Slow failures are worse than hard failures. A service that returns a 500 immediately is annoying. A service that hangs for 30 seconds per request, multiplied across a thread pool, will take down everything upstream of it through resource exhaustion. I learned this the hard way on a lending platform, where a downstream credit-check dependency started timing out intermittently. No crash, no error logs screaming at us, just... slow. Thread pools filled up. Healthy services started rejecting requests because they'd run out of capacity waiting on a dependency that wasn't even critical to the main flow. @CircuitBreaker(name = "creditCheckService", fallbackMethod = "fallbackCreditScore") @TimeLimiter(name = "creditCheckService") public CompletableFuture<CreditScore> getCreditScore(String applicantId) { return CompletableFuture.supplyAsync(() -> creditClient.fetchScore(applicantId)); } public CompletableFuture<CreditScore> fallbackCreditScore(String applicantId, Throwable t) { return CompletableFuture.completedFuture(CreditScore.unavailable()); } That's a simplified version, but the shape matters more than the exact config: a timeout, a circuit breaker, and a fallback that degrades gracefully instead of cascading. Set your timeouts aggressively too. A 30-second timeout on an internal service call is basically no timeout at all. 5. Observability you actually build for, not bolt on later Th...