Every alert that fires at 3am should earn its place. I wrote this after too many nights chasing noise instead of real signal. Here's what actually separates useful alerting from pager fatigue in production systems.
Most teams have too many alerts, and the ones they have are wired to the wrong signal. Everyone alerts on CPU, memory, disk. Those are resource metrics. They tell you a machine is working hard. They don't tell you whether your customers are having a bad time. What matters is whether your service is meeting its contract with users. If your API's p99 latency spikes from 200ms to 4 seconds, do you care that CPU is at 85%? Not directly. You care that requests are slow. The CPU number is a supporting detail, not the headline. This is the core idea behind Google's SRE approach to alerting, and it's one of the few "best practices" from that book I apply verbatim: alert on symptoms, not causes. Symptoms are things your users experience. Causes are things engineers investigate after being paged. The Four Golden Signals (and Why I Add a Fifth) Google's SRE book gives you four golden signals: latency, traffic, errors, saturation. I use all four, but I add a fifth for anything backed by a queue or async pipeline: lag . Latency , not just the average, but p95 and p99. Averages hide the pain. On the ATS console at ITJobOpportunities, average response time looked fine for weeks while p99 crept up because a handful of company-admin queries were doing full table scans on the applications table. Averages lied. Percentiles didn't. Traffic , request rate obviously, but also request shape. A sudden drop can be just as dangerous as a spike. If nobody's hitting your Easy Apply endpoint for twenty minutes during business hours, something upstream broke, maybe DNS, maybe a load balancer misconfig. Errors , scoped by type. A 4xx spike from bad candidate input is not the same emergency as a 5xx spike from an exhausted database connection pool. Lump them together and you'll page people for the wrong reasons. Saturation , how full the system is: thread pools, connection pools, queue depth. This is where resource metrics earn their place, as a saturation proxy, not a standalone alert. Lag (my addition), for anything running through Kafka, SQS, or an async job queue. Consumer lag is often the earliest warning you'll get. On a Kafka pipeline where we streamed change events with Debezium, consumer lag on the downstream service consistently moved first, sometimes 10 to 15 minutes before latency or error rate budged at all. That lead time matters. Lag gave us a head start almost every time, and it's the reason I now treat lag dashboards as a first-class citizen in any tool I set up, not an afterthought bolted on after an incident. SLOs Before Alerts, Not the Other Way Around I used to write alerts first and figure out what "good" looked like later. Backwards. The right order is: define your Service Level Objective , then build alerts around your error budget burn rate. Say your SLO is 99.9% availability over a rolling 30-day window. That's a monthly error budget of about 43 minutes of downtime. The question an alert should answer isn't "is something wrong right now?" It's "are we burning through our error budget faster than we can afford?" That's where multi-window, multi-burn-rate alerting comes in, and it changed how I think about paging thresholds. The idea, popularized by Google's SRE workbook, is to alert on the rate at which you're consuming your error budget across two windows at once: a short window (say, 5 minutes) to catch fast-moving incidents, and a longer window (say, 1 hour) to confirm the problem is sustained and not a blip. Here's a version of a Prometheus alerting rule I adapted for a job-fit scoring service with a 99.5% success-rate SLO: groups: - name: job-fit-slo-burn-rate rules: - alert: JobFitHighBurnRateFast expr: | ( sum(rate(job_fit_requests_total{status="error"}[5m])) / sum(rate(job_fit_requests_total[5m])) ) > (14.4 * 0.005) for: 2m labels: severity: page annotations: summary: "Job Fit error budget burning 14x too fast (5m window)" - alert: JobFitHighBurnRateSlow expr: | ( sum(rate(job_fit_requests_total{status="error"}[1h])) / sum(rate(job_fit_requests_total[1h])) ) > (6 * 0.005) for: 15m labels: severity: page annotations: summary: "Job Fit error budget burning 6x too fast (1h window)" The multipliers (14.4x, 6x) aren't arbitrary. They're tuned to how much of your monthly error budget you'd burn if the condition persisted for the full window. Google's SRE workbook lays out the math if you want to derive your own instead of copying mine. I'd recommend doing that exercise once, by hand, even if it feels tedious. It changes how you think about every threshold you set afterward. The Toolchain: What I Actually Run and Why This is the part most posts skip, or handle with a vague "use Prometheus and Grafana." I'll be specific about what runs where and why, because the right tool depends heavily on team size and budget, not just architecture. Metrics: Micrometer plus Actuator, not a full observability platform for everything For a team our size at ITJobOpportunities, we don't run a full Prometheus, Grafana, and Alertmanager stack for every service. It's overkill for a lean team, and the operational overhead of maintaining Prometheus itself becomes its own SRE burden. We do use it for core API metrics, because Spring Boot Actuator plus Micrometer makes exporting metrics almost free. @Bean public TimedAspect timedAspect(MeterRegistry registry) { return new TimedAspect(registry); } @Timed(value = "job.fit.scoring", description = "Time to score a job fit request") public JobFitResult scoreJobFit(ResumeInput input, JobPosting job) { // scoring logic return jobFitEngine.evaluate(input, job); } That @Timed annotation, paired with Micrometer's Prometheus registry, gives you latency histograms with almost no boilerplate. Add the management endpoint and Prometheus can scrape it directly: management: endpoints: web: exposure: include: health, info, prometheus, metrics metrics: distribution: percentiles-histogram: job.fit.scoring: true percentiles: job.fit.scoring: 0.5, 0.95, 0.99 That last block is easy to forget and it's the difference between having averages and having the percentiles you actually need for the golden signals above. It's one of my favorite small wins in the Spring ecosystem, and it's part of why I still reach for Actuator even on projects where I've considered lighter frameworks. Grouping and routing: Alertmanager, used narrowly Running on-call rotations at Encora for a Kafka-based pipeline, we once had well over a hundred active alert rules for a single service. Alertmanager's grouping and inhibition rules did more to fix that than any dashboard redesign. route: receiver: default-slack group_by: ['service', 'alertname'] group_wait: 30s group_interval: 5m repeat_interval: 4h routes: - match: severity: page receiver: pagerduty-oncall continue: false inhibit_rules: - source_match: alertname: DatabaseDown target_match: severity: page equal: ['service'] That inhibit_rules block is the piece most teams never configure, and it's the single change that cut our page volume by more than half during an incident on a healthcare microservices program running Kinesis. When the database goes down, every dependent service starts failing, but you don't need twelve pages telling you that. You need one, with the root cause, and the rest suppressed. Log-based alerting: CloudWatch metric filters, not a full ELK cluster For catching error patterns that metrics miss, we use structured JSON logging from Spring Boot into CloudWatch Logs, with metric filters that turn specific patterns into alarms. { "timestamp": "2026-01-14T03:12:44Z", "level": "ERROR", "service": "job-fit-scoring", "errorCode": "LLM_PROVIDER_TI...