Alert Strategy

Overview

An alert strategy defines what should trigger attention, who responds, and how severity is classified.

Strategy Principles

• Alert on symptoms that impact users
• Route alerts to responsible teams
• Define severity levels and escalation rules
• Prioritize actionable alerts only

flowchart LR
    A[Service Signal] --> B{Threshold Breach?}
    B -->|No| C[Observe]
    B -->|Yes| D[Create Alert]
    D --> E[On-call Response]

Practical Example

alert: HighErrorRate
expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.02
for: 10m
labels:
  severity: critical

Production Example

Customer-facing APIs commonly alert on sustained error rate and p95 latency breaches rather than single short spikes.

Common Pitfalls

• Alerting directly on infrastructure noise
• No ownership metadata on alert rules
• Static thresholds that do not match traffic patterns

Key Takeaways

• Effective alerting focuses on user impact and actionability.
• Ownership and severity definitions reduce response confusion.
• Rule quality matters more than rule quantity.