Platform Runbooks
Platform runbooks document operational procedures, troubleshooting steps, and response protocols. Well-maintained runbooks enable consistent incident response, reduce mean time to recovery, and distribute operational knowledge across teams.
Understanding Platform Runbooks
Runbooks are operational playbooks that guide teams through specific scenarios.
The Knowledge Silos Problem
Without runbooks, knowledge remains in individual heads:
Problems Without Runbooks:
├── Only senior engineers know how to fix issues
├── Different approaches for same problem
├── Repeated mistakes during incidents
├── Slow incident response
├── Knowledge lost when people leave
├── Training takes months
└── Late-night debugging from scratch
Impact: Prolonged outages and burnt-out team members.
Runbook Benefits
Documented procedures enable effective response:
graph LR
A[Incident Occurs] --> B[Consult Runbook]
B --> C[Follow Steps]
C --> D[Issue Resolved]
B --> E[Runbook Missing]
E --> F[Manual Investigation]
F --> G[Eventually Resolved]
D --> H[15 minutes MTTR]
G --> I[3 hours MTTR]
style D fill:#d4f1d4
style G fill:#ffcccc
Runbook Structure
Effective runbooks follow a consistent structure.
Standard Template
Every runbook should include:
# Runbook: [Issue Name]
## Overview
Brief description of what this runbook covers.
## Symptoms
- How to recognize this issue
- Relevant alerts or error messages
- Affected services/components
## Severity Assessment
- **Critical (SEV1)**: Customer-facing outage, revenue impact
- **High (SEV2)**: Degraded service, subset of customers affected
- **Medium (SEV3)**: Minor issues, no customer impact
- **Low (SEV4)**: Cosmetic issues
## Prerequisites
- Required access/permissions
- Tools needed
- Knowledge requirements
## Diagnosis Steps
Step-by-step investigation procedure.
## Resolution Steps
Detailed mitigation and fix procedures.
## Verification
How to confirm the issue is resolved.
## Rollback Plan
How to undo changes if resolution fails.
## Prevention
Long-term fixes to prevent recurrence.
## Related Resources
- Links to relevant documentation
- Related runbooks
- Architecture diagrams
- Monitoring dashboards
## Revision History
| Date | Author | Changes |
|------|--------|---------|
| 2024-08-15 | alice@example.com | Initial version |
Real Example: Database Connection Pool Exhaustion
Complete runbook:
# Runbook: Database Connection Pool Exhaustion
## Overview
Services cannot connect to the database due to connection pool exhaustion.
This typically happens during traffic spikes or connection leaks.
## Symptoms
- **Alert**: "High Database Connection Count" fires
- **Logs**: `"could not obtain connection from pool"` errors
- **Metrics**:
- `db_connections_active` near `max_connections`
- `db_connections_waiting` increasing
- **User Impact**: API requests timeout or return 500 errors
## Severity Assessment
**High (SEV2)**: Service is degraded, subset of requests failing
## Prerequisites
- Access to Kubernetes cluster
- Access to database admin console
- Grafana dashboard access
## Diagnosis Steps
### 1. Confirm Connection Pool Status
```bash
# Check current connections
kubectl exec -it deployment/my-api -- \
platform db connections status
# Expected output shows:
# Active: 95/100
# Waiting: 12
2. Identify Connection Sources
-- Connect to database
psql -h db.example.com -U admin -d production
-- Check active connections by application
SELECT
application_name,
COUNT(*) as connection_count,
state
FROM pg_stat_activity
WHERE datname = 'production'
GROUP BY application_name, state
ORDER BY connection_count DESC;
3. Check for Long-Running Queries
-- Find queries running > 5 minutes
SELECT
pid,
now() - query_start as duration,
application_name,
state,
query
FROM pg_stat_activity
WHERE state != 'idle'
AND now() - query_start > interval '5 minutes'
ORDER BY duration DESC;
4. Review Application Logs
# Check for connection errors
kubectl logs deployment/my-api --since=30m | \
grep -i "connection\|pool\|timeout"
Resolution Steps
Option 1: Scale Application (Immediate Relief)
# Reduce pod count to decrease total connections
kubectl scale deployment/my-api --replicas=3
# Wait 30 seconds for connections to drain
# Scale back up
kubectl scale deployment/my-api --replicas=10
When to use: Quick mitigation during incident
Option 2: Terminate Idle Connections
-- Terminate idle connections > 10 minutes
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE state = 'idle'
AND now() - state_change > interval '10 minutes'
AND datname = 'production';
When to use: Many idle connections observed
Option 3: Increase Connection Pool Size (Temporary)
# Update deployment with larger pool
kubectl set env deployment/my-api \
DB_POOL_SIZE=150 \
DB_MAX_OVERFLOW=50
# Restart pods
kubectl rollout restart deployment/my-api
When to use: Traffic spike exceeds normal capacity
Option 4: Kill Long-Running Queries
-- Terminate specific problematic query
SELECT pg_terminate_backend(12345); -- Replace with actual PID
When to use: Specific query causing issues
Verification
1. Check Connection Count
# Should be back to normal range (< 80%)
platform db connections status
2. Monitor Error Rate
# Check Grafana dashboard
open https://grafana.example.com/d/api-health
# Error rate should drop to < 1%
3. Verify Application Health
# Check health endpoints
curl https://my-api.example.com/health
# Should return 200 OK
Rollback Plan
If resolution steps cause issues:
# Revert to previous deployment
kubectl rollout undo deployment/my-api
# Reset environment variables
kubectl set env deployment/my-api \
DB_POOL_SIZE=100 \
DB_MAX_OVERFLOW=20
Prevention
Short-term (Do Immediately)
- Set up connection pool monitoring alerts
- Implement connection timeouts in application
- Add health checks that verify database connectivity
Long-term (Plan for Next Sprint)
-
Investigate Connection Leaks
- Add connection lifecycle logging
- Review code for unclosed connections
- Add automated tests for connection management
-
Optimize Connection Usage
# Use connection pooling best practices from sqlalchemy import create_engine engine = create_engine( DATABASE_URL, pool_size=20, # Base connections max_overflow=10, # Additional connections during spikes pool_timeout=30, # Wait time before timeout pool_recycle=3600, # Recycle connections hourly pool_pre_ping=True # Verify connections before use ) -
Implement Circuit Breaker
from circuitbreaker import circuit @circuit(failure_threshold=5, recovery_timeout=60) def query_database(query): # Database operation pass -
Add Database Read Replicas
- Separate read traffic to replicas
- Reduces load on primary database
Related Resources
- Database Architecture
- Connection Pool Configuration Guide
- Grafana Dashboard
- Related Runbooks:
Revision History
| Date | Author | Changes |
|---|---|---|
| 2024-08-15 | alice@example.com | Initial version |
| 2024-08-20 | bob@example.com | Added circuit breaker prevention step |
### Runbook Categories
Organize runbooks by type for easy discovery.
#### Incident Response Runbooks
Handle production issues:
runbooks/incidents/ ├── database-connection-exhaustion.md ├── high-memory-usage.md ├── pod-crashloop.md ├── certificate-expiration.md ├── api-gateway-timeout.md ├── disk-space-full.md └── redis-eviction.md
#### Operational Runbooks
Regular operational tasks:
runbooks/operations/ ├── database-backup-restore.md ├── certificate-renewal.md ├── kubernetes-upgrade.md ├── adding-new-team.md ├── rotating-secrets.md ├── scaling-cluster.md └── disaster-recovery.md
Example operational runbook:
```markdown
# Runbook: Rotating Database Credentials
## Overview
Periodic rotation of database credentials for security compliance.
Required quarterly for all production databases.
## Prerequisites
- Database admin access
- Access to secrets management system (Vault)
- Access to Kubernetes cluster
- Approval from security team
## Steps
### 1. Generate New Credentials
```bash
# Generate strong password
NEW_PASSWORD=$(openssl rand -base64 32)
# Store in temporary secure location
echo $NEW_PASSWORD > /tmp/new_db_pass
chmod 600 /tmp/new_db_pass
2. Create New Database User
-- Connect to database
psql -h db.example.com -U admin -d production
-- Create new user with same permissions as old
CREATE USER app_user_new WITH PASSWORD 'NEW_PASSWORD';
GRANT CONNECT ON DATABASE production TO app_user_new;
GRANT USAGE ON SCHEMA public TO app_user_new;
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO app_user_new;
3. Update Secrets in Vault
# Store new credentials
vault kv put secret/production/database/app_user_new \
username=app_user_new \
password=$NEW_PASSWORD \
host=db.example.com \
port=5432 \
database=production
4. Update Application Configuration
# Update Kubernetes secret
kubectl create secret generic db-credentials \
--from-literal=username=app_user_new \
--from-literal=password=$NEW_PASSWORD \
--dry-run=client -o yaml | \
kubectl apply -f -
# Restart application pods to pick up new credentials
kubectl rollout restart deployment/my-api
5. Verify Application Connectivity
# Check pod logs for successful connections
kubectl logs deployment/my-api --tail=50 | grep "database"
# Verify no connection errors
kubectl logs deployment/my-api --since=5m | grep -i "error\|fail"
6. Monitor for Issues
Wait 30 minutes and monitor:
- Application error rates
- Database connection metrics
- Health check endpoints
7. Remove Old Credentials
-- Only after verifying new credentials work
DROP USER app_user_old;
8. Update Documentation
# Update credential rotation log
echo "$(date): Rotated credentials for production database" >> \
docs/credential-rotation-log.md
Rollback Plan
If issues occur:
# Revert to old credentials
kubectl create secret generic db-credentials \
--from-literal=username=app_user_old \
--from-literal=password=$OLD_PASSWORD \
--dry-run=client -o yaml | \
kubectl apply -f -
kubectl rollout restart deployment/my-api
Schedule
- Production: Quarterly (Jan, Apr, Jul, Oct)
- Staging: Biannually (Jan, Jul)
- Development: Annually (Jan)
Checklist
#### Troubleshooting Guides
Debug common issues:
runbooks/troubleshooting/ ├── pod-not-starting.md ├── service-not-reachable.md ├── high-latency-investigation.md ├── memory-leak-detection.md └── network-connectivity-issues.md
### Runbook Automation
Automate runbook execution where possible.
#### Executable Runbooks
Runbooks as code:
```python
# executable-runbook: database-connection-exhaustion
from platform_sdk import Kubernetes, Database, Monitoring
import logging
logger = logging.getLogger(__name__)
class DatabaseConnectionExhaustionRunbook:
"""
Automated runbook for handling database connection exhaustion.
Can be executed automatically or manually.
"""
def __init__(self, service_name: str):
self.service_name = service_name
self.k8s = Kubernetes()
self.db = Database()
self.monitoring = Monitoring()
def execute(self, auto_resolve: bool = False):
"""Execute runbook steps"""
logger.info(f"Starting runbook for {self.service_name}")
# Diagnosis
status = self.diagnose()
if not status['is_exhausted']:
logger.info("Connection pool is healthy")
return
logger.warning(f"Connection exhaustion detected: {status}")
# Automatic resolution if enabled
if auto_resolve:
self.resolve_automatically(status)
else:
self.provide_manual_steps(status)
def diagnose(self) -> dict:
"""Diagnose connection pool status"""
# Get connection metrics
connections = self.db.get_connection_stats(self.service_name)
# Get long-running queries
long_queries = self.db.get_long_running_queries(
self.service_name,
min_duration_seconds=300
)
# Get pod status
pods = self.k8s.get_pods(self.service_name)
return {
'is_exhausted': connections['active'] > connections['max'] * 0.9,
'active_connections': connections['active'],
'max_connections': connections['max'],
'waiting_connections': connections['waiting'],
'long_queries_count': len(long_queries),
'pod_count': len(pods),
'idle_connections': connections['idle']
}
def resolve_automatically(self, status: dict):
"""Automatically resolve the issue"""
logger.info("Attempting automatic resolution")
# Strategy 1: Terminate idle connections
if status['idle_connections'] > 20:
logger.info("Terminating idle connections")
terminated = self.db.terminate_idle_connections(
self.service_name,
idle_minutes=10
)
logger.info(f"Terminated {terminated} idle connections")
# Strategy 2: Rolling restart if still exhausted
if self.diagnose()['is_exhausted']:
logger.info("Performing rolling restart")
self.k8s.rolling_restart(self.service_name)
# Wait and verify
time.sleep(30)
if not self.diagnose()['is_exhausted']:
logger.info("Issue resolved automatically")
self.monitoring.create_incident_note(
service=self.service_name,
message="Connection exhaustion auto-resolved"
)
else:
logger.error("Automatic resolution failed, escalating")
self.escalate()
def provide_manual_steps(self, status: dict):
"""Provide manual resolution steps"""
print(f"\n=== Manual Resolution Required ===\n")
print(f"Service: {self.service_name}")
print(f"Active Connections: {status['active_connections']}/{status['max_connections']}")
print(f"Waiting: {status['waiting_connections']}")
print(f"Idle: {status['idle_connections']}")
print("\nRecommended Actions:")
if status['idle_connections'] > 20:
print("\n1. Terminate idle connections:")
print(f" platform db terminate-idle {self.service_name} --idle-minutes=10")
if status['long_queries_count'] > 0:
print("\n2. Check long-running queries:")
print(f" platform db list-queries {self.service_name} --long-running")
print("\n3. Perform rolling restart:")
print(f" kubectl rollout restart deployment/{self.service_name}")
print("\n4. Monitor recovery:")
print(f" watch platform db connections {self.service_name}")
# CLI integration
if __name__ == "__main__":
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("service", help="Service name")
parser.add_argument("--auto", action="store_true", help="Auto-resolve")
args = parser.parse_args()
runbook = DatabaseConnectionExhaustionRunbook(args.service)
runbook.execute(auto_resolve=args.auto)
Usage:
# Manual mode: provides steps to execute
platform runbook database-connection-exhaustion my-api
# Automatic mode: executes resolution steps
platform runbook database-connection-exhaustion my-api --auto
Runbook Triggers
Automatically trigger runbooks from alerts:
# AlertManager configuration
groups:
- name: database-alerts
rules:
- alert: DatabaseConnectionExhaustion
expr: db_connections_active / db_connections_max > 0.9
for: 5m
labels:
severity: high
runbook: database-connection-exhaustion
annotations:
summary: "Database connection pool nearly exhausted"
description: "{{ $labels.service }} has {{ $value }}% connections in use"
runbook_url: "https://runbooks.example.com/database-connection-exhaustion"
auto_resolve: "true"
# Runbook execution from alert
apiVersion: v1
kind: ConfigMap
metadata:
name: alert-runbook-mapping
data:
mappings: |
database-connection-exhaustion: platform runbook database-connection-exhaustion {{service}} --auto
high-memory-usage: platform runbook high-memory {{service}}
pod-crashloop: platform runbook crashloop {{namespace}}/{{pod}}
Runbook Maintenance
Keep runbooks accurate and useful.
Review Schedule
Regular runbook validation:
# Runbook maintenance tracking
from dataclasses import dataclass
from datetime import datetime, timedelta
@dataclass
class RunbookMetadata:
path: str
last_updated: datetime
last_used: datetime
author: str
review_due: datetime
accuracy_rating: float
class RunbookMaintenance:
def __init__(self):
self.runbooks = []
def find_stale_runbooks(self) -> list:
"""Find runbooks needing review"""
now = datetime.now()
stale = []
for runbook in self.runbooks:
# Not updated in 6 months
if (now - runbook.last_updated).days > 180:
stale.append({
'runbook': runbook.path,
'reason': 'Not updated in 6 months',
'last_updated': runbook.last_updated
})
# Never used (might be outdated)
if runbook.last_used is None:
stale.append({
'runbook': runbook.path,
'reason': 'Never used in production',
'last_updated': runbook.last_updated
})
# Low accuracy rating
if runbook.accuracy_rating < 3.0:
stale.append({
'runbook': runbook.path,
'reason': f'Low accuracy rating: {runbook.accuracy_rating}/5',
'last_updated': runbook.last_updated
})
return stale
def schedule_review(self, runbook: str, reviewer: str):
"""Schedule runbook review"""
# Create GitHub issue for review
github.create_issue(
title=f"Review runbook: {runbook}",
body=f"""
Please review and update this runbook:
- Verify all commands still work
- Update any changed procedures
- Add any new insights from recent incidents
- Update screenshots if applicable
Runbook: {runbook}
""",
assignee=reviewer,
labels=['runbook-maintenance']
)
Post-Incident Updates
Update runbooks after incidents:
## Post-Incident Runbook Update Process
After every incident:
### 1. Review Runbook Effectiveness
- Did the runbook exist?
- Was it easy to find?
- Were the steps accurate?
- Were any steps missing?
- What would have helped?
### 2. Update Runbook
```bash
# Open runbook for editing
vim runbooks/incidents/[issue].md
# Add lessons learned section
# Update resolution steps
# Add new diagnosis steps discovered
3. Add Incident Reference
## Related Incidents
- [INC-2024-08-15](https://incidents.example.com/INC-2024-08-15)
- Discovered that scaling helped more than connection termination
- Added new monitoring query for connection leaks
4. Submit for Review
# Create PR with runbook updates
git checkout -b update-runbook-connection-exhaustion
git add runbooks/incidents/database-connection-exhaustion.md
git commit -m "Update runbook based on INC-2024-08-15"
git push
# Request review from incident responders
gh pr create --reviewer alice,bob
### Key Takeaways
- Platform runbooks document operational procedures and troubleshooting steps, distributing knowledge and enabling consistent incident response
- Effective runbooks follow a standard structure including symptoms, diagnosis steps, resolution procedures, verification, and rollback plans
- Categorize runbooks by type (incident response, operations, troubleshooting) for easy discovery during high-pressure situations
- Executable runbooks combine documentation with automation, enabling automatic or semi-automatic issue resolution
- Integrate runbooks with monitoring systems to automatically trigger appropriate procedures when alerts fire
- Regular runbook maintenance ensures accuracy through scheduled reviews, post-incident updates, and tracking usage metrics
- Well-maintained runbooks reduce mean time to recovery (MTTR), decrease incident stress, and accelerate new team member onboarding