Distributed Systems

Overview

Distributed systems run across multiple machines that coordinate through the network. They improve availability and scale, but introduce latency, partial failures, and consistency tradeoffs.

Fundamental Challenges

• Network partitions and timeouts
• Node failure and recovery
• Data consistency across replicas
• Clock drift and ordering uncertainty

flowchart LR
    A[Client] --> B[Service Node A]
    A --> C[Service Node B]
    B --> D[(Replica 1)]
    C --> E[(Replica 2)]
    D -. replication .-> E

CAP-Oriented Decision Example

A product catalog may prioritize availability over strict consistency during spikes. A payment ledger usually prioritizes consistency over availability for correctness.

Catalog service: AP-leaning behavior for read availability
Payment ledger: CP-leaning behavior for transactional correctness

Reliability Techniques

• Timeouts, retries, and circuit breakers
• Leader election and quorum writes
• Idempotent request handling
• Bulkheads to limit failure blast radius

Common Pitfalls

• Retrying non-idempotent operations blindly
• Assuming synchronous calls always succeed quickly
• Ignoring backpressure in downstream systems

Key Takeaways

• Distributed systems require explicit failure handling.
• Consistency choices must match business criticality.
• Resilience patterns are mandatory, not optional.