Distributed Systems Engineering
Distributed Systems Engineering — Reference Architecture · A request travels edge → gateway → services → data; state changes flow out as events. Every hop can fail, be slow, or be partitioned. · reso…
On this board
- Distributed Systems Engineering — Reference Architecture
- A request travels edge → gateway → services → data; state changes flow out as events. Every hop can fail, be slow, or be partitioned.
- resolve
- miss
- HTTPS
- lease
- CAS
- register
- lookup
- mTLS · retries
- 1
- 2
- event
- fail
- WAL
- 1. write
- 2. AppendEntries
- T1
- T2
- T3
- ok
- ① Clients & Edge
- ② Traffic Management
- ③ Coordination & Consensus
- Mechanism A — Raft log replication (CP write path)
- 3. ack (majority = 2/3)
- Clients
- GeoDNS
- CDN / Edge PoP
- WAF / DDoS
- Load Balancer
- API Gateway
- Rate Limiter
- Leader Election
- etcd / ZooKeeper
- Distributed Lock
- Client
- Leader
- Follower 1
- web · mobile · IoT
- latency-based routing
- static cache, TLS
- filter bad traffic
- L4/L7, health checks
- auth, routing, versioning
- token bucket
- one active leader
- Raft / ZAB quorum
- lease + fencing
- SET x=5
- term 7, appends to log
- persists, acks
- Follower 2
- gRPC / REST
- ④ Control Plane (Orchestration)
- ⑤ Stateless Microservices
- ⑥ Async Messaging (Event Log)
- 4. Leader commits entry 5. Applies to state machine 6. Replies to client • Election timeout 150–300 ms • Higher term wins; stale leaders step down • 3 nodes tolerate 1 failure
- slow / partitioned
- Autoscaler
- Kubernetes
- Service Discovery
- Service Mesh
- Order Service
- Checkout Service
- Payment Service
- Kafka Log
- Consumer Group
- Dead Letter Queue
- Term & log index order every entry
- scale on CPU / lag
- scheduling, self-heal
- registry + health
- Envoy sidecars
- owns order data
- Saga orchestrator
- exactly-once effect
- partitioned, replicated
- offsets, rebalancing
- poison messages
- Mechanism B — Saga with compensations (checkout)
- tx + outbox row
- cart state
- idempotency keys
- Checkout
- Create Order
- Reserve Stock
- Charge Payment
- Order Done
- ⑦ Replicated Relational Store
- ⑧ Partitioned / Specialized Stores
- ⑨ Observability
- Read Replicas
- Primary (Leader)
- CDC (Debezium)
- Object Storage
- Search Index
- Cassandra Ring
- Redis Cluster
- OTel Collector
- Prometheus + Jaeger
- Grafana / Alerts
- orchestrator
- PENDING
- local tx
- idempotency key
- CONFIRMED
- T3 fails → C2 release stock
- follower, may lag
- all writes, ACID
- tails WAL / outbox
- snapshots, erasure coding
- eventually consistent
- consistent hashing, N=3
- cache, 16384 slots
- metrics · logs · traces
- time series, spans
- SLO burn rate
- C1 cancel order
- outbox events → topic
- Core Concepts of Distributed Systems Engineering
- 1. WHY DISTRIBUTE? • Scalability: scale out (horizontal) beyond one machine • Availability: survive node / rack / region failures • Latency: put data close to users (geo-distribution) • Isolation: independent teams & deploys (microservices) Cost: partial failure, no global clock, network is unrelia…
- 2. 8 FALLACIES OF DISTRIBUTED COMPUTING The network is reliable · Latency is zero · Bandwidth is infinite · The network is secure · Topology doesn't change · There is one administrator · Transport cost is zero · The network is homogeneous → design assuming every one of these is false
- 3. FAILURE MODELS • Crash-stop / crash-recovery • Omission (lost messages) • Timing (slow node ≈ dead node) • Network partition (split brain) • Gray failure (partially working) • Byzantine (arbitrary / malicious) Detect with heartbeats + timeouts (φ-accrual)
- 4. CAP & PACELC CAP: during a Partition choose Consistency or Availability PACELC: Else (normal operation) choose Latency or Consistency • CP: etcd, ZooKeeper, Spanner, HBase • AP: Cassandra, DynamoDB, Riak Real choice is per operation, not per system
- 5. CONSISTENCY MODELS (strong → weak) Strict serializable > Linearizable > Sequential > Causal > Read-your-writes / Monotonic reads > Eventual Quorum rule: R + W > N gives overlapping reads/writes Conflict resolution: LWW, vector clocks, CRDTs
- 6. TIME & ORDERING • Physical clocks drift; NTP is not enough for ordering • Lamport clocks: happens-before total order • Vector clocks: detect concurrency • Hybrid Logical Clocks (CockroachDB) • TrueTime + commit wait (Spanner) Fencing tokens protect against stale leaders
- 7. CONSENSUS Agree on one value / ordered log despite failures • Paxos, Multi-Paxos, Raft, ZAB, Viewstamped Replication • Needs majority quorum: 2f+1 nodes tolerate f failures • FLP: no deterministic consensus in pure async systems → use timeouts • BFT (PBFT, HotStuff): 3f+1 nodes
- 8. REPLICATION • Single-leader (Postgres, MySQL): simple, failover risk • Multi-leader: multi-region writes, conflicts • Leaderless (Dynamo): quorums, hinted handoff, read repair, anti-entropy (Merkle trees) Sync = durable but slow · Async = fast but can lose data
- 9. PARTITIONING (SHARDING) • Range: good scans, hot spots risk • Hash: even spread, no range queries • Consistent hashing + virtual nodes: minimal movement on resize • Rebalancing, secondary indexes (local vs global) • Watch for hot keys & skew
- 10. DISTRIBUTED TRANSACTIONS • 2PC: atomic, but blocks if coordinator dies • Saga: local tx + compensating actions • Transactional Outbox + CDC: no dual writes • Idempotency keys + dedup → effectively-once Delivery: at-most-once / at-least-once / exactly-once (effect)
- 11. RESILIENCE PATTERNS Timeouts everywhere · Retries with exponential backoff + jitter · Retry budgets · Circuit breaker · Bulkheads · Backpressure · Load shedding · Graceful degradation · Hedged requests (tail latency) · Health checks & readiness probes
- 12. OPERATE & VERIFY • SLI / SLO / error budgets; golden signals: latency, traffic, errors, saturation • Distributed tracing (trace-id propagation) • Chaos engineering, game days • Jepsen tests, TLA+ specs, deterministic simulation • Canary deploys, feature flags, blameless postmortems