Distributed Heartbeats and Failure Detection: Partitions, Leases, and Deterministic Design

blog@baifan.site (ByF (BaiFan)) — Sat, 27 Sep 2025 14:00:00 +0800

Anyone who has debugged an early-morning production outage knows that the most dangerous failure in a distributed system is rarely a clean crash (Fail-Stop). It is the gray failure: transient unresponsiveness caused by Stop-the-World garbage collection, virtual machine CPU steal time, packet drops in network switches, or TCP half-open sockets. In an asynchronous network, physical wall-clock time offers no causal order guarantees. Handing a periodic ping packet the unilateral authority to decide node mortality turns that probe into a hair-trigger for cascading failovers and split-brain states.

Kubernetes on ByF (BaiFan) · Daily Deep Think

Distributed Heartbeats and Failure Detection: Partitions, Leases, and Deterministic Design