YOUTUBE · PURE-TEXT EDITION

How Distributed Systems Scale to 10M RPS (Zero-Omission Publication Sample)

Systems Engineering Summit · Prof. Alex ChenDuration 43:002026-11-18Open Source Video
Executive Summary · TL;DR

Tail-latency amplification degrades synchronous RPC fan-out past 8 shards. Combining hedged requests at the local p95 trigger budget (18ms), cancellation propagation, and strict 5% retry token buckets cuts production p99 from 580ms down to 31ms.

[00:12] Tail Latency Amplification in Microservice Fan-Out

When staging tests run under low load, single-hop tail latency stays below 14ms. Once an aggregator fans out to 20 backend shards concurrently, overall request completion waits for the slowest shard.

KEY POINT

At a 1% per-leaf tail probability across 20 parallel leaf RPCs, end-to-end probability of hitting slow p99 surges from 1% to 18.2%.

[04:18] Hedged Requests vs Tied Requests Benchmarks

StrategyExtra TrafficProduction p99 LatencyBest Topology Fit
Synchronous Baseline0%580 msSingle-hop calls
Hedged Reads (p95 wait)+4.8%31 msReplicated read caches
Tied Requests w/ Queue Token+2.1%34 msObservable server queues

"Production reliability stems from bounding worst-case queues, never hoping averages stay low."

[13:00] Zero-Omission Protocol Configuration

grpc_deadline_ms=120 hedge_delay_ms=18 max_retry_budget_ratio=0.05 ./gateway-server
  1. Propagate caller remaining deadline across every downstream RPC header [13:00].
  2. Enforce circuit-breaker backoff whenever retry consumption approaches 5%.
Complete Raw Timestamped Transcript (3 lines)
[00:12] Today we analyze why tail latency dominates throughput past 8 shards. [04:18] Look at the p95 hedged delay trade-off. [13:00] Always enforce 120ms end-to-end deadlines.