Skip to content

.tip

· one of the small things

Retries multiply, so give them a budget

One request, three services deep, each layer retrying three times: twenty-seven calls arrive at the dependency that was already failing. Backoff and jitter spread them out, they do not make them fewer. Retry in one layer, and give that layer a budget.

let tokens = 10; // +0.1 per success, capped at 100
function mayRetry() {
  if (tokens < 1) return false;
  tokens -= 1;
  return true;
}

Successes pay for retries, so a dependency with a bad minute gets a few and a dead one gets none. The layers below just return the error.

distributed-systemsreliability