Retries multiply, so give them a budget
One request, three services deep, each layer retrying three times: twenty-seven calls arrive at the dependency that was already failing. Backoff and jitter spread them out, they do not make them fewer. Retry in one layer, and give that layer a budget.
let tokens = 10; // +0.1 per success, capped at 100
function mayRetry() {
if (tokens < 1) return false;
tokens -= 1;
return true;
}Successes pay for retries, so a dependency with a bad minute gets a few and a dead one gets none. The layers below just return the error.
distributed-systemsreliability