Sample traces at the tail, keep every error
Head-based sampling at 10 percent drops 90 percent of the failures, because the coin flips before the request knows it will fail. Decide at the end: keep everything with an error or above the p99, and 2 percent of the rest by trace id.
tail_sampling:
decision_wait: 30s
policies: [status_code: ERROR, latency: 800ms, probabilistic: 2]Set decision_wait longer than your slowest request, or the slow ones get dropped as "not slow yet".
observabilityopentelemetry
Longer version: the post this came from.