Skip to content

.tip

· one of the small things

Sample traces at the tail, keep every error

Head-based sampling at 10 percent throws away 90 percent of failures, because the coin is flipped before the request knows it's going to fail. Decide at the end instead: keep everything with an error or above p99, plus 2 percent of the rest by trace id.

tail_sampling:
  decision_wait: 30s
  policies: [status_code: ERROR, latency: 800ms, probabilistic: 2]

Set decision_wait longer than your slowest request, or the slow ones get dropped as "not slow yet".

observabilityopentelemetry

Longer version: the post this came from.