Skip to content

#opentelemetry

2 posts on this topic, newest first. all tags →

Tracing an agent like a distributed system

An agent that runs for forty turns, calls twelve tools and talks to three models is a distributed system with one process. We spent two years learning to debug those with traces. OpenTelemetry's GenAI semantic conventions let you do the same here, and the first trace I looked at explained a bug that logs had hidden for a month.

observabilityai-agentsopentelemetry+1 more
8 min read9 min listen

The observability bill is a design problem

In March the telemetry bill for one service passed its compute bill. The instinct is to sample harder and log less. We went the other way: fewer, wider events, tail-based sampling that keeps every error, and a one-week retention tier for the boring 95 percent. The bill dropped by two thirds, and incidents got easier to debug.

observabilityopentelemetrycost+1 more
10 min read13 min listen