We gave an agent access to production on-call in stages over five months: read only, then propose, then act on a short list, then act on a longer one. Each stage had a specific thing it had to prove before the next. The rules that came out of it are simple enough to fit on a page, and they are not the rules I would have guessed at the start.
The telemetry bill for one service passed its compute bill in March. The instinct is to sample harder and log less. We did the opposite: fewer, wider events, tail-based sampling that keeps every error, and a one-week retention tier for the boring 95 percent. The bill went down by two thirds and the incidents got easier to debug, not harder.