The pitch for an on-call agent is obvious and every vendor is making it: an alert fires at 3am, the agent reads the runbook, checks the dashboards, finds the cause, and either fixes it or wakes a human with a diagnosis instead of a pager message. Nobody wants the 3am page. Everybody wants the diagnosis.

The fear is also obvious. The agent misreads a graph, decides the fix is to restart the database, and it is 3am and nobody is watching.

We spent March to August finding out where the truth is between the two. The agent is now on the rotation, in a limited way, and I would not go back. But the way it got there was four stages, each with a gate, and the gates are the useful part of the story.

Stage one: read everything, touch nothing#

For the first six weeks the agent had read access to logs, metrics, traces, the runbooks, the deploy history and the incident history, and no write access to anything. When an alert fired, the agent was triggered in parallel with the human page. It investigated, wrote up what it found, and posted the write-up in the incident channel. The human on call did their normal job and, at some point, read the write-up.

The gate for moving on was that the on-call engineers rated the write-up as useful, on a two point scale, at least 80 percent of the time over 30 incidents.

It failed the gate the first time. The write-ups were long, they restated the alert, and they hedged. Engineers stopped reading them by week two. The fix was in what the agent was asked to produce: one line for the most likely cause, one line for the evidence, one line for the recommended action, and everything else in a thread. Same investigation, different output shape. After that change the rating went to 86 percent, and by week six the on-call engineers were opening the write-up before opening the dashboard, which is the behaviour that told us it was ready.

Two things we learned in this stage that shaped everything after. The agent was better than the median on-call engineer at correlating a deploy with an alert, because it always checked the deploy history and humans at 3am do not. And it was worse at knowing when a graph was normal, because it had no memory of what the graph usually looked like. We fixed the second one by giving it seven days of baseline for every metric it looked at, as a tool. That single tool cut its false "this is anomalous" rate roughly in half.

Stage two: propose the action, a human clicks#

In the second stage the agent could produce an action, from a fixed list, as a button in the incident channel. Restart this service. Scale this deployment from 3 to 6. Roll back this deploy. Flip this feature flag off. The human on call clicked the button or did not. The agent could not click.

The list was short on purpose: eight actions, all reversible, all things that a runbook already told a human to do in the corresponding situation. The gate for moving on was that over 40 proposals, the human clicked the proposed button without modification at least 75 percent of the time, and that no proposal, if it had been executed, would have made things worse.

That second condition needed a review. Every proposal, clicked or not, was reviewed the next day by someone not on call, who answered one question: had this been executed automatically, would it have been correct. Out of the first 40, 34 would have been correct, 4 would have been harmless but useless, and 2 would have been wrong. Both of the wrong ones were rollbacks of a deploy that was correlated with the alert but was not the cause. That was the same failure, twice: correlation from the deploy history, which was the agent's strength in stage one, becoming over-confidence in stage two.

The fix was a rule, not a prompt. A rollback proposal must cite a difference between the deployed change and the failing behaviour, on top of the timing. The agent had to read the diff and say what in it could cause the symptom. If it could not, it could propose the rollback with a "timing only" label, and the human treated that label as a reason to look harder. Wrong proposals in the next 40 went to zero.

Stage three: act on the short list, tell everyone#

Stage three is the one people mean when they say "the agent is on call". For four of the eight actions, the agent could execute without a click. The four were restart, scale up, flag off, and the rollback with a diff citation. Not scale down, not rollback on timing alone, not anything involving data.

Three rules governed every automatic action.

It announces before it acts, in the channel, with a 60 second window during which a human can type stop. At 3am nobody will, and that is fine, but during the day someone often does, usually with "wait, I'm already on it".

It does one action per incident. If the first action does not resolve the alert, the agent goes back to proposing and a human decides. This rule exists because the failure mode of an autonomous system is not one bad action, it is a sequence of individually reasonable actions that compound. Restart did not help, so scale up, which did not help, so roll back, and now three things have changed and nobody knows which one mattered. One action, then a human.

It records everything as an incident timeline entry, in the same format a human's actions are recorded, with the reasoning attached. The post-incident review reads the agent's actions exactly the way it reads a person's.

The gate for this stage was the same as stage two's second condition, over 30 automatic actions: none made things worse. It took eleven weeks to get 30 automatic actions, because most incidents in that period resolved at the proposal stage or were things the agent correctly declined to act on. None made things worse. Two were unnecessary, a restart of a service that would have recovered on its own within a minute, and those were fine.

Stage four, which we are in#

The list of automatic actions is now eleven. The additions came one at a time, each after a period of being a proposal with a perfect record. The rules from stage three still apply to all of them.

The thing that changed most in stage four is not the agent. It is the rotation. The human on call now gets paged for about 40 percent of the alerts they used to get, because the other 60 percent are either resolved by the agent or diagnosed by it to the point where the page is "this is the known flaky job, agent has restarted it, no action needed" and the engineer acknowledges from bed without opening a laptop. Mean time to a first diagnosis, measured from alert to the first timeline entry that names a cause, went from around 14 minutes to under 3.

The pages that remain are the real ones. Those are the pages that should wake someone up.

The rules, on one page#

Read before propose, propose before act. Each stage is gated on the previous one's record, and the record is reviewed by someone not on call.

Actions come from a fixed list. The agent does not invent an action. Adding to the list is a decision made in daylight.

Every automatic action is reversible and announced with a window to stop it.

One action per incident, then a human.

Timing is not causation. A rollback needs a cited diff.

Give the agent a baseline for every metric, or it will think everything is an anomaly.

The output is three lines, everything else is in the thread.

Record agent actions the way you record human ones, and review them the same way.

The one incident#

There was one, in stage three, and it is worth describing because it was not the agent's fault and the rules still mattered. A restart proposal was executed automatically, announced, and completed correctly. A human, who had been paged for a different alert on the same service, was in the middle of a manual restart of the same pod at that moment. Two restarts overlapped and the service was down for 40 seconds longer than it needed to be. The fix was a lock: the agent checks for an active human session on the target before acting, and defers if there is one. The daylight review found it the next morning, which is the whole reason the review exists.

What I would not do#

I would not start at stage three because a vendor's demo did. The demo is stage three on a scripted incident. Your incidents are not scripted.

I would not give the agent an action that touches data. Not a migration, not a cache flush, not a queue purge. Those stay with humans, possibly forever, because they are the actions where "reversible" is a matter of opinion.

I would not skip the daylight review of proposals. It is tedious, it takes ten minutes a day, and it is the only mechanism that turns the agent's mistakes into rules instead of into incidents.

And I would not measure it by pages avoided. Measure it by time to first correct diagnosis and by actions that made things worse. The first is the value. The second is the cost, and if it is not zero the agent is not ready for the next stage, no matter how quiet the pager has become.