The feature was a summary at the top of every long thread in a customer's inbox. Three sentences, generated when the thread was opened, cached after that. It had been in production for five months and it worked well enough that customers had started to mention it.

On a Tuesday in May the model provider had an overloaded afternoon and started returning 529 for a meaningful fraction of requests. Our code caught the error, logged it, and returned an empty summary. The UI rendered an empty summary box. For four hours, every customer who opened a long thread saw a grey rectangle with nothing in it, and the on-call engineer saw an error rate graph that was elevated but not alarming, because the errors were being handled.

Nobody had decided what the feature should do when the model is not there. The code had made that decision by default, and the default was to show the customer a broken feature and tell nobody.

That afternoon is where the four things in this post come from. None of them are about the model.

The switch#

The first thing is the most boring. The feature is behind a flag, and the flag can be turned off in under a minute by anyone on call, without a deploy, and turning it off makes the feature disappear rather than degrade.

We had a flag. It was the rollout flag from launch, and after launch it had been set to 100 percent and forgotten. It technically could have been flipped. Nobody on call knew it existed, it was not in the runbook, and flipping it would have hidden the feature, which is what we wanted, but the on-call engineer did not know that hiding the feature was an option, because it had never been framed as one.

The kill switch is now a separate flag, named summaries.enabled, listed in the incident runbook under "things you can turn off", with a sentence next to it saying what the customer sees when it is off. That sentence is the point. A kill switch is only useful if the person holding it knows what happens when they use it, and "the summary box does not render" is a perfectly good outcome that beats "the summary box renders empty" every time.

export async function threadSummary(threadId: string): Promise<Summary | null> {
  if (!(await flags.enabled("summaries.enabled"))) return null;
  // ... the rest
}

The UI treats null as "do not show the box". That was a one line change in the component and it should have been there from the first day.

The fallback#

The second thing is what the feature does when the model call fails and the switch is still on.

For a summary, the honest fallback is the previous summary if there is one, and nothing if there is not. A thread that was summarised yesterday and gained two messages today can show yesterday's summary with a small "may be out of date" marker. A thread that has never been summarised shows no box. Neither of those is a broken feature. Both are the feature being slightly less good, which is what a fallback is.

The other option, which we also built, is a second provider. The summariser calls one model by default and a different one, from a different company, when the first returns a 5xx or times out. The prompt is the same, the output format is the same, and the eval set scores the second model about two points lower, which is fine for a fallback. It is not free: it means keeping two API keys, two sets of rate limits and two bills, and it means running the eval against both models on every prompt change. For a feature that customers have started to mention, that cost is worth it. For an internal tool, it would not be.

The order is: try the primary, on failure try the secondary, on failure return the cached previous summary, on no cache return null. Each step is a decision that was made in daylight, in code, rather than at 3pm on a bad Tuesday by whatever the catch block happened to do.

The budget#

The third thing is a limit on how much the feature is allowed to spend, in tokens and in money, per hour and per day, enforced in code.

Nobody thinks about this until the first surprising bill. Ours came from a different feature, a bulk export that called the model once per row and was pointed, by a customer, at a table with 400,000 rows. It ran for six hours before anyone noticed and the bill for that afternoon was larger than the feature's previous month.

The budget is a counter in Redis, incremented with the token usage from each response, checked before each call. When the hourly budget is exhausted, the feature falls back as if the model had failed, and an alert fires. When the daily budget is exhausted, the switch flips off automatically and a person is paged, because a feature that has spent its daily budget by 11am is either much more popular than yesterday or is being abused, and both need a human.

const budget = { hourly: 4_000_000, daily: 40_000_000 }; // tokens
 
async function withinBudget(): Promise<boolean> {
  const [h, d] = await redis.mget(hourKey(), dayKey());
  if (Number(d) >= budget.daily) { await flags.disable("summaries.enabled"); page("summaries daily budget"); return false; }
  return Number(h) < budget.hourly;
}

The numbers are set from the feature's actual usage plus headroom, and reviewed monthly. They are not there to save money in normal operation. They are there so that an abnormal afternoon costs an abnormal afternoon's worth and not a quarter's.

The canary#

The fourth thing is the one that would have turned the four hours into twenty minutes. A synthetic request, every two minutes, that exercises the whole feature path with a fixed input and checks the output.

The check is not "did the call succeed". The call succeeded during the bad Tuesday, in the sense that it returned an error we handled. The check is "did the feature produce a summary that looks like a summary": non-empty, under 400 characters, mentions a word from the fixed input. When the canary fails three times in a row, it pages, with the failure reason, which for the 529 afternoon would have been "provider returned 529" within six minutes of the problem starting.

It also checks the fallback. Once an hour the canary runs with the primary provider deliberately disabled and verifies that the secondary produced a summary. The one time that check failed, it was because the secondary provider's API key had expired, which we would otherwise have discovered during the next primary outage, at the worst possible moment.

What this is not#

This is separate from prompt engineering and from model quality. The eval set handles quality and it is a separate thing. These four are about the feature continuing to be a feature when the model, the provider, or the usage pattern does something unexpected, and they are the same four things you would build around any external dependency: a way to turn it off, a way to degrade, a limit on what it can consume, and a probe that tells you when it is unwell.

The reason they get skipped for LLM features specifically, I think, is that the model feels like the hard part, and once the model is working the feature feels done. The model is not the hard part. The model is a vendor API that sometimes returns 529, and it deserves exactly the wrapping that every other vendor API has earned.

Rolling out a model change behind the same switch#

The switch, the fallback, the budget and the canary were built for outages. They turned out to be what we needed for the thing that happens far more often than an outage, which is changing the model.

A provider retires a model version. A newer one scores better on the eval set. A cheaper one scores nearly as well. Each of those is a change to the feature that customers see, and before this setup, each was a deploy that either went fine or produced a Slack thread three days later saying the summaries "feel different". Nobody could say how, and there was nothing to roll back to except the previous deploy.

Now a model change is a flag value with a percentage. summaries.model is a flag whose value is the model name, and a rollout is setting the new value for 5 percent of tenants, then 25, then all, over a week. The eval set gates the first step: a model that scores below the current one on the set does not get 5 percent. The canary runs against both values throughout, so a regression that the eval set missed shows up as a canary failure on the 5 percent cohort before it reaches anyone else. The budget is per model, because a new model with a longer default output can double the token spend without changing anything visible.

The rollback is the flag going back to the old value, which takes as long as the flag propagates, under a minute. The previous deploy is not involved.

The cost of doing it this way is that the code has to handle two models at once, which mostly means the output parser cannot assume one model's formatting habits. That was true anyway, because the fallback provider already forced it. Once a feature has two models it can have three, and the third is free.

The runbook entry#

Everything above is worth exactly as much as the person on call at 3am can find. Here is the entry, in full, because a runbook that says "see the design doc" is not a runbook.

Summaries. Flag summaries.enabled turns the feature off, the summary box disappears, customers see nothing broken. Flag summaries.model picks the model, current value in the flag UI. Provider errors are handled with a second provider, then a cached previous summary, then nothing. If the canary alert fires and the error is 5xx from the primary, do nothing for ten minutes and check that the fallback is producing summaries. If the fallback is also failing, turn the feature off. If the budget alert fires, look at the per tenant usage panel for one tenant doing something unusual, and if there is one, rate limit that tenant rather than turning the feature off for everyone. Re-enable by setting the flag back.

Eight sentences. The on-call engineer who read that on the bad Tuesday would have turned the feature off in the first fifteen minutes and gone back to bed.

The thing that was actually hard#

None of the four pieces took more than a day. The hard part was the decision they all depend on: what should the customer see when the feature cannot work. That question had never been asked. The product manager, when asked, took a week to answer it, because the honest answer required admitting that the feature was optional, which is not how anyone had talked about it in the launch deck.

It is optional. Every LLM feature is optional in the sense that the product worked before it existed, and the day the model is unavailable, the product has to be able to work without it again. Writing that down, in one sentence next to a flag name, is the entire design. The four pieces are the implementation of that sentence.

The checklist#

Before an LLM feature goes to customers:

A kill switch, separate from the rollout flag, in the runbook, with a sentence describing what the customer sees when it is off.

A fallback chain, decided in code: second provider, cached previous output, or nothing. null is a valid output and the UI knows what to do with it.

A budget in tokens per hour and per day, enforced before the call, that trips the switch and pages when exhausted.

A canary that checks the output shape, not the status code, and that exercises the fallback on a schedule.

Four things, a day or two of work, and the grey rectangle never appears again.