Every LLM feature needs a kill switch
A model provider had a bad Tuesday, and our summarisation feature spent four hours returning empty strings to customers because nothing between the API and the user knew what to do with a 529. The feature now has a kill switch, a fallback, a budget and a canary. Without those four it was a demo.
llmreliabilityfeature-flags+1 more
10 min read13 min listen