Every LLM feature needs a kill switch
A model provider had a bad Tuesday and our summarisation feature spent four hours returning empty strings to customers, because nothing between the API and the user knew what to do with a 529. The feature now has a kill switch, a fallback, a budget, and a canary. Those four things are the difference between an LLM feature and an LLM demo.
10 min readRead more →