Give every LLM feature a token budget
A customer pointed a bulk export at a 400,000 row table and the feature spent a month's budget in an afternoon. A counter in Redis, checked before each call, turns that into an alert at the hourly limit and a tripped switch at the daily one.
if (Number(await redis.get(dayKey())) >= budget.daily) disableAndPage();Set the numbers from real usage plus headroom. They are there for the abnormal afternoon, not the normal one.
llmreliability
Longer version: the post this came from.