Give every LLM feature a token budget
A customer once pointed a bulk export at a 400,000 row table, and the feature burned a month's budget in an afternoon. A counter in Redis, checked before every call, turns that into an alert at the hourly limit and a tripped switch at the daily one.
if (Number(await redis.get(dayKey())) >= budget.daily) disableAndPage();Base the numbers on real usage plus headroom. They're there for the abnormal afternoon, and a normal one should never touch them.
llmreliability
Longer version: the post this came from.