Say hello to the assistant on this site and, until recently, it would tell you about setting temperature to zero for classification. Not a greeting back. A tip about temperature, every time, for "hello" and for "thanks" and for "good morning".

The reason is boring and it is worth knowing. The assistant answers from a vector index. A greeting gets embedded like anything else, the index returns whatever document sits closest to it in that space, and for a short friendly sentence with no subject the closest document is the same one every time. Then a model gets paid to write a sentence about it. I measured the scores on the eval's questions in September: small talk's best match landed between 0.51 and 0.56, and a question the site really answers landed at 0.61 and up. So I drew a line at 0.6 and the tip stopped appearing under greetings.

That line is a single number, fitted to a gap of five hundredths, standing in for the question "is this page about what was asked". It held. It was also the most obviously wrong thing in the codebase, because the real question is about meaning and the answer was a distance.

This post is about replacing that kind of number with Jev, a decision model from TypeSafe that reads the sentence instead of measuring the distance to it. It picks, rates and judges, and it writes nothing. There are ten of them on this site now, from the assistant's front gate to the note moderation on a public room. The model turned out to be the easy part. The work was deciding what to do when it was unsure.

What Jev is, quickly#

Jev is from TypeSafe, and it is the first of what they call System One models, after Kahneman's fast thinking. It takes some state and some questions, and it returns typed answers with probabilities. It does not write. Their docs say it plainly: jev-1.13 is not trained to generate text.

That sentence does most of the explaining. A generative model writes you a paragraph and you pull a decision out of the paragraph. Jev skips the paragraph. You hand it the text and the question, and back comes a number, or one of your own option names, or a position on a ladder you described. There is no JSON to repair, no run where it answers in prose instead, no schema to validate, because the answer space is the question you asked.

The price is $42 per billion input tokens and output tokens are free, which follows from there being almost no output. Context is 64k per request, 32k of it for the state.

That is the part you can read off a spec sheet, and it is the least interesting part. What I spent my time on was the rest: what to do with an answer you are only 60 percent sure of, who pays for the question, and what breaks when the answer never arrives.

The three questions, as we ask them#

A noul asks whether something is true and gives you a number from 0 to 1. A choice picks one of your named options and hands back the whole probability distribution plus a confidence. A score puts something on a ladder you describe, and it can land between two rungs.

They are evaluated in parallel against the same state and none of them sees another's answer. That has a practical consequence I keep using: one more question costs its own tokens and almost no time, so a question whose answer only matters sometimes is nearly free. Ask it in the same request rather than in a second one the code has to wait for.

Here is the gate in front of the site's assistant, which is a single choice:

const KIND = choice("What is `message`?", {
  question: "A question about this site, its posts, its tips or its author",
  small_talk:
    "A greeting, a thank you, a goodbye, or a remark that asks for nothing",
  search: "A request to find or list something, or just a subject to look up",
  action:
    "An instruction to the site itself: change the theme, open a page, play the radio",
  unclear: "None of these, or too little to tell",
});

The unclear option is not decoration. The keys you write are the whole answer space, so if your list does not cover an input the model still has to pick something, and it will. Give it a way out every time.

Note the backticks around message. The question refers to a field in the state you send, and you can walk into it: pages[2].title works, and that is how I ask about several things at once without repeating their text in every question.

A probability is not a confidence#

This one cost me an afternoon.

A choice gives you confidence, which describes how peaked the distribution is. Flat means the model does not know. A noul gives you no confidence at all, because the number itself is one: 0.5 is genuinely undecided, and both ends are certain about opposite things.

So a noul threshold has two sides, and they are not the same decision. When I ask "did the visitor ask to be taken somewhere", a reading of 0.9 is a yes, a reading of 0.1 is a no, and 0.5 means I learned nothing and should use whatever I used before. Writing if (probability > 0.5) throws away that middle and turns every shrug into a yes:

if (probability >= SURE_ENOUGH) return true;
if (probability <= 1 - SURE_ENOUGH) return false;
return null;

SURE_ENOUGH is 0.85 there. Null goes back to the two regexes that decided it before. I will come back to why that matters more than the threshold.

The wording is the specification#

The criteria you write are the whole of what the model knows about your problem. A person reading only the criteria, with none of your context, should sort things the way you would. If they could not, the model cannot either, and no threshold will save it.

The clearest lesson I got came from ranking posts. I wanted two axes to lay the writing out on a map, and the second one started as "how practical is this subject". On 21 September that question put 22 of the 28 posts on the same rung. That is not a map, it is a line with everything piled at one end. The question was the problem. Almost everything I write is practical, so I had asked something with no spread in it. Reworded to "how settled is the subject", with rungs from "the same advice would have held five years ago" through to "nobody has settled how to do this", the posts spread out.

When a reading comes back flat, suspect the question before the model.

The state matters as much. The enrichment judge asks whether each FAQ answer is carried by the post, and I first sent it the body with the frontmatter stripped, which felt tidy. It cost three false alarms in a run. An excerpt is the post's own summary and an answer is entitled to draw on it: the RAG post says "eleventh out of ten" in its excerpt and nowhere else, and the agent identity post says "every agent I audited this year" in its excerpt and nowhere else. Judged against a body that had neither, both answers looked invented. Send the thing the answer is allowed to lean on, which is rarely the thing you happen to have in a variable.

One more, for nouls. Keep both sides pointing the same way. A true that means "no, this is fine" reads worse than writing no criteria at all, and you will misread your own threshold six weeks later.

Every threshold is its own argument#

There are eleven thresholds across the ten call sites and no two of them were chosen the same way. Suggesting a link and deleting somebody's note do not deserve the same certainty.

Here is every one of them, and what happens when the reading falls short.

The assistant's front gate sits at 0.75 on a choice. Above that, small talk gets answered from the theme's own greeting lines and never opens a stream to the index at all, and a search gets handed to Pagefind, the index that ships with the build. Both of those save a Workers AI call on a message that was never going to get a good answer from one.

Moderation on the public notes room runs four nouls and a score in one request, for advertising, abuse, personal details, prompt injection and overall harm. A single hazard at 0.85 rejects on its own. Anything at 0.6 holds the note for a human. The harm score is out of two, holding at 1 and rejecting at 1.5. Those came from reading what the model said about notes I had already judged myself.

The radio sits at 0.35, which looks reckless until you see what it is doing. You describe a mood and it picks a mix. All five mixes are the same kind of music, so a non-temporal mood spreads its probability across all of them and the model is never confident. Measured on 21 September, "something to focus on" peaked at 0.42 and "something for the night" hit 1.00. The thing keeping an unrelated sentence off the dial is not the bar, it is the __none__ option. And the cost of a wrong pick is that you press skip.

The command palette sits at 0.7, because turning "make it quieter" into fx grain off is a change the visitor sees immediately.

Write the thresholds in code, not in the prompt. The model is better at reading a note than at remembering that your policy says 0.85, and you want to move the number without touching the question.

Two purses#

Every caller goes through one function, askJev, which checks the key, checks the day's budget, asks, and writes back what the answer cost. The cost comes from usage.input_tokens, the number the API reports, not an estimate, so a question whose state grew gets charged at its real size. Output tokens are free, which means input is the whole bill.

The part worth copying is the split into two budgets:

const PURSES: Record<Purse, { prefix: string; cap: () => number }> = {
  shared: { prefix: "jev:tokens", cap: () => env.TYPESAFE_DAILY_TOKENS },
  moderation: { prefix: "jev:tokens:mod", cap: () => env.TYPESAFE_MOD_TOKENS },
};

Everything cosmetic shares the first one: the assistant's gate, the palette, the radio, what to read next. Moderation has the second to itself. It is the only caller that exists to stop something rather than to add something, and a day of somebody hammering the command palette must not be a day of notes going unread.

The numbers are 500,000 input tokens a day for the shared purse and 100,000 for moderation. At $42 per billion that is about two cents a day if both are spent to the last token, and the balance in the account is$5, which is 119 million tokens. A question to the assistant's gate measures around 421 tokens, so 500k is roughly 1,180 of them. A note measures around 640, so 100k is about 150 notes. A moderate day on this site measures around 139k across everything.

The budget is read before the request goes out, not after it comes back, so a run of large questions can overshoot by at most the one already in flight. When it is spent, askJev returns null and every caller goes back to what it did before Jev existed. Reaching the cap costs nothing except the readings.

The counter that never counted#

The first bug is the one I would most like somebody to avoid.

The write-back was fired and forgotten. It looked careful. A lost write costs a few tokens of accuracy, a lost answer costs the visitor, so keep the critical path clear and let the counter catch up on its own:

void store.incrBy(budgetKey(), result.usage.input_tokens, BUDGET_TTL);

A Cloudflare Worker stops executing the moment it returns a response. The promise was dropped. I measured it against production on 21 September: six calls moved the counter once, from nothing to 426, and then never again.

So the read-before-ask check was comparing spend against a number that essentially never grew. The budget was decorative. Spend was still bounded, because every route has its own per-IP limit, but the one number meant to cap the lot was doing nothing at all. The fix is await instead of void, which is one extra D1 write beside a request that has already spent half a second asking. An uncounted call is an uncapped one, and the counter is the only reason the cap exists.

If you take one thing from this post, take that: whatever you use to bound spend, prove it moves. Spend two minutes calling your own endpoint and watching the number.

Spending somebody else's budget#

The second bug is a new shape of attack surface, and I had not thought about it before.

Three of the routes went out guarded by one check: does this request say it came from this site, in the sec-fetch-site header. A browser sets that header honestly. A script sets it to whatever it likes, or leaves it out. So the guard turned away a real browser on another site and let a script straight through.

On a route that returns data, that is a leak. On a route that spends a shared token budget, it is a way to switch off every reading on the site for the rest of the day, for the price of a loop and about 1,200 requests a minute. The fix was to put the same Turnstile check on them that the assistant already had, and the clients now fetch a token before they ask.

The general point: once a model sits behind an endpoint, that endpoint costs money per call, and the usual "this only returns public data so it can be open" reasoning stops applying. Rate limit by IP, and also cap the total, and also make the total real.

The model reads English#

Nothing I read before shipping prepared me for this, so here it is plainly. Jev is trained on English. The content on this site is English, so most of it is fine. The assistant takes questions in whatever language the visitor types, and mine get a lot of Turkish.

You can see it in the readings. "bloga ucur bizi kaptan" is Turkish, roughly "fly us to the blog, captain", and it asks to be taken somewhere. The model read it at 0.95. It got that right. But the same instinct that makes you raise the bar for a language the model reads less well also empties the feature, because an unsure reading with a high bar is a no, and a no means nothing happens.

The answer I ended up with is to make the middle mean something. Do not raise the bar and accept fewer yeses. Raise the bar on both ends and let the middle fall back to whatever decided it before. In Turkish that means the old regexes keep running, badly, exactly as badly as they ran last month, while English gets better and nothing anywhere gets worse.

Reading something and not acting on it#

The front gate classifies into five kinds and acts on two of them. The action class gets read and recorded, and then nothing is done with it.

Here is why. It put 0.77 on the message "I want to read posts", which is a person telling you what they like, and the separate question that decides whether to actually move somebody off the page they are reading rated the same message 0.68. Two readings of the same sentence, disagreeing, both above a half. Acting on the first would take a reader who said something mild and throw them onto another page.

So it goes to analytics as gate_kind and gate_confidence and nothing else happens. That is a real state for a feature to be in and there should be more of it. You can put a model in the path, keep its answer, and change nothing until the numbers tell you the bar is in the right place.

The same goes for probability distributions. Store the whole distribution. The pick on its own says nothing about how close it was. When you want to move a threshold in three months, the distribution is the only data you will have, and by then you will not remember what "0.42" felt like.

It is never the only answer#

The rule that survived everything: Jev is not the only source of authority anywhere on this site.

The moderation route passes on every failure. No key, empty note, request timed out, budget spent: the note goes up. A public room whose writes stop because a classifier is down is worse than one showing a rude line until somebody removes it. I have argued this before about anything with a model in it, and a decision model changes nothing about the argument. The last word still belongs to the rate limit and the ownership check, which are code.

The last piece I built is the clearest form of it. Remember MIN_LINK_SCORE = 0.6, the number from the opening. It is still there. What changed is that a reading can overrule it, and only when it is sure:

const verdict =
  probability >= WORTH_SHOWING ? true
  : probability <= WORTH_HIDING ? false
  : null;

0.7 and 0.3. A clear yes rescues a page the distance would have dropped, at 0.56, which is inside that five-hundredth gap I fitted the number to. A clear no drops the tip that shows up under "hello". Everything between hands the decision back to 0.6. The model cannot make the links worse than they were, because where it has nothing to say, the thing that decided before still decides.

That property is worth more than accuracy. It means I could ship it without an eval that proves it beats the old number, and it means the feature degrades to last month's behaviour rather than to nothing.

What it costs#

Four things run at build time on this site, once per change. Ranking all 30 posts against each other for related links costs 48,000 tokens. Placing them on a two-axis map costs 12,600. Judging the enrichment text, the FAQ answers and cover alt text and social descriptions against what the post actually says, costs 111,000 across every post. A full eval run of the assistant, 18 cases, costs 22,500. All four from scratch is 194,000 tokens, which is 0.8 cents.

On the request path it is cheaper than that per call and the ceiling is the daily cap, not the balance. The cap exists because the failure I care about is not cost, it is somebody finding an endpoint and making the readings stop for everybody else.

The latency is the thing to plan around, and it is not the 70 to 500 milliseconds in the marketing. That number is the model. Yours is the model plus your worker's network hop plus whatever else the request is doing. The front gate gets 1.5 seconds because a visitor is waiting with nothing on screen. The question about moving pages gets 5 seconds because the reply has already been written and a second more is free. Links get 2.5. Build scripts get 30.

That gap matters. On 21 September a reading came back at 0.95 in 782 milliseconds from a laptop, and the worker still did not answer within the two-second budget it had then, fell back to the patterns, and dropped the page change the visitor had asked for. The model was fast enough and the budget I had given it was not.

Where it stops#

It cannot count, do arithmetic, or compare dates, and their own docs say so. Reading time, prices, ordering and anything with money stays in code, and it always did.

It is not a parser. Terminal command parsing, URL matching, file paths: all of that is regex and it should stay regex. The places worth replacing are the ones where the code was guessing at meaning and pretending a number was an opinion.

The accuracy claims are the vendor's. I have not run a comparison against a small classifier on the same data, and the one independent benchmark I found was somebody's 40 hand-labelled tickets. It is early access, not GA, and there is no free tier, so a balance runs down while you experiment.

And it does not remove the work. It moves it. Instead of tuning one cosine threshold I now maintain eleven thresholds, two budgets, a fallback for every call site, and a set of questions that have to be worded so a model reading only the criteria would answer the way I would. That is more surface to look after than I had before. Better surface, because each piece is about one decision and says what it is for, but anybody selling you the idea that this deletes code is selling you something.

How I measured, so you can argue with it#

Token counts are usage.input_tokens as the API reports them, read back from the same field the budget is charged against. The per-call sizes are from a smoke script that makes one real request and prints the count.

The 0.51 to 0.56 and 0.61 figures are retrieval scores over the eval's 18 questions in September, from the index as it was that day. The confidence readings quoted for the radio and the front gate are single measurements against production on 21 September, not averages, and I have said so each time rather than dressing them up.

The counter bug was found by calling production six times and reading the D1 row. That was the whole method. The $42 per billion and the free output tokens are TypeSafe's published prices, and the 119 million figure is the account balance divided by that.

Every threshold in this post is a constant you can read in the repository, and where a number came from a judgement rather than a measurement I have tried to say which.