In February I made a list of everything in my daily workflow that called a hosted model. It was longer than I expected. The coding agent, obviously. But also: a git hook that drafts the commit message, a script that summarises a pull request for the changelog, a tool that reads a stack trace and guesses which file to open, a shell function that turns a sentence into a jq expression, a test runner plugin that names a failing test's likely cause, a thing that rewrites my Slack drafts into shorter Slack drafts, and a small classifier that sorts incoming GitHub notifications into "read now" and "read later".

Nine tools. Eight of them were sending code, logs, or messages to an API for tasks that a strong model finishes in under two seconds and a weak model would also finish in under two seconds. The coding agent was the only one that needed the frontier.

I moved the eight to a model running on the laptop. Three months later they are still there and I have not missed the API for any of them. This is about which tasks that works for, what the setup looks like, and where the line is.

The category#

The tasks that moved share a shape. The input is small, a few hundred to a few thousand tokens. The output is small and constrained: a commit message, a category, a one paragraph summary, a file path. The task has a right answer that a reasonable engineer would agree on, or a narrow space of acceptable answers. And the cost of a wrong answer is low, because a human sees the output immediately and can discard it.

For that shape, the difference between a frontier model and a good 30 billion parameter open-weight model is invisible. I checked. I ran 200 commit diffs through both and had two colleagues blind rank the messages. They preferred the hosted model's message 52 percent of the time, which is a coin flip. On the stack trace to file task, both got the right file 94 percent of the time on 100 traces from our error tracker and disagreed with each other on four.

The tasks that did not move share the opposite shape. Large input, open ended output, many steps, and a high cost of being wrong. The coding agent doing a refactor across twenty files. Anything where the model has to plan. There the frontier is still clearly better and I am not pretending otherwise.

The setup#

MacBook Pro, M4 Max, 64 GB. The model is a Qwen 3 variant at around 30B parameters in a 4 bit quantisation, which takes about 18 GB of memory and generates at roughly 40 tokens a second on this machine. I run it through a local server that speaks the OpenAI style chat API, because every one of my eight tools already spoke that, and switching was changing a base URL and a model name.

# ~/.config/tooling/env
LLM_BASE_URL=http://127.0.0.1:11434/v1
LLM_MODEL=qwen3-30b-a3b
LLM_API_KEY=local

That is the entire migration for six of the eight tools. The other two had hard coded the provider's SDK and needed a ten line change to use a generic client.

The server starts at login and idles at about 2 GB until the first request, when it maps the weights. First request after idle takes about four seconds. Subsequent ones are under two seconds for the typical commit message. The laptop fan does not come on. Battery cost over a working day is noticeable but not dramatic, maybe an extra ten percent, and on mains it does not matter.

The one piece of engineering worth mentioning is that the model of this size with a mixture-of-experts layout only activates about 3B parameters per token, which is why it is fast on a laptop despite having 30B in memory. That family of models is the reason this became practical this year. Two years ago the local options were either small and weak or large and slow.

The commit message hook, since people ask#

#!/usr/bin/env bash
# .git/hooks/prepare-commit-msg
set -euo pipefail
[[ "${2:-}" == "merge" || "${2:-}" == "squash" ]] && exit 0
diff=$(git diff --cached --no-color | head -c 12000)
[[ -z "$diff" ]] && exit 0
 
msg=$(curl -s "$LLM_BASE_URL/chat/completions" \
  -H "content-type: application/json" \
  -d "$(jq -n --arg d "$diff" '{
    model: env.LLM_MODEL, temperature: 0.2, max_tokens: 120,
    messages: [
      {role:"system", content:"Write a git commit subject line under 72 characters, imperative mood, conventional commits prefix, no trailing period. Output only the line."},
      {role:"user", content:$d}
    ]}')" | jq -r '.choices[0].message.content' | head -1)
 
# Put the suggestion above whatever git already put in the file.
{ echo "$msg"; echo; cat "$1"; } > "$1.tmp" && mv "$1.tmp" "$1"

The suggestion lands at the top of the editor and I accept it or rewrite it. I accept about 70 percent unchanged. The 30 percent I rewrite are mostly cases where the diff does not explain why, and no model can guess why from a diff.

Why not cost#

People assume the motivation is the API bill, and for eight tools making a few hundred calls a day the bill was around 20 dollars a month. The laptop cost more than that per month in depreciation. Cost is not the argument.

The argument is that these tools see everything. The commit hook sees every diff before it is pushed, including the ones on branches that will never be pushed. The log triage tool sees production stack traces with customer identifiers in them. The Slack rewriter sees drafts I decided not to send. The notification classifier sees the titles of private repositories.

I trust the hosted providers' data handling more than I trust most companies. That is not the point. The point is that for a task where the local model is as good, there is no reason for the data to leave the machine, and "no reason" should win. It also removed a compliance conversation at one client, where the log triage tool was the only thing on my laptop sending their production data anywhere, and now it sends it nowhere.

The other thing is latency and availability. The hook runs on a train with no signal. The API had a bad afternoon in April and every one of the eight tools fell over at once, which was the moment I noticed how many there were.

Where it fell short#

Two of the eight tasks needed prompt changes to work as well locally. The PR summariser was verbose with the local model in a way the hosted one was not, and a sentence in the system prompt saying "three sentences maximum" fixed it. The jq generator got the syntax right less often, about 85 percent against 96, and I added three examples to the prompt, which brought it to 93. Small models want examples more than large ones do. That is the whole adjustment.

Structured output is the other place to be careful. The hosted APIs guarantee valid JSON if you ask for it. The local server does too, through grammar constrained decoding, but you have to turn it on per request, and before I did the classifier occasionally returned a category with a trailing explanation that broke the parser. One flag.

And there is a ceiling. I tried moving the coding agent's simple mode, the one I use for "rename this and fix the imports", to the local model. It worked for the rename. It got lost the moment the task needed it to look at more than five files. That is the line, and it is not close to moving for the 30B class on a laptop yet.

The team version#

What works on one laptop does not automatically work for a team, and three colleagues asked for the setup within a month. The version we landed on is slightly different from mine and the differences are worth writing down.

Not everyone has 64 GB. Two people have 16 GB machines, and an 18 GB model does not fit. For them the answer was a smaller model in the same family, around 8B parameters, which fits in 5 GB and runs the same eight tasks. I ran the same blind comparison on the commit messages and the 8B model lost to the hosted one 61 to 39, which is a real difference but still means it wins four times in ten and the loser is still a usable message. For the classifier it was indistinguishable. For the jq generator it was noticeably worse and that person kept the hosted API for that one tool.

The tools themselves moved into a shared repository with the prompts, the base URL configuration and an install script, so that everyone runs the same version and a prompt improvement reaches everyone. The prompts had been drifting between machines within a week.

And we added a small shared eval, 50 inputs per tool with a known good output, that runs on each person's machine when they change model or prompt. It is the same idea as the eval sets I use for production features, at a fraction of the size. It caught a case where a model update changed the commit message format from "feat: ..." to "feat(scope): ..." across the board, which would have annoyed everyone for a week before someone tracked down why.

What it costs to keep running#

Local models are not free in the way people imagine, and the honest accounting is this.

Model updates are on you. A hosted API improves underneath you without a change on your side. The local model is the version you downloaded until you download another one, and the family I use has shipped three updates since February, each of which was worth taking and each of which took an hour to evaluate and roll out. The shared eval is what makes that hour an hour and not a day.

Memory is shared with everything else. On the 64 GB machine it does not matter. On the 16 GB machines, running the model alongside a browser, an editor and a container or two means something gets swapped, and the something is occasionally the model, which turns a two second commit message into a fifteen second one. The people on those machines run the server on demand rather than at login.

There is no rate limit, which sounds like a benefit and is also a way to write a tool that hammers the model in a loop and pins the CPU for a minute. The notification classifier did exactly that on the first day, processing 400 notifications one at a time on startup. A batch endpoint and a small concurrency limit fixed it.

None of that changes the conclusion. It is a different set of costs from the hosted API, smaller in money and larger in attention, and for tools that see everything I type, that is the trade I want.

What I would tell someone#

Make the list. You probably have more of these than you think, and they are probably all pointed at an API because that was the path of least resistance when you wrote them.

Sort the list by input size and by the cost of a wrong answer. Everything small and cheap to be wrong is a candidate.

Put the local model behind the same API shape the tools already use, and change the base URL. Do not rewrite anything.

Check the quality with a blind comparison on a hundred real inputs, because your intuition about which model is better is worth less than you think.

Keep the frontier for the agent. Use the laptop for everything else. The split comes down to size. Most of what I ask a model to do all day is small, and small does not need to leave the room.