In September 2026 I tested a small decision model as the judgment layer of an AI agent fleet. It decided which agent takes each request, how risky the request is, and whether to hold it for a check before any agent writes code.
The headline result was strong: 16 of 16 correct, against 13 of 16 for a frontier model. The results that went wrong taught me more.
The model and the first result
The model is a typed judgment model. It does not write text. It answers one question at a time from a fixed set of options (multiple choice or yes or no) and returns a confidence for its answer. I used it to classify requests by type and risk, to answer yes-or-no risk questions such as “does this change send data to an outside service?”, and to route each request to an agent and decide whether to hold it.
On 16 hand-labeled prompts, the small model labeled all 16 correctly. A frontier model labeled 13 correctly (0.813).
| Small model | Frontier model | |
|---|---|---|
| Correct | 16 of 16 | 13 of 16 |
| Mean wait | 919 ms | 8,966 ms |
| Tokens per case | 794 | 21,167 |
I would read the token figure with care. Most of the 26× gap came from the 20,000-token system prompt the frontier model carried in that setup, not from the task itself. Sixteen cases is also a small sample. It showed the small model was good enough for this job, not that it is better in general.
Risk checks and the holdout
I tuned the wording of each risk question on a set of known cases. The tuned questions agreed with my labels on 93% of them.
But that number was in-sample. I had chosen the wording on the same cases I reported. So I froze the wording and wrote a new holdout set after the freeze, and agreement dropped to 77–86%. Worse, it had one false negative that repeated 30 times out of 30: a change that sent a customer’s personal details to an analytics vendor passed as safe.
For a safety check, the gate is zero false negatives, not a high average. I added explicit wording for the missed case (“sends any data to an external service, vendor or off-site destination”) and wrote a fresh holdout of 10 cases, because a holdout you have already used is no longer a holdout. The new wording caught every hazard in it. Each check cost about 1/600 of a mid-tier model call, with a mean wait of 345 ms against 15.7 s.
Routing
The router picked which agent should take each request. I ran it four times on the same frozen set of cases, with no change in between. It scored 74.3% to 76.3% (one earlier run reached 79.6%). The model is not deterministic, so I quote about 76% ± 3 and never a single run.
It chose the right agent on 33 of 33 scored cases. It never granted a forbidden permission in any run. Holds were the weak part. Only 28.6–31.6% of holds were needed, so about two in three held requests waited for nothing.
Both came from the same design: when unsure, hold. That is the right default for safety, and it costs speed. About two weeks later I retired the router, and the measurement method stayed.
Mistakes in my own numbers
The first one was a baseline I forgot. One rule set had “zero under-classified cases across 403 real cases”. That was true and meant nothing. The rule could only answer the top risk tier, so it could never under-classify. A constant answer of “top tier” matched 59% of all cases, and the rule matched 53% of the cases it covered. Without the baseline, 53% looked like progress.
Then there was evidence that raised confidence and lowered safety. I added a keyword scan as extra evidence for the question “does this change have side effects that cannot be undone?” The model went from answering 2 of 403 cases to 265 of 403, and every one of the 265 answers was “no”. A keyword scan cannot tell a change that mentions email from a change that sends email. More answers looked like progress, but all the new answers were in the unsafe direction, so I reverted it. I now check the direction of a change against the safe default, not only how many cases get an answer.
One label was spelled with an underscore in the data and with a hyphen in the options. 164 of 384 labels could not match, and one question looked like 3% agreement. After the fix it was 71%. I now fail the build when a label is not in its option set.
And small samples got me too. On an open-weights model, four runs of a puzzle suggested maximum reasoning effort beat low effort (3 of 4 against 2 of 4). Ten runs gave 6 of 10 at both. Four runs on a noisy task cannot tell 50% from 75%.
What I do now
I report the trivial baseline next to every headline number, because if a constant answer passes the test, the test is not evidence. I freeze the setup before writing the holdout, and I write a fresh holdout after every fix. For a safety check, one false negative fails it, whatever the average. Non-deterministic results get quoted as a range over repeated runs on the same data. And I validate labels in the build.
Want evals like these for your own agents or LLM features? Email hello@jeffkent.dev.
