I spent a weekend running two small language models against three jobs I would normally hand to a language model without thinking about it. For one of those jobs I also wrote a regular expression, as a control.
The regex won. Thirty test cases out of thirty, in zero milliseconds, from a pattern that took an afternoon to write. The better of the two models scored twenty-eight, took five seconds a call, and the two it missed are both mistakes a regex is structurally incapable of making.
That is the kind of result I was looking for, and the question behind it is one I keep avoiding: what is the smallest amount of computation that reliably does this job? Not the cheapest model. Not the best small model. The smallest thing that works, which means the ladder has four rungs and the bottom one is not a model at all. Deterministic code. A regex, a lookup table, an if statement. A small local model. Runs on hardware you already own. A larger local model. Needs a GPU. A remote frontier model. Someone else's hardware, metered.
Most projects start at rung four and never ask about the others. This is a weekend spent asking.
The hardware was a Raspberry Pi 4, 4 GB of RAM, heat sinks and a fan, a hundred and seventy dollars most of which I already owned. The models were smollm2:360m and qwen2.5:1.5b through Ollama. The test cases are ninety-five questions about a fictional small winery, so nothing can be looked up and every answer can be judged by reading. Sixteen of the fifty classification messages are deliberately ambiguous, because real inboxes are mostly ambiguous.
Four verdicts, assigned per workflow, to force a result you can act on instead of a number: Unnecessary: deterministic code already does this. Sufficient: a small local model does it well enough to run unattended. Marginal: usable with a person checking the output. Insufficient: the error profile or the latency is not acceptable.
And one rule, which is the part I would insist on if you copy nothing else: I wrote predictions for every workflow and committed them before I installed Ollama. Twenty-two of them, each marked confident, leaning, or coin flip. Once there is a timestamped expectation there is a defined moment when you are done, and no room to decide afterward that you knew all along. I will come back to how they scored, because it is not flattering.
Find ORD- followed by six digits, or report that there isn't one. Thirty cases: twenty-two with an identifier, five without, and three decoys. The decoys are a membership number, a UPS tracking number and a phone number, all of which look like identifiers and none of which are order numbers.
| | regex | qwen2.5:1.5b | smollm2:360m | | --- | ---: | ---: | ---: | | Overall | 30/30 | 28/30 | 8/30 | | Declined correctly when absent (8) | 8 | 8 | 0 | | Decoys ignored (3) | 3 | 3 | 0 | | Invented an answer | 0 | 0 | 7 |
The regex won. Thirty out of thirty, in zero milliseconds, at zero watts above idle.
On ORD-999000: wrong address, please hold. it answered 999000, dropping the prefix. Annoying, recoverable, the kind of thing you fix with a post-processing step you should not need.
That is the shortest and least decorated input in the whole set. Almost the entire message is the order number. And the model returned a confident, correctly formatted assertion that there is no order number present. In a pipeline, that is a refund request for an order that exists arriving at a system that has decided it does not.
A regex cannot make that mistake. Not "rarely does" — cannot. The two errors separating 28 from 30 are both of a class that the deterministic baseline is structurally incapable of producing, which is a different and much stronger statement than a two-point accuracy gap.
By the contract, smollm2:360m scored 8 of 22 on the cases that contained an identifier. That reads like a model that cannot find things.
It is not. All fourteen of its wrong answers are the correct six digits with ORD- stripped. ORD-556120 came back as 556120, fourteen times out of fourteen.
It located the right identifier in 22 of 22 and formatted it correctly in Those are two different skills and the single score hides one of them. The model understood the task perfectly and could not obey the output contract, which is a completely different engineering problem than "the model is too small," and it has a completely different fix. (Constrained decoding. I have not tested it. It is the obvious follow-up and it is not in this post.)
The decoys follow from the same gap. It handed back MEM-443322, 1Z999AA10123456784 and 503-555-0142 verbatim, all three. It has no model of the format at all. It returns whatever in the message looks most like an identifier.
A model with perfect recall and no precision is not a bad extractor. It is a candidate generator that has been asked to do the wrong job.