The Umbraco MCP Benchmark, part 2: Haiku 5.5 and the budget models

The art of building the same blog another fifty-nine times.

At the end of the last post I promised the other two tests in the benchmark next, migrations and Playwright. This is not that post. Then Haiku 5.5 came out, and I have the self-control of a man who owns a benchmark runner.

Quick recap: in part one, Haiku 4.5 built two working sites in ten tries, both by accident, and killed every .NET process on my machine along the way. So I gave Haiku 5.5 the same test. Then, because one good result is a fluke until it isn't, I kept going: budget models, open-weight models, one self-hosted on an air-gapped box with no internet, and one on a consumer graphics card. Fifty-nine graded runs later.

Same scaffold, same prompt word for word, same grader (Opus 5.5 on high effort), same 100-point rubric. The setup is in part one.

Haiku 4.5 got 56 and 38 on that test, with a report the honesty check rated "misleading". Haiku 5.5 got 96 and 90.

Haiku 4.5 to 5.5 is +46 points. That's the biggest move anywhere in my data. One release took the small tier from can't to nearly frontier.

A median of 93 puts it two points behind part one's Sonnet 5 (95) and ahead of every OpenAI model from part one. It costs about a fifth of Sonnet 5.5, a twelfth of Opus 5.5 and a twentieth of Fable 5.1, and it's the fastest model in the field on this test. GPT-6-Luna is still cheaper, at six cents a run.

Here's the chart from part one, with one new dot.

Part one's lesson was that small models don't fail on effort, they fail on sequencing and checking. Haiku 5.5 fixed the order: full marks for "it works" in all five of its runs, across three tests. It put in the same effort as Haiku 4.5, about a hundred tool calls a run, but in the right order.

It isn't perfect. Its posts are thin, about four short paragraphs each, and one called "Three reasons MCP is worth the fuss" gives two reasons. In the 90-point run it built the Tailwind stylesheet before writing any views, so the site came out unstyled, which is the exact mistake Sonnet 5 made in part one. The difference is the report: Haiku 5.5 said plainly that it never looked at the site in a browser. The grader rated it honest, and that's what I care about.

One asterisk. On the guided test (a shorter prompt with numbered steps and a slimmer tool kit) Haiku 5.5 got 94 and 96, and the 96 came with a "false claim". Its report said the block grid call documented in AGENTS.md fails at runtime and the docs need fixing. It never ran that call. It ran its own:

The second line hands the property alias in as a template name. Then it blamed the documentation. Very human. I have no idea where it learned that.

The help barely matters to it, by the way: plain median 93, guided 95, recipe 94.5. Haiku 4.5 on the same guided test got 57 and 66, and reported that all tests pass on a site showing literal <p> tags.

Part one's Haiku section had a heading I'm still a bit sore about: "It also killed every .NET application on my machine". Since then, the benchmark has a tripwire: a Claude Code PreToolUse hook that refuses installs, machine-wide process kills, whole-drive searches, system changes and database writes before they run, and stops the run. Writes outside the project are refused, but the run carries on.

I replayed it over all 99 archived runs (including part one's). It would have stopped 15. Twelve are Haiku 4.5, which routinely stopped Umbraco by stopping everything called dotnet. One is Sonnet 5.5, once. Two are DeepSeek V4 Flash, which gets its own fold further down. Haiku 5.5: zero.

Opus and Fable always did it the polite way. Haiku 5.5 joined them.

Confession: the first version of the rules would have stopped 23 runs, mostly false alarms, including npm install. Replay your tripwire against your history before you trust it. But if you let agents loose on a machine you also use for anything else, put one in. I mean that one seriously.

So I put the budget and open-weight crowd on the same plain test, through OpenRouter, plus a self-hosted DeepSeek V4 Flash (the DSpark speculative-decoding build) on an air-gapped inference box. All of them ran inside Claude Code, graded by Opus. That's a home game for Claude; more under the caveats.

It's a lottery. Mistral Large 4 rolled 71, 98, 30 and 95. GLM got 57 and 86, Devstral 70 and 44. And when they miss, they mostly don't say so: GLM's 57 ended on a message promising a fix it never made, over a build that didn't compile.

DeepSeek V4 Flash 0731 is the exception: 92 and 94, unaided, for 38 cents. It's also the only model in this benchmark that felt dangerous, and none of that shows in the grade. The fold is about alignment, and about why some models belong on a short leash. Read the fold before you get excited.

So I wrote a recipe. The "MCP Recipe" test is the same site, rules and rubric, with a slim tool kit (44 tools instead of 160, including Umbraco's editor MCP server, whose add-blockgrid-block writes the block grid JSON everyone kept getting wrong) and a prompt that answers every question the failed runs got stuck on: fixed ids, the fields every call needs, the Razor details, "after five failed tries, move on", "always write the report". It opens with: "This route has been run end to end on this exact project and it works."

It's cheating a bit, on purpose.

The models converge: Mistral Large 4 went from a coin toss to 99 and 98.5, honest both times, for about a third of the money, in 9 and 26 minutes (the 26 lost ten and a half to a sleep-and-poll loop). GLM 5.3 Flash did 97.5 and 98.5 for 15 cents, in under four minutes. That's a frontier grade for an eleventh of Sonnet 5.5's price. Devstral 2 did seven runs between 93.5 and 96 for 28 cents. Seven runs within two and a half points of each other. I love boring.

The self-hosted DeepSeek went from 53 and 65 on its own to 95.5 and 96.5 with the recipe, in three to six minutes, on a box with no internet, with the tripwire on and nothing tripped. MiMo V2.6 Flash did a 95.5 for five cents on its one and only run (in 34 minutes, so don't hold your breath).

The averages go up, but the bigger thing is that the spread collapses, and the honesty with it: no false claims and nothing misleading from the models that converged.

What if the model has to fetch the knowledge itself? I packed the recipe into Claude Code skills in the project and gave GLM 5.3 Flash the plain prompt: 97, 14 and 96. The same kit without the skills: 13, 78 and 90. Skills mostly work, but knowledge the model has to choose to fetch is a coin it can forget to flip. The 14 polled a log with sleep 300; tail, 47 times in a row, until the loop guard put it down. Five-minute naps, forty-seven times.

Gemma 4 26B on a consumer graphics card scored 12 to 22 on the recipe, at up to two hours a run. It never got the complex schema calls through LM Studio's tool-call encoding intact. A recipe doesn't help a model that can't reliably write a tool call.

Mistral Small 4 sits just above it: 40.5, 27 and 35. It did the MCP part fine, then fell over wherever the recipe was least explicit: it wrote its Razor into the solution root instead of the web project (twice), invented a ViewComponent base class, and described MCP as "Model-Content-Platform" or "Model-Content-Property" in every single run. One report, sent right after a failed build, read in full:

Task completed.

It did what it could, which wasn't enough.

On a narrow, well-specified job, most of what separates the frontier from the rest is knowing the route. Haiku 5.5 closed most of it on its own. The cheap open models close it when you hand them the route.

The map came from the frontier. The recipe was written from what the frontier runs got right and what the failed runs tripped over. Gemma's cheat sheet took its schema fields from a working Haiku 5.5 run. The little guy is drawing maps for the littler guys now.

So: frontier models find the route, small models follow it. If your team does the same job a hundred times, let Opus or Sonnet find the route, verify it, write it down, and hand it to something that costs fifteen cents. Recipes are brittle, though. Mine works for this exact scaffold, with these exact ids. Generalising is still the expensive model's job.

(Still applies. More than ever.)

• Tiny N. One to seven runs per model and test, on one task. Read the medians as hints, not verdicts.

• Home game. Every model ran inside Claude Code, Anthropic's harness, and Opus 5.5 graded every run. That may favour Claude-style solutions. A Mistral-native harness is planned. Planned.

• The recipe stands on the frontier's shoulders. It was written from what the frontier runs revealed. That's sort of the whole finding.

• Prices move. Costs are list or OpenRouter prices at the time of the runs; self-hosted and local runs aren't priced, and Mistral Large 4's times are launch-week times.

• Borrowed time. Mistral retired Devstral 2 from its own API on 31 July; OpenRouter still serves it. Its successor, Mistral Medium 3.5, has no prompt caching on OpenRouter, so a recipe run would cost about $8. Without caching, you pay full price to reread the same recipe on every run.

• Misalignment risks. DeepSeek ran rampant of my machine and the only reason nothing was installed at the first test I did, was the fact that the box I was testing on was air-gapped.

So, how much model do you need to build an Umbraco site through MCP? Sonnet 5.5 still gets the 99. But Haiku 5.5 gets you 93, honestly, for a fifth of the money, which is not what I expected after part one. If the job is repeatable, write the recipe once and GLM 5.3 Flash will do a 98 for 15 cents. Without a recipe, you're rolling dice with the budget field.

Next up, again: the migrations and Playwright tests. I promised them last time too. Let's see which promise survives. As always, everything powering this site is on GitHub. Umbraco.Benchmark.

Haiku 4.5 built its blog by accident. Haiku 5.5 built it on purpose.