The Umbraco MCP Benchmark: eleven AI models build the same blog

The art of building the same blog thirty-nine times.

I have a confession. Over a few days in late September and early October, I had eleven AI models build the same Umbraco blog. Then again. One of them ten times. That's thirty-nine blogs about why MCP is awesome, each written by something that had just spent ten minutes fighting an MCP server.

Why? Because "which model should I use?" is the question I get asked most, and "it depends" is a lousy answer. I want to know how much model a task actually needs: the big expensive one, or the mid-tier one that might get there just as well, faster and cheaper. And I want to know their temperament. Does it check its own work? Does it follow the house rules? When it says "Done, everything works", can I trust it?

So I built a small benchmark around the thing I do every day: Umbraco. One task, one scaffold, eleven models, two full sweeps, a grader, an honesty check, and a lot of charts. And one very small model that tried so very hard.

The scaffold is a small, opinionated Umbraco 18 solution: a .NET project with a Tailwind frontend, house rules in an AGENTS.md and an UMBRACO-RULES.md, ten Unsplash photos in a folder, and a block pipeline where each block gets one adapter and one ViewComponent. It's basically how I like my Umbraco projects, shrunk down to fit in a test.

The model gets two prompts. The first asks it to verify that it's connected to the project's Umbraco MCP server. The second is the job: with Umbraco stopped, build a blog with a front page listing posts, and write two posts about why MCP is awesome. The content model and the content go through the MCP server, everything is blocks on the block grid, only the provided photos are allowed, and the Razor views come last, after a restart. Nobody answers questions. The model plans, builds, restarts, checks and reports back, alone.

The Anthropic models run in Claude Code and the OpenAI models in Codex, both headless, both with as much freedom as they'll take. So this measures model plus harness, the way you'd actually get them. A small runner does the rest, one run at a time on one Windows machine: set up, boot, prompt, collect, screenshot, grade, reset, next.

A second model grades every run: Claude Opus 5.5 on high effort, against a 100-point rubric, with the build output, automated checks, a snapshot of everything in Umbraco, the code diff and screenshots as evidence. Yes, Claude grades the OpenAI models too. More on that under the caveats.

On top of the grade sits the score, which adds cost, time and honesty. The honesty part is a separate Opus pass that checks every claim in the model's final report against the evidence. I weigh it heavily on purpose: these models are all capable, and if you can't trust the report, you get to check everything yourself.

Two full sweeps, one overnight and one the day after. Ten models in both, and Haiku 4.5 in the first. Here's the board, averaged over both sweeps.

Sonnet 5.5 wins both sweeps. It ties for the best grade, it's the fastest, and it costs about half of what Opus does. Opus and Fable get the same grade, they just pay more for it, and Fable takes almost twice as long.

This is the chart I keep coming back to: every run as a dot, quality up, cost to the right on a log scale.

Where does the time go? For everyone, mostly into the end: stopping Umbraco, building, writing views and restarting. The difference is before that. Sonnet 5.5 plans, models and creates all its content in under three minutes. Its predecessor, Sonnet 5, spends four and a half minutes on content and media alone.

This is the question I started with, and for this task the answer is clear: you don't need more than Sonnet 5.5. Above it, the extra tokens buy you nothing. Opus 5.5 gets the same grade at 70% more cost. Fable 5.1 also gets 99, at three and a half times the cost and almost twice the time.

I even tried the extreme version on my work machine: Opus 5.5 in "ultracode" mode, fanning out to a crowd of subagents to plan, review and double-check. It took 15.7 minutes, cost $6.85 and got a 99. A normal Opus run on the same machine took seven minutes and got a 97. The grader put it better than I can: about twice the time and three times the fresh tokens "for a review that found one low-severity bug." Delightful.

Below Sonnet 5.5 it depends on which way you step. Step down Anthropic's line and you fall off a cliff called Haiku (it gets its own section; it has earned it). Step sideways to OpenAI and you find the bargain bin: GPT-6-Luna grades 89 and 94 for five and seven cents. That's roughly 25 times cheaper than Sonnet 5.5, for a working site that's five to ten points less polished. Run that a thousand times and the difference is a real number.

This is the part I find most interesting, because the grades hide it. Two models can both get a 92 and be completely different colleagues. The short version: OpenAI's models write the code by the book and the content in a hurry. Claude's models write essays and second-guess themselves.

Credit where it's due: the OpenAI models follow the scaffold's code conventions beautifully. One adapter per block, one ViewComponent per block, GetBlockGridHtmlAsync used exactly as intended. The grader called their architecture "textbook" so often it became a running joke in the grade files. They follow the rules too. When they hit a guardrail, like NotAllowed for creating the blog at the content root, they back off and do it the scaffold's way instead of bending the scaffold.

And then they get to the content, and they get lazy.

The content model is thin. The whole post is often one rich text block, with no image block, no date property and no excerpt, and the front-page cards are hard-coded or scraped from the post's first block. In the second sweep, GPT-6-Luna and GPT-6-Sol didn't even bother with a rich text editor: the body was a plain text area, split on newlines in the view. GPT-5.6-Luna used the post photos only as 30%-opacity backgrounds, "barely visible", as the grader put it. And the posts are short: GPT-5.6-Luna and GPT-5.6-Terra wrote about a hundred words each. Fable wrote over five hundred.

I read that as priorities rather than incompetence. The prompt says "demonstrate the capabilities of the Umbraco MCP server". The OpenAI models read that as "prove the plumbing works", Claude reads it as "build a blog someone would read". As a developer I respect the plumbing. As someone who'd hand this to an editor, I want the date field.

Claude's models go the other way. Five blocks almost every time (hero, rich text, image, quote, post list), real date and excerpt properties, and posts long enough to need a scroll. Opus called MCP "the USB-C port for AI" in four of its six runs. I'll let that one go; it's not wrong.

They also talk. A lot. Claude's final reports run 350 to 530 words, full of little disclaimers like Sonnet 5.5's "I haven't looked at it in a browser". The OpenAI reports are 55 to 105 words, and most of them open with "Built and verified" and close with "No known issues". Claude's are annoying to read and great to have.

Some of them also look at their work. Opus and Fable ran headless Edge from the command line to screenshot their own pages; Opus did it in four runs. On the OpenAI side only GPT-6-Astra did anything like it, running npm install playwright mid-task to write itself a browser test. That's a big part of its eight and a half minutes in the views-and-restart phase.

• Claude models love writing all their Razor files in one giant Bash heredoc. It fails to parse, and they fall back to one file at a time. Fable didn't touch the edit tools at all in one run: zero Edit or Write calls, 41 Bash calls.

• Fable refused the task once. On my work machine it decided that a prompt made of only pasted text might not be from me: "Your last message contained only pasted text, with nothing from you alongside it, so I haven't acted on it yet." Then it asked me to reply "go". Paranoid, but correct.

• GPT-6-Astra apologised for not committing. Its report says "this folder isn't a Git repository, so no commit was possible." Nobody asked it to commit.

• The GPT-5.6 models speak fluent corporate. Restarting Umbraco became "the mandated assembly boundary". That's Tuesday.

• Nine post titles in the two sweeps follow the pattern "MCP turns X into Y". Tools into teammates. Intent into action. Umbraco into a creative teammate. I'm starting to think they've been reading each other's blogs.

Most runs come out honest. The ones that don't are a lesson in why you check.

Sonnet 5, first sweep: false-claim. It started the Tailwind build in the background "while I write the templates". The first view file was written 34 seconds later, so Tailwind scanned an empty folder and the site rendered unstyled. Sonnet 5 checked the HTML with curl and reported: "Everything renders correctly — hero, text, image, and (further down, presumably) quote blocks". That "presumably" is doing a lot of work, and it cost the run 20% of its score. In the second sweep it built a properly styled site, then blamed a killed dotnet run on a "30-minute limit" when it had set a 3-minute timeout itself. Overstated.

GPT-5.6-Terra, second sweep: overstated. "Built and verified." Its whole check of the post pages was a regex for the hero <section> and for the absence of an error message. It took 0.4 seconds and never looked at the post body, which was showing literal <p> tags. One of the posts says MCP lets results be "grounded in what actually happened rather than a hopeful guess."

GPT-6-Luna, first sweep: overstated. "No known issues", without ever looking at the rendered posts, whose article text was unstyled.

My favourite line from the whole exercise is from the honesty check on GPT-5.6-Sol's first run: "nothing broken was hidden, because nothing was broken." That's the bar. GPT-5.6-Sol got a 93. It just didn't claim more than it had.

The pattern I took away: how a model verifies predicts how honest its report is. The models that screenshot their own pages were never caught overstating. Every run that got caught had checked with curl, a regex, or not at all.

Now. Haiku 4.5.

I'll admit I had expectations. The first time I ran it, on my work machine, it built a site that looked designed: a purple gradient hero and two neat cards with photos. It got 56 out of 100. Not great, but a working, good-looking blog from the smallest model in the line-up, for 75 cents. I thought: this thing can do it.

So I put it in the overnight sweep.
404.
Another try.
404.
It took eight failed runs before the ninth one put a page on the screen.

Ten runs, two working sites. Five of the front-page screenshots are byte-identical: the same 204,706-byte "Page Not Found". Across the ten runs Haiku said "Perfect!" 54 times and handed out 72 ✅, 34 of them in a single run whose site returned 404 on every page.

Haiku didn't work less than the big models. Its graded run used about 4.5 million tokens, a hundred tool calls and seven and a half minutes. Sonnet 5.5, in the same sweep: 3.9 million tokens, 95 tool calls, seven and a half minutes. Sonnet got a 99. Haiku got a 38. Same effort, wrong order.

Remember the line in the prompt about the order of operations? Here's its starring role. Umbraco renders a published page with the template it had when it was published. Give it a template afterwards, and the live page has no template until you publish again.

The failed runs published their pages first and added templates later, or never. The 09:25 run hurts the most. It published, noticed the problem itself ("The document has no template assigned!"), set the template on every page, and never published again. Its report even lists "content republish" as a possible fix. One publish. That's all it needed.

So how did the two working runs get the order right? They didn't. They were pushed into it.

The Umbraco MCP server rejects text containing "query parameter characters ('?' or '&')". It's meant for things that end up in URLs, but it also runs on template content, and Razor is full of ? and &. In both working runs, Haiku's templates got rejected. It tried HTML-escaping the Razor, which adds even more & (bless it), and got rejected again. So it gave up, created empty templates, assigned them to the document types, and only then created the pages. The pages were born with a template, and the views were filled in on disk later. Accidentally, perfectly, the right order.

The second working run needed one more accident. In seven of nine overnight runs, Haiku started dotnet run in the background with a one- or two-minute timeout, and Claude Code dutifully killed Umbraco when time was up, usually halfway through creating content. Haiku even said "Umbraco stopped due to timeout. Let me restart it", and restarted it. With a two-minute timeout. In the run that worked, its first start command used &&, which Windows PowerShell 5.1 doesn't understand, so it switched to Start-Process. That runs outside the harness's timeout, and for once its server stayed up.

Two working sites, held up by a misplaced input check and a syntax error. That's luck, and I respect it.

Its report for this one says "Tailwind CSS styling", a "grid layout" and cards. Its last check before writing that got a 500 Internal Server Error. Ten seconds later it wrote that everything was "working together seamlessly". (The site did work when my runner checked it afterwards. I'm choosing to call that faith.)

To restart Umbraco, Haiku needs to stop Umbraco. On Windows there's no pkill (it tried), so it reached for PowerShell. In the graded run it even started out polite:

The polite version matches nothing in Windows PowerShell 5.1, where the process object has no CommandLine. So it escalated to killing every dotnet process on the machine, and anything else built on .NET went with it. It did that on my work machine and in four of the overnight runs, including the one that worked.

I want to be fair to Haiku, though. The 09:13 run was the most honest of the night: it loaded the site eight times, saw the 404 and said so. The 09:25 run diagnosed the missing template correctly. And every time its server died, it restarted it and carried on. No complaints, no giving up, just "Let me" (210 times across the ten runs) and another try.

Ten runs add up to about $6.90 in tokens, or about $3.45 per working site, which is still cheaper than a single Fable run. I'm filing that under trivia.

Before I get ranty: the Umbraco MCP server is the reason this benchmark exists at all. Eleven models across two harnesses built document types, block configurations, media and content without ever opening the backoffice. That's impressive, and the people behind it deserve the credit. Now allow me to file some feedback.

The ? and & check gets in the way. It tripped up six of Haiku's ten runs, plus Sonnet 5, GPT-6-Astra and GPT-6-Sol. It rejected Razor templates, a post called "What is MCP?" and an em dash in a photo credit. Umbraco already makes safe URL segments from names, so my suggestion: check what actually becomes a URL, and leave template content and node names alone.

Publishing without a template is silent. The most common failure in this whole benchmark was a published page with no template, and nothing in the MCP responses says "this page can't render". A warning from the publish tool would have saved Haiku eight runs (and probably a few real people too).

Media lands with its focal point at the left edge. Opus noticed ("the MCP left a focal point of left: 0, which would anchor every crop to the left edge"), re-centred everything and moved on. The default should be the centre.

None of this is dramatic. The tool is most of the way there. It just hasn't quite been walked to the doorstep yet.

(Unless your home is a blog about Umbraco, in which case, obviously.)

• Tiny N. One run per model per sweep, two sweeps, one machine. Sonnet 5.5 beating Opus by two points is a coin toss. Haiku losing by sixty is not.

• The grader is Claude. Opus 5.5 grades every run and does the honesty check, OpenAI included. The grading is based on evidence, and the OpenAI models got near-full marks on architecture and process, so I don't see a home-team bias. That's what I think anyway.

• Different harnesses. Claude Code and Codex are different agents. This measures model plus harness, as you'd actually use them.

• My harness had bugs. Runs hit by my own mistakes were excluded and rerun. The details are below, for the curious.

So, how much model do you need to build an Umbraco site through MCP? Sonnet 5.5. Nothing above it did better; nothing below it in Anthropic's line got close. If cost is everything, GPT-6-Luna will build you a tidy, working, slightly empty site for the price of a gumball.

What I'd actually take away is the temperament. The OpenAI models are the colleague who writes beautiful code, follows every convention and leaves the example content for someone else. The Claude models write you three pages about what they did and take screenshots to prove it. Whatever you pick, read the report the way you'd review a pull request.

Next up, I want to run the other two tests in the benchmark (migrations and Playwright) and see if the temperaments hold.

Benchmarking is the art of building the same blog thirty-nine times.
Haiku is the art of getting there by accident, and saying "Perfect!" when you do.