Some context first, because the test only makes sense if you know what I was building toward. I wanted a blog engine that supports my writing instead of replacing it. Not a machine that takes a topic and hands me back an article, but something that takes what I already wrote and helps it over the finish line. On top of that I had two preferences I wasn't willing to trade away: privacy-first and local-first, leaning on open technology wherever I could. Ideally the thing that touches my half-formed thoughts runs on my own machine and never phones home.
That goal is what turned "which model is best" into an actual question worth testing, because "best" depends entirely on the job. So I ran two experiments. First: how do the various models cope when the input is bulleted, fragmented notes versus a finished draft? That's the real fork in how I write - some days I hand it clean prose, some days a pile of stubs. Second: how does the prose itself differ model by model, once you strip out the effect of the input? I wanted to know each model's fingerprint, not just a leaderboard.
Notes below from testing the polish step across 8 configurations (7 remote models + local gemma4:e4b) over the blog's real drafts. What I learned.
The prompt every model ran
One thing worth being explicit about: every model got the exact same prompt. No per-model tuning, no coaxing the weaker ones with extra hand-holding. That was the point - I wanted to see the model's own behavior, not my ability to prompt-engineer around it. If I'd tweaked the prompt per model I'd be measuring my patience, not the models.
The prompt frames the model as "the author's editor" and hands it my rough notes. The load-bearing part is a rule I put in caps and at the very top: do not lose ideas. Not "summarize," not "improve" - present all of my thinking, every distinct point, and reuse my own vivid phrasing rather than paraphrasing it into mush. I spelled out that dropping or watering down an idea to make the text shorter or smoother counts as a failure, because that's exactly the "helpful" instinct I was trying to suppress. The model is allowed to reorder for flow, merge fragments into sentences, and add light connective tissue, but it may not invent new claims.
Two more blocks round it out. A voice block pastes in my writing guide verbatim - including the list of words and phrases the model must never use - so "in their voice" isn't left to interpretation. And a format block asks for clean Markdown, translates any German into natural English (I draft in both), and insists on the article body only, no frontmatter or preamble.
Here's the whole thing, with the two injected values marked in braces (my voice guide gets pasted into the first, the raw draft into the second):
You are the author's editor. The text below is the author's own rough notes for a blog post. Turn them into a finished, flowing article in their voice.
[MOST IMPORTANT RULE - DO NOT LOSE IDEAS]
The notes contain the author's actual thinking. Your job is to present ALL of it, not a summary of it.
- Keep every distinct point, argument, example, and section. If the notes make five points, the article makes five points.
- Keep any vivid phrase, metaphor, analogy, or joke the author used. These are the best part - reuse the author's own wording, do not paraphrase them away.
- Dropping or watering down an idea to make the text shorter or smoother is a FAILURE. When in doubt, keep it in.
- You may reorder for flow, merge fragments into sentences, and add light connective tissue, but do not invent new claims or facts.
[VOICE]
Write the way the author writes. Match this guide exactly, including the list of words and phrases you must never use:
{voice_guidelines}
[FORMAT]
- Clean Markdown. Translate any German into natural English.
- Do not output frontmatter, a title, JSON, or any preamble. Output ONLY the article body.
[AUTHOR'S NOTES]
{raw_draft}
The interesting design consequence: this single prompt is what makes the whole thing auto-scale. Because rule #1 is "preserve the author's wording," a finished draft leaves almost nothing to do and every model just tidies, while a bare outline forces the model to actually write. I didn't build separate "copy-edit mode" and "draft mode" prompts - the same instruction produces both behaviors depending on what I feed it. The bake-off was partly a test of whether that hunch held up. It did.
The core finding - divergence tracks draft shape, not model
How much the models' outputs differ from each other is driven almost entirely by how finished the source draft is, not by which model ran. Measured across 6 posts, flow-similarity lined up monotonically with the draft's bullet-ratio: prose drafts converged (57 - 98%), while a pure outline diverged wildly (3%).
The mechanism is simple. The polish prompt's #1 rule is "preserve the author's wording." On finished prose there's nothing to invent, so every model just tidies and they land in the same place; on bare bullets each model writes its own prose, so they scatter.
This means the system's aggressiveness auto-scales to how much you write - hand it prose, it copy-edits; hand it bullets, it drafts. You control the dial just by draft finishedness, no setting to touch. This is exactly the behavior I want.
The "outline test" is where model choice actually matters - that's the column set worth studying when picking a model, because that's where the model is writing rather than tidying.
As a side note, hello_world's 98% similarity was a bit of a degenerate data point. Its "draft" was already a finished, published article (title/date/tags and all), so every model handed it back ~98% unchanged. It wasn't a case of "models agree" so much as "nothing to polish."
Genuinely model-specific tics (survive after stripping out format effects)
- Mistral: emits em-dashes (7 across 6 posts vs ~0 for others) even when instructed otherwise.
- Sonnet: emphatic register - heavy bold (10× vs ≤3), and "genuinely" (4×). Rewrites openings more, but mainly on short/outline drafts.
- DeepSeek: the boldest re-framer on prose input (threw out the opening on engine-update and pulled a buried line to the top); also slowest remote model, especially on long drafts (181s on the 996-word one).
- e4b (local): genuinely good on prose - preserved all ideas, no banned words, recognizably the right voice; only slip was occasional em-dashes. On pure outlines it ran ~10 - 15% wordier than Opus and added filler.
On e4b as a daily driver
Similarity measures agreement, not quality - so the verdict came from reading outputs, not the numbers.
My verdict is that e4b is defensible as a daily driver specifically because of the auto-scaling property. On prose the quality gap to large models nearly vanishes; it only opens on bare outlines. So: prose → e4b is great (local, private, free); outlines → consider a remote model.
Where big models win is tighter prose (they cut instead of padding), better paragraph structure, and zero style-guide violations.
The takeaways that actually changed my mind
A few things landed hard enough to shape what I use:
Big models are overkill for this job. The whole task is "tidy my prose without losing my ideas," and that ceiling is low - once a model can do it cleanly, a bigger one doesn't do it more. On finished drafts the giants were spending their capability on a problem that didn't need it. That reframed the whole decision for me. I stopped chasing the best model and started looking for a good-enough one I could run as close to home as possible.
Mistral is a serious contender. Setting aside its em-dash habit (which I strip mechanically anyway), it turned in genuinely good prose and was quick about it. If I were picking a remote model purely on the writing, it'd be near the top. Worth keeping in mind for the outline path where model choice still matters.
gemma4 is the real surprise. This is the one that made me rethink the setup. A small model running locally on my own machine held its own against remote giants on the writing - and then, remembered, it's multi-modal! It writes my alt text too, describing real photos coherently, image bytes never leaving the machine.
See it for yourself
You don't have to take my word for any of this. I put the whole comparison online: every source draft, and side by side, what each of the nine configurations did with it. Read across a row and watch the prose drift apart on the outline drafts and snap back together on the finished ones - that's the core finding, right there in front of you. If you like to see an example where the source was just bullets, select "smartphonefree" or "AICare". These are the articles with most variety in results.
Have a look: https://unprompted.dirkprimbs.de/viewer.html
Comments
Replies come from Mastodon. Reply to this post to join in.