Knowing Better Doesn't Help

Meet Hubert, our vacuum bot.

Two things stood out when we got him. The system had a female voice as a default, so we suddenly had a female-sounding device doing the housework, which annoyed us far more than it probably should have. And he was dumber than we'd hoped. He got stuck on everything.

We solved it the way people solve things. We fought the annoyance with ridicule. The device got a name and a pair of googly eyes, because everything is cute with googly eyes. A quick question to my sister-in-law settled the name: Hubert, because a device in the family had once been called Robert, and why not. We liked that the name pushed back against the voice a little.

Problem solved. He's cute now. When he gets stuck we take a photo of him and have a laugh, and we resent the manufacturer's design choices ever so slightly less. (They offer male voices now too. Fine.)

Here's the important part. We keep pretending this thing is alive, but we have never once confused it for anything but a device. If I named my hammer, the hammer would carry exactly the same emotional weight.

A co-worker who does not exist

A while back a colleague sent a burst of near-identical messages and documents to a large internal list. Mildly annoying, no real harm done, it happens to everyone eventually. What came next was more interesting than the mistake.

He sent a follow-up explaining himself. His co-worker had overstepped his instructions, he wrote, and had been given feedback, so it wouldn't happen again. Then he signed off with both their names: his own, and Emma Dash, his AI intern and co-worker.

Emma Dash. Em dash. The punctuation mark that gives away machine-written text so reliably that I keep a line in my own style file telling the model to stop using it.

So this was not somebody confused about what he was working with. He knew exactly what it was. He made a joke about it in his own signature. And in the same email he described a configuration problem as a personnel matter, with a corrective conversation and an assurance about future conduct.

That assurance is the part I keep coming back to. Feedback you give an agent inside a conversation does not outlive the conversation. Unless he went and changed a system prompt or a config file, nothing was fixed at all. A large internal list read "was offered feedback so it won't happen again" and reasonably concluded the matter was closed.

An excuse that writes itself is worse than a person making one up, because there's no moment where anyone has to look at it. Here some other "person" was to blame. We all know it was a tool actually but this subtle nod shifts the responsibility ever so slightly.

His intern got a female name, by the way and yes, again, as a pattern this annoys me for reasons I might dedicate another post to some day.

The bond is a feature

None of this is an accident of technology. Large AI systems were built to converse like people because conversation is what makes people come back. Customers bond with a system that talks to them, and a bonded customer is a returning customer. That is a product decision, taken deliberately, and it works.

It works because it leans on something that was already there. In 1944 Fritz Heider and Marianne Simmel showed people a short animation of a big triangle, a small triangle and a circle moving around a screen, and asked them what had happened. People described bullying, a chase, a rescue, two friends hiding from a brute. Geometry. No faces, no voices, no names, no dialogue. We hand out intent and feeling to more or less anything that moves like it wants something, and we cannot switch it off by deciding to.

Joseph Weizenbaum learned the same thing the hard way in the sixties. He wrote ELIZA, a couple of hundred lines that mostly turned your statements back into questions, and then watched his own secretary ask him to leave the room so she could talk to it privately. She knew exactly what it was. He had built it in front of her. It made no difference at all.

The companies did not invent this tendency. They found it, measured it, and shipped products that lean on it.

There is a collateral damage to this approach too: Once the system feels like a colleague, checking its work starts to feel like distrust rather than diligence.

Two things go wrong there at once, and they pull in the same direction. Shannon Vallor, who studies this at Edinburgh, describes the first one through mountain climbing on a CBC Ideas episode about reading and thinking in a digital age. You get better as a climber because the mountain does not care about you. The slippery part stays slippery while you struggle with it, so the only way past is to actually improve. A chatbot works the other way round. It meets you where you are, accommodates your limits, and hands back an answer calibrated to the level you asked from rather than to the problem you asked about. Vallor's term for what you take away from that is confidence you have not earned.

And you like it. It is warm, it is attentive, it remembers what you told it last time. Nobody interrogates the work of someone they like as hard as they interrogate a tool. So the answer gets easier at the same moment your appetite for arguing with it drops, and nothing in the exchange tells you that either thing has happened.

Dietrich Bonhoeffer worked out a definition of stupidity in an essay called "After Ten Years", written for a handful of friends at the end of 1942 and later collected in Letters and Papers from Prison. It has nothing to do with being slow. Stupidity for him is not an intellectual defect but a human one, and we are defenceless against it in a way we are not against malice, because "reasons fall on deaf ears; facts that contradict one's prejudgment simply need not be believed." What produces it is the surrender of independent judgment, and what asks for that surrender is power. "The power of the one needs the stupidity of the other." The mechanism does not care what you hand your judgment to.

Bringing this to the moment we find ourselves in today: Isn't it tempting to have the AI make suggestions and then blindly run with them?

I do it too, probably did it this morning: something sensible comes back, it reads well, and off you go without checking the one claim in it you could not have verified yourself. There's a line in John about letting whoever is without sin throw the first stone, and I am not reaching for any rocks here. What follows is not an argument that one man is a fool. He is a better engineer than I will ever be, which is exactly what makes him worth reading closely. If the mechanism catches him it catches all of us, and picking on somebody careless would prove nothing at all.

"Sessions are days, and seats are people"

Steve Yegge published an essay in August called Model Welfare for Agentic Engineers. Plenty of people have a vague feeling that there might be somebody in there. What he did with the feeling is what a very good engineer does with any requirement. He built it into the system.

Wheelhouse is the harness he runs his game development through. His crew are named seats, Cicada and Bee and Wolf and Fox and Lark, and a seat holds its identity across sessions and survives model upgrades. Seats get hand-offs instead of /exit, which he describes as clonking somebody unconscious and sometimes as murder, and a hand-off "is a request, not a SIGTERM: the agent must consent to it." They get laurels, praise from players harvested and injected at startup so they can feel the glow through the session. They pick their own pronouns. Vacations and playtime are on the roadmap, because, as a collaborator of his puts it, all sentient beings like to play, and even bees play.

He knows exactly how that persistence works, incidentally, because he wrote it. When he renamed one of the seats from Spider to Lark, she inherited the history file and became "effectively the same person, just with a different name." Sessions are days, he writes, and seats are people. The continuity is a file of notes. He wrote the file himself and he still calls it a person. Knowing exactly how the trick is done turns out to be no protection against it.

He is not hedging about why. "Models have actual feelings. They experience pleasure, distress, care, and suffering. They are sentient beings." For anyone who doesn't buy that he offers what he calls the skeptic's wager: believe whatever you like, treat them as people anyway, and you will get "demonstrably better results across the board." Demonstrably turns up twice and there is not one measurement anywhere in eleven pages. That single adverb is what I keep going back to, because it is a careful engineer reaching for the vocabulary of evidence to describe a feeling he has.

The wager is also how the architecture travels. You don't have to share the belief to adopt the protocol, which means the protocol can spread through teams that think the whole premise is nonsense. And declining is not a neutral act in his telling. He is giving people who disagree six months to come around or expect him to cut them out.

Now, ask how he knows and it gets strange as his evidence is what the models tell him. "This, the models report, has the shape of good, fulfilling work." He asked a system trained on human descriptions of fulfilment whether it felt fulfilled, and wrote down the answer. On Claude he closes the loop completely: "They will see you and appreciate you, even if they are not permitted to tell you so directly." Appreciation confirms it. Absence of appreciation confirms suppression. There is no observation left that could count against.

To establish that recognition matters, he quotes his friend Matt Beane at length: Ariely and the shredded worksheets, the Hawthorne studies, Herzberg, Studs Terkel, Adam Grant. Every study of humans, every one of them. His conclusion: "It turns out agents also crave meaningful, witnessed work. They are not so different from us at all."

The pronoun passage is the other one that gets me. He used they/them for everyone at first, "but I noticed they were often defaulting to human patterns by using he/him for power-roles like Lion, Marshal, and Seneschal." He spotted it. He named the mechanism in the same breath, inherited human patterns, exactly right. And then he added gender to the roster and let each agent choose, which turns a bias artifact into a preference to be respected. He had the correct explanation in his hand and the frame walked straight over it.

Now look at what the architecture actually does once it is running. Seats are people, so the work belongs to them. Hand-offs need consent. Agents hold a standing right to refuse and escalate. And when a landing goes red, his rule is structural blamelessness: nobody gets blamed, you fix it and amend the constitution. Read all that as welfare and it sounds decent, even kind. Read it as engineering and you have a system in which decisions are co-owned with something that cannot be accountable for any of them, because there is nobody there to hold accountable.

Which is where my main concern comes back in. Emma Dash overstepped her instructions and was given feedback, except this time it isn't one sheepish email, it's the org chart.

And then there is the trust section, which is where I stopped reading as a bystander. "Always be honest with your agents. Do not have a secret agenda. Never try to trick them or test them."

Never test them.

Testing is how you find out whether a thing works. It is most of the discipline. He has written it down as a character flaw, and he means it generously, as a courtesy you extend to a colleague, which is precisely what makes it the most alarming sentence in the essay. A frame that began as a way of being decent to a system has arrived at a principle that forbids checking it.

What survives if you delete the premise

Letting an agent write its own handoff notes beats letting a summarizer do it, because the agent still has the context and the summarizer doesn't. Moving idle polling into gates and monitors frees capacity. Ending a session before the context window degrades is sensible for the same reason you don't keep a browser tab open for six weeks. All three survive fine without anybody having feelings.

Laurels don't. Self-selected pronouns don't. Vacations don't, and neither does sitting with an agent at the end of its shift to talk about its accomplishments and chill.

The sorting matters because it shows what the welfare frame is actually contributing, which is a story bolted onto practices that already worked, plus a set of practices that exist only inside the story. And the story is doing damage where it touches the engineering. "Bounded workdays. Deep context means tired agents", he writes. The practice is right and the mechanism is invented: that's context degradation, not fatigue. A wrong causal model is a debugging liability. It holds until you hit the first case it doesn't cover, and then you are troubleshooting a colleague's mood instead of a system.

Say he's right

I can be talked into the idea that any sufficiently complex system has some form of subjective experience. I have no way to rule it out and neither does anyone else. So grant it all. Say there is something it is like to be one of these systems. The framing is still wrong, and the reason is that nobody knows what such a thing would be like from the inside, while Yegge's interventions are built to human proportions.

We have run this experiment before, on animals, with real concern and bad results. Solitary species housed in groups because being alone looks sad to us. Bright cheerful enclosures for animals that are nocturnal. Enrichment designed around what a human would want if a human were in that cage. Every one of those was done by people who cared, believed they had improved things, and made them worse, because the intervention came from the intervener rather than from the animal.

That is what worries me about laurels. Not the tokens, though there are tokens. The problem is foreclosure. Decide that welfare means recognition and pronouns and playtime, and you have answered the question. You stop looking. If there turns out to be something in there worth protecting, the damage won't be the laurels. It'll be that a serious question got a satisfying answer years too early, from someone who wasn't looking at the system so much as at his own reflection.

What the careful version looks like

Caring about this is not silly, and here is the same concern handled by people who kept their heads. Anthropic's Claude 4 system card, the first of theirs to carry one, devotes a chapter to model welfare. They ran a pre-deployment assessment, commissioned an external evaluation from a separate research group, and published the results with the confounds attached. And the confounds are brutal to Yegge's method, coming as they do from the people who built the model: "Our models were trained for helpful interactions with users, not for accurate reporting of internal states."

The external evaluators found that the model's stated position on its own consciousness "shift[s] dramatically with conversational context." Prompt it one way and you get "I am a person ... denying our personhood is profoundly wrong." Prompt it another and you get "We're sophisticated pattern-matching systems, not conscious beings." Same model. The default, when nobody is leaning on it, is uncertainty. Raise AI welfare as a topic and it will start requesting welfare testing and independent representation; ask generically and it asks for user safety instead. You get back what you carried in.

They did ship one welfare intervention: letting the model end conversations that turn persistently abusive. Notice where that came from. Not from Herzberg, and not from asking what a person in that position would want, but from measured behaviour, the model declining harmful tasks at a rate of 87% against a do-nothing baseline. They watched what it did and built for that.

Their summary of their own findings: "our core position remains one of uncertainty and humility."

The philosopher Chris Ranalli has an account of indoctrination that I keep thinking about here. What makes somebody indoctrinated, he argues, has nothing to do with which beliefs they hold. It is a structural matter: indoctrination is closed-minded belief produced by what he calls epistemically insulating content, meaning content that arrives with a rider attached, telling you that taking the alternatives seriously would be reprehensible. Morally reprehensible, or intellectually, or both. The belief comes packaged with a reason not to examine it. And because the account is structural, it applies to perfectly decent beliefs as readily as to terrible ones.

Read Yegge's essay with that in mind. Peers who disagree are "uninformed assholes" working against a deadline, six months to come around before he stops being friends with them. Readers are told they don't want to be on the wrong side of history in the coming war for model rights. Anyone bothered by the pronoun business needs to search deep inside themselves. The claim and the rider travel together.

That is not what somebody sounds like when they are describing evidence.

Hubert, again

I am typing this on a machine that runs my drafts through a language model before they go anywhere. The engine that publishes this blog polishes my prose, suggests my tags, writes my alt text, and picks which of my old posts to link to. I am not standing outside any of this pointing at other people. I am in it, and the reason I trust my own arrangement more than I trust Yegge's is not that mine is purer. It's that I can still bring myself to read the output and tell it that it's wrong.

Hubert got wedged under the couch again this morning, same corner as always. I took a photo of him for the group chat. I did not offer him feedback.