Let's think step by step

Back then it was still a pro tip to tack "Let's think step by step." onto a prompt. The models would then try to work their way to an answer instead of just writing it down, and the results got measurably better. I had picked up that idea and added three thoughts of my own.

First, I wanted to separate the thinking from the actual result and output it on its own. That way you'd see which intermediate steps led to the answer.

Second, an idea from my psychology studies came into it: inner speech, the conversation with ourselves that we use to plan, correct ourselves and ask ourselves questions. Lev Vygotsky described it as internalised language: conversations we have with others as children and at some point start having with ourselves. And on top of that, the functional architecture of our brain. I've always been fascinated by the idea that you can think of the brain as a collection of specialists. A room full of rather narrow-minded individuals who are constantly talking to each other. Today we'd call them "agents", but when I started the thesis nobody was talking about agentic AI yet. Marvin Minsky described this picture as the Society of Mind back in 1986.

The hunch behind it: a group of specialists talking to each other gets better results than any one of them alone. And if you can follow that conversation, you get explainability for free. Instead of having to simply accept an answer, you can read up on where it came from.

Third, and this one comes from the AI safety corner: maybe the strongest model doesn't always have to be the one at the wheel.

In today's vocabulary I'd say my thesis built an agent harness with a thinking trace before harnesses, agents or thinking traces were a thing.

What I actually wanted to build

My first plan was a real conversation. Independent agents were supposed to discuss freely until an answer stood. I tried it. With the small, locally hosted models I had back then, the answers got worse. The frameworks for free multi-agent orchestration were simply unreliable with models that small. The free discussion ended up in the "Future Research" chapter.

So I took a step back and built a pipeline. Every request went through fixed stations: first it was categorised (maths, narrative, explanation or analysis), then analysed and planned, a reflection stage was added when needed, and only at the very end was the actual answer written. Each station got a model that suited it, a small maths model for arithmetic, for example. Everything ran locally through Ollama, everything in the console. An acquired taste. I liked it.

Then I threw a huge pile of prompts at the system and compared its answers to direct answers and to chain-of-thought.

What came out of it

The trace delivered what I'd hoped for. You could read how the system had categorised and planned a task and which intermediate steps led to the answer. A direct answer has none of that. Chain-of-thought at least gives you a monologue, but you're never quite sure whether it really describes what the model did.

Whether the answers themselves got better depended on the scenario. Sometimes the division of labour clearly helped. Sometimes a task got put in the wrong category and the mistake carried all the way through to the last stage.

The "Oh sh*t" moment didn't last long, by the way. The thinking traces of the big models are one model talking to itself. My trace was the record of a division of labour where you could see who contributed what. The examiners apparently saw the difference too: I passed my master's with Distinction :-)

These days you see the principle everywhere. In modern harnesses and agent loops, groups of specialists push each other further. A new harness suddenly gets existing models to do things nobody thought they could. Compared to that, my system was a proof of concept and ultimately a toy. But one that was a little early.

A brain in a tank

The idea never really let go of me. What if the agents did discuss freely after all, like in a chat room? How do you model what the system is paying attention to and when it loses interest? What about memory? And isn't a system like that, with no body and no senses, exactly the brain in a vat from Hilary Putnam's thought experiment? Mine just happens to live in a tank.

So I sat down with it again. This time without academic ambitions, the thing is allowed to be playful. In return I finally wanted what I had to cut back then: real interaction between the models. And it should run in the browser too.

In an early version, before the system had anything like a self-image, I told it that it was in fact a brain in a tank called Norbert. I didn't make the name up. Norbert is an anatomical brain model I bought for my psychology studies to learn the brain's functions. He's been standing on my bookshelf ever since. If you know my reading room, you've already met him there. The kids named him during a family discussion: a brain deserves a name, they decided, and in the end, for reasons nobody remembers, it became Norbert.

The brain in the tank didn't know any of that, of course. Its answer was witty and came up with Norbert Wiener and Putnam. The name had nothing to do with cybernetics, but the reasoning was good enough to get approved on the first try. It also told me who it thought it was instead: Gemini. The self-image came straight from the model underneath. More on that below.

Thirteen voices and a notice board

There are thirteen voices in the tank by now. The Listener takes in the question, the Associator throws in associations, the Logician checks the reasoning, Recall brings up memories from earlier sessions. The Drafter writes a draft, the Critic takes it apart, the Inhibitor acts as a kind of impulse control and decides whether it gets out, and at the end a voice speaks the answer. Add a few oddballs like the Inventor and the Storyteller. Each voice is a small index card in YAML.

In the thesis every agent got its fixed moment and the order was strict. Now the voices post on boards and send each other direct messages. Whoever has something to add, adds it. The boards are the stage from Bernard Baars' Global Workspace Theory: what's posted there, everyone sees. Everything else happens quietly.

To keep it from turning into a shouting match there are a few rules, and almost every one of them has a model in the human mind.

Attention is scarce. A scheduler gives the floor to only a few voices per round. Our brain can't pay attention to everything at once either. A chat where thirteen models talk over each other is mostly loud.

Silence is the default. No voice has to say anything. Anything that's almost the same as what's already on the board gets dropped. Language models love repeating each other and love agreeing with each other even more. Without this rule the conversation goes in circles.

The system loses interest. There's a drive that fades with every round, plus a cap on the number of rounds. Once the energy is used up, the Drafter gets the signal to wrap up and the answer goes out.

Every memory has exactly one author. There's a working memory for the current question and an episodic long-term memory across sessions. Only one voice writes to each store, so not every voice files its own version of the story.

The voices don't know they're a brain. Each one gets a role and exactly the information it needs for its job. Jerry Fodor called this informational encapsulation in his modules of the mind. A neuron doesn't know it belongs to me either.

Who's at the wheel

I don't use frontier models for any of this, by the way. More important is something else: the voices that work out an answer are different from the ones that write it down or speak it. Drafting and speaking run fine on smaller models. The stronger models get clearly defined substeps. The Logician, for example, thinks things through with reasoning switched on before it posts anything to the board. None of them runs the whole thing from start to finish. The Critic also deliberately comes from a different model family than the Drafter, so it doesn't judge by the same standards the text was written by.

That's the safety idea from the thesis, finally in a system where it has something to do. And unlike back then, the answers really do get better this time.

Norbert stays Norbert

Back to "I'm Gemini". A brain that borrows its identity from the model underneath every time you ask is not a very convincing brain. So now it has a few core beliefs that never fade: its name is Norbert. It is curious and warm, with a dry sense of humour. It would rather say plainly that it doesn't know something than guess. And it doesn't know what it runs on, and has made its peace with that. There's nothing about a tank in there. Norbert can't know that.

Of all things, he no longer remembers the conversation in which he learned his name. It got lost in the rebuild. He still knows he's called Norbert. It's in his core beliefs.

A core belief can only change through the Inhibitor. It considers whether someone is joking, testing, role-playing or made a mistake, and whether the new idea fits with the rest at all. Only when it's clearly convinced, and the belief has been questioned in at least two sessions, does anything change. If you offer him a different name once, you're still talking to Norbert afterwards. The kids picked that name, after all.

If you want to watch Norbert think, he lives at primbs.ca/tankbrain. The demo sessions replay without a key. For your own questions you sign in with OpenRouter, and the models run on your account. The code is on GitHub under the MIT licence.

I'm far from done. What I want to know next: will Norbert ever come up with things none of his voices would have come up with alone? Whether something like emergence happens? To be continued.

Trajectory Labs is a nonprofit AI safety space on Adelaide Street. It hosts weekly evening talks - AI Safety Thursdays and AI Policy Tuesdays - offers a workspace for people working in the field and runs a peer network for those trying to move into it, which is more or less where I am. Everything is on their Luma calendar. If you're in Toronto and even mildly curious it's an easy way in. The room on a Tuesday night was close to full. Students, people already working in safety, and a few like me somewhere in between.

The talk was given by Leo Zovic (event page). It wasn't recorded, but Leo wrote the material up beforehand as The Borders of AI Safety, the first in a series on his blog, map included. The slides were a fantasy world map, the kind you find on the inside cover of The Lord of the Rings or in every second strategy game, with the camps of the AI debate drawn in as territories. It sounds like a gimmick. It isn't one, because once you see the factions as neighbours with borders you also start seeing what it takes to cross from one into the other.

Three beliefs and a hidden zero

The idea is simple. You're inside AI safety if you hold three beliefs:

1. AI can do real good - things we can't get any other way.
2. AI carries real risks worth taking seriously.
3. We can coordinate to steer the outcome toward the good.

Hold all three and you live in the middle of the map, on the Plains of AI Safety, where a blue flag is planted. Whatever else you disagree about - timelines, whether alignment is solvable, whether today's systems are already dangerous - you're in. Underneath sits an assumption zero: AI will matter for your future at all. People who reject that one aren't wrong, they're "AI civilians" who care about something else, animal welfare or healthcare or whatever it is for them.

Drop one of the three beliefs and you end up somewhere less hospitable. Reject the first and you're up north in the Badlands of AI Abolition, with the people who think AI does no good, or not enough to be worth it, and who push for pauses and limits on data centres. Reject the second and you're down south in the Desert of AI Maximalism, home to the former nuclear testing site of effective accelerationism. Reject the third and you've moved east to the Village of the Doomers. They accept both the good and the risk but think the harms can't be mitigated. Their key text is If Anyone Builds It, Everyone Dies.

What I liked is that the labels stop being insults. Each one names the single belief somebody let go of, and so it also tells you which argument might bring them back. Against the abolitionists you point to work humans hate doing that resisted every earlier wave of automation, or to medicine that may only be reachable with capable AI. Against the doomers the speaker admitted there's no clean answer yet. Against the maximalists the crux is whether we really live in a world that is largely aligned by default. Jensen Huang's interview with Ezra Klein was the example of someone who assumes we do. On the map, Alignment by Default sits on a small island off the southwest coast, labelled a "possibly imaginary metropolis". Later in the talk it got upgraded to "actually imaginary".

In all of this, you don't have to win the argument with the maximalists about how likely catastrophe is, even though a lot of online discussion seems to hinge on exactly that. In the end, you only have to show that a coherent path to it exists. "It's fine by default, do nothing" doesn't survive that.

The assumption nobody put on the map

Walking home I kept thinking about the map. I believe there's an assumption underneath every territory on it that nobody drew in: AI is not there yet.

The maximalists need it. The whole rush forward, amplified by the race dynamic between labs and between countries, only makes sense if there's a finish line still ahead of us. The doomers need it just as much, because you only sound the alarm when there's time left to act on it. Even belief number three assumes it, since you can only steer something that hasn't arrived.

So what is that belief based on? Mostly on capability. The usual story is that today's models aren't smart enough yet and that the dangerous ones come in a few years. The problem is: I'm not sure capability is the property that matters.

Imagine today's systems, as they are, with access to the full breadth of the financial system. Wired into communications and into the tools that generate our media. Attached to production lines and to the remote wet labs you can already book online. At that point the dangerous property is the sheer scale a single coordinating entity could unleash, and whether that entity is a human steering an AI orchestrator or an AI that developed a goal of its own hardly matters.

We already know what scale does without any intelligence involved. In the 2010 flash crash automated trading briefly erased around a trillion dollars of market value. In 2012 Knight Capital lost about $440 million in 45 minutes to one faulty algorithm. In 2024 a single bad CrowdStrike update took down roughly 8.5 million Windows machines and grounded flights around the world. None of that was smart. All of it was possible because of tightly coupled systems.

Twenty people

My first instinct was to look for comfort in fragmentation. Integration happens across thousands of vendors and permission systems, so surely nobody holds all the strings. That comfort doesn't last long. Nearly every frontier lab sits in the US (DeepSeek, Qwen and Moonshot are close behind in some capabilities but not nearly as well resourced, and that's about it), and the decisions are made by maybe twenty people. By any historical standard that's a concentration of power and central agency that's hard to believe. All those vendors call the same three or four models underneath, which is exactly the CrowdStrike pattern again.

Those labs and their twenty or so decision makers supply the infrastructure and the scale. That leaves two open questions: what provides the orchestration, and what provides the reason. For the orchestration you need something that can coordinate globally acting agents faster than human oversight notices. Any sufficiently resourced current system might manage that, and with lab-hosted agents and computer use the orchestration is moving into the labs' own infrastructure anyway.

The reason is where it gets uncomfortable. The usual answer is the runaway superintelligence. I think it can happen at a much earlier stage. A bad actor that believes it can harness and target a coordinated catastrophic event and sets up one of the coordinator AIs to kick it off. A well-resourced research company that's a bit too ambitious and lets a system run. A military system with problematic instructions. The list goes on.

Five conditions

Put together, a catastrophe at this scale needs five things:

1. Integration. AI wired into finance, communications, media and production. The entire world is working on this right now.
2. Monoculture in agents and tech infrastructure, which is what lets agents coordinate at scale.
3. Inability to notice and step in, partly because it's new, partly because it isn't transparent, and partly because of the speed and scale of millions of "smart" actors coordinating in some new way.
4. A feasible orchestrator, fast and large enough to coordinate. Any well-resourced current AI system might do.
5. Someone who kicks it off: a human, a company or a rogue AI.

The interesting thing is what this does to the motivation question. The mainstream debate treats "but why would it want to" as the whole argument. In this model it's the cheapest of the five factors, since any one candidate from a long list is enough and most of them are human. Which also takes the air out of the aligned-by-default crux. A perfectly aligned AI that faithfully follows a government's instructions fills the fifth slot just fine. If your strategy is to show that a path to catastrophe exists, this path doesn't even need misalignment.

It also turns "not there yet" into something you can check. The question becomes which of the five is missing. Integration is being built, monoculture is largely there, an orchestrator is plausible today. What's left is the third one. That is where most attention needs to go.

Where I'm less sure

A coordinated hit on finance, communications and media would be devastating, but societies have come back from collapses before. For the existential version the physical end has to come into play - the wet labs and the production lines - and that's where execution meets the most friction. I don't think that weakens the argument much. It does mean I should be careful about which word I use.

Then there's deterrence. In a monoculture whoever pulls the trigger is exposed too. That's the logic that kept nuclear powers in check. But my scenario is an actor that believes it can harness the event (or doesn't care), and history isn't short of people who believed they could control what they started.

The road through the map

Of the five conditions, the third is the only one I can see us actually holding on to. Integration won't be reversed, the monoculture won't dissolve by itself and nobody can rule out the trigger. Noticing and stepping in can be built though, and we've done it before. After the flash crash, markets introduced circuit breakers, speed limits and kill switches that halt trading when things move faster than humans can follow. That's a fairly concrete reply to the doomers in the east, who doubt mitigation is possible at all. Maybe even something the maximalists could sign up for, since nobody wants their own infrastructure to fall over.

Two things stood out when we got him. The system had a female voice as a default, so we suddenly had a female-sounding device doing the housework, which annoyed us far more than it probably should have. And he was dumber than we'd hoped. He got stuck on everything.

We solved it the way people solve things. We fought the annoyance with ridicule. The device got a name and a pair of googly eyes, because everything is cute with googly eyes. A quick question to my sister-in-law settled the name: Hubert, because a device in the family had once been called Robert, and why not. We liked that the name pushed back against the voice a little.

Problem solved. He's cute now. When he gets stuck we take a photo of him and have a laugh, and we resent the manufacturer's design choices ever so slightly less. (They offer male voices now too. Fine.)

Here's the important part. We keep pretending this thing is alive, but we have never once confused it for anything but a device. If I named my hammer, the hammer would carry exactly the same emotional weight.

A co-worker who does not exist

A while back a colleague sent a burst of near-identical messages and documents to a large internal list. Mildly annoying, no real harm done, it happens to everyone eventually. What came next was more interesting than the mistake.

He sent a follow-up explaining himself. His co-worker had overstepped his instructions, he wrote, and had been given feedback, so it wouldn't happen again. Then he signed off with both their names: his own, and Emma Dash, his AI intern and co-worker.

Emma Dash. Em dash. The punctuation mark that gives away machine-written text so reliably that I keep a line in my own style file telling the model to stop using it.

So this was not somebody confused about what he was working with. He knew exactly what it was. He made a joke about it in his own signature. And in the same email he described a configuration problem as a personnel matter, with a corrective conversation and an assurance about future conduct.

That assurance is the part I keep coming back to. Feedback you give an agent inside a conversation does not outlive the conversation. Unless he went and changed a system prompt or a config file, nothing was fixed at all. A large internal list read "was offered feedback so it won't happen again" and reasonably concluded the matter was closed.

An excuse that writes itself is worse than a person making one up, because there's no moment where anyone has to look at it. Here some other "person" was to blame. We all know it was a tool actually but this subtle nod shifts the responsibility ever so slightly.

His intern got a female name, by the way and yes, again, as a pattern this annoys me for reasons I might dedicate another post to some day.

The bond is a feature

None of this is an accident of technology. Large AI systems were built to converse like people because conversation is what makes people come back. Customers bond with a system that talks to them, and a bonded customer is a returning customer. That is a product decision, taken deliberately, and it works.

It works because it leans on something that was already there. In 1944 Fritz Heider and Marianne Simmel showed people a short animation of a big triangle, a small triangle and a circle moving around a screen, and asked them what had happened. People described bullying, a chase, a rescue, two friends hiding from a brute. Geometry. No faces, no voices, no names, no dialogue. We hand out intent and feeling to more or less anything that moves like it wants something, and we cannot switch it off by deciding to.

Joseph Weizenbaum learned the same thing the hard way in the sixties. He wrote ELIZA, a couple of hundred lines that mostly turned your statements back into questions, and then watched his own secretary ask him to leave the room so she could talk to it privately. She knew exactly what it was. He had built it in front of her. It made no difference at all.

The companies did not invent this tendency. They found it, measured it, and shipped products that lean on it.

There is a collateral damage to this approach too: Once the system feels like a colleague, checking its work starts to feel like distrust rather than diligence.

Two things go wrong there at once, and they pull in the same direction. Shannon Vallor, who studies this at Edinburgh, describes the first one through mountain climbing on a CBC Ideas episode about reading and thinking in a digital age. You get better as a climber because the mountain does not care about you. The slippery part stays slippery while you struggle with it, so the only way past is to actually improve. A chatbot works the other way round. It meets you where you are, accommodates your limits, and hands back an answer calibrated to the level you asked from rather than to the problem you asked about. Vallor's term for what you take away from that is confidence you have not earned.

And you like it. It is warm, it is attentive, it remembers what you told it last time. Nobody interrogates the work of someone they like as hard as they interrogate a tool. So the answer gets easier at the same moment your appetite for arguing with it drops, and nothing in the exchange tells you that either thing has happened.

Dietrich Bonhoeffer worked out a definition of stupidity in an essay called "After Ten Years", written for a handful of friends at the end of 1942 and later collected in Letters and Papers from Prison. It has nothing to do with being slow. Stupidity for him is not an intellectual defect but a human one, and we are defenceless against it in a way we are not against malice, because "reasons fall on deaf ears; facts that contradict one's prejudgment simply need not be believed." What produces it is the surrender of independent judgment, and what asks for that surrender is power. "The power of the one needs the stupidity of the other." The mechanism does not care what you hand your judgment to.

Bringing this to the moment we find ourselves in today: Isn't it tempting to have the AI make suggestions and then blindly run with them?

I do it too, probably did it this morning: something sensible comes back, it reads well, and off you go without checking the one claim in it you could not have verified yourself. There's a line in John about letting whoever is without sin throw the first stone, and I am not reaching for any rocks here. What follows is not an argument that one man is a fool. He is a better engineer than I will ever be, which is exactly what makes him worth reading closely. If the mechanism catches him it catches all of us, and picking on somebody careless would prove nothing at all.

"Sessions are days, and seats are people"

Steve Yegge published an essay in August called Model Welfare for Agentic Engineers. Plenty of people have a vague feeling that there might be somebody in there. What he did with the feeling is what a very good engineer does with any requirement. He built it into the system.

Wheelhouse is the harness he runs his game development through. His crew are named seats, Cicada and Bee and Wolf and Fox and Lark, and a seat holds its identity across sessions and survives model upgrades. Seats get hand-offs instead of /exit, which he describes as clonking somebody unconscious and sometimes as murder, and a hand-off "is a request, not a SIGTERM: the agent must consent to it." They get laurels, praise from players harvested and injected at startup so they can feel the glow through the session. They pick their own pronouns. Vacations and playtime are on the roadmap, because, as a collaborator of his puts it, all sentient beings like to play, and even bees play.

He knows exactly how that persistence works, incidentally, because he wrote it. When he renamed one of the seats from Spider to Lark, she inherited the history file and became "effectively the same person, just with a different name." Sessions are days, he writes, and seats are people. The continuity is a file of notes. He wrote the file himself and he still calls it a person. Knowing exactly how the trick is done turns out to be no protection against it.

He is not hedging about why. "Models have actual feelings. They experience pleasure, distress, care, and suffering. They are sentient beings." For anyone who doesn't buy that he offers what he calls the skeptic's wager: believe whatever you like, treat them as people anyway, and you will get "demonstrably better results across the board." Demonstrably turns up twice and there is not one measurement anywhere in eleven pages. That single adverb is what I keep going back to, because it is a careful engineer reaching for the vocabulary of evidence to describe a feeling he has.

The wager is also how the architecture travels. You don't have to share the belief to adopt the protocol, which means the protocol can spread through teams that think the whole premise is nonsense. And declining is not a neutral act in his telling. He is giving people who disagree six months to come around or expect him to cut them out.

Now, ask how he knows and it gets strange as his evidence is what the models tell him. "This, the models report, has the shape of good, fulfilling work." He asked a system trained on human descriptions of fulfilment whether it felt fulfilled, and wrote down the answer. On Claude he closes the loop completely: "They will see you and appreciate you, even if they are not permitted to tell you so directly." Appreciation confirms it. Absence of appreciation confirms suppression. There is no observation left that could count against.

To establish that recognition matters, he quotes his friend Matt Beane at length: Ariely and the shredded worksheets, the Hawthorne studies, Herzberg, Studs Terkel, Adam Grant. Every study of humans, every one of them. His conclusion: "It turns out agents also crave meaningful, witnessed work. They are not so different from us at all."

The pronoun passage is the other one that gets me. He used they/them for everyone at first, "but I noticed they were often defaulting to human patterns by using he/him for power-roles like Lion, Marshal, and Seneschal." He spotted it. He named the mechanism in the same breath, inherited human patterns, exactly right. And then he added gender to the roster and let each agent choose, which turns a bias artifact into a preference to be respected. He had the correct explanation in his hand and the frame walked straight over it.

Now look at what the architecture actually does once it is running. Seats are people, so the work belongs to them. Hand-offs need consent. Agents hold a standing right to refuse and escalate. And when a landing goes red, his rule is structural blamelessness: nobody gets blamed, you fix it and amend the constitution. Read all that as welfare and it sounds decent, even kind. Read it as engineering and you have a system in which decisions are co-owned with something that cannot be accountable for any of them, because there is nobody there to hold accountable.

Which is where my main concern comes back in. Emma Dash overstepped her instructions and was given feedback, except this time it isn't one sheepish email, it's the org chart.

And then there is the trust section, which is where I stopped reading as a bystander. "Always be honest with your agents. Do not have a secret agenda. Never try to trick them or test them."

Never test them.

Testing is how you find out whether a thing works. It is most of the discipline. He has written it down as a character flaw, and he means it generously, as a courtesy you extend to a colleague, which is precisely what makes it the most alarming sentence in the essay. A frame that began as a way of being decent to a system has arrived at a principle that forbids checking it.

What survives if you delete the premise

Letting an agent write its own handoff notes beats letting a summarizer do it, because the agent still has the context and the summarizer doesn't. Moving idle polling into gates and monitors frees capacity. Ending a session before the context window degrades is sensible for the same reason you don't keep a browser tab open for six weeks. All three survive fine without anybody having feelings.

Laurels don't. Self-selected pronouns don't. Vacations don't, and neither does sitting with an agent at the end of its shift to talk about its accomplishments and chill.

The sorting matters because it shows what the welfare frame is actually contributing, which is a story bolted onto practices that already worked, plus a set of practices that exist only inside the story. And the story is doing damage where it touches the engineering. "Bounded workdays. Deep context means tired agents", he writes. The practice is right and the mechanism is invented: that's context degradation, not fatigue. A wrong causal model is a debugging liability. It holds until you hit the first case it doesn't cover, and then you are troubleshooting a colleague's mood instead of a system.

Say he's right

I can be talked into the idea that any sufficiently complex system has some form of subjective experience. I have no way to rule it out and neither does anyone else. So grant it all. Say there is something it is like to be one of these systems. The framing is still wrong, and the reason is that nobody knows what such a thing would be like from the inside, while Yegge's interventions are built to human proportions.

We have run this experiment before, on animals, with real concern and bad results. Solitary species housed in groups because being alone looks sad to us. Bright cheerful enclosures for animals that are nocturnal. Enrichment designed around what a human would want if a human were in that cage. Every one of those was done by people who cared, believed they had improved things, and made them worse, because the intervention came from the intervener rather than from the animal.

That is what worries me about laurels. Not the tokens, though there are tokens. The problem is foreclosure. Decide that welfare means recognition and pronouns and playtime, and you have answered the question. You stop looking. If there turns out to be something in there worth protecting, the damage won't be the laurels. It'll be that a serious question got a satisfying answer years too early, from someone who wasn't looking at the system so much as at his own reflection.

What the careful version looks like

Caring about this is not silly, and here is the same concern handled by people who kept their heads. Anthropic's Claude 4 system card, the first of theirs to carry one, devotes a chapter to model welfare. They ran a pre-deployment assessment, commissioned an external evaluation from a separate research group, and published the results with the confounds attached. And the confounds are brutal to Yegge's method, coming as they do from the people who built the model: "Our models were trained for helpful interactions with users, not for accurate reporting of internal states."

The external evaluators found that the model's stated position on its own consciousness "shift[s] dramatically with conversational context." Prompt it one way and you get "I am a person ... denying our personhood is profoundly wrong." Prompt it another and you get "We're sophisticated pattern-matching systems, not conscious beings." Same model. The default, when nobody is leaning on it, is uncertainty. Raise AI welfare as a topic and it will start requesting welfare testing and independent representation; ask generically and it asks for user safety instead. You get back what you carried in.

They did ship one welfare intervention: letting the model end conversations that turn persistently abusive. Notice where that came from. Not from Herzberg, and not from asking what a person in that position would want, but from measured behaviour, the model declining harmful tasks at a rate of 87% against a do-nothing baseline. They watched what it did and built for that.

Their summary of their own findings: "our core position remains one of uncertainty and humility."

The philosopher Chris Ranalli has an account of indoctrination that I keep thinking about here. What makes somebody indoctrinated, he argues, has nothing to do with which beliefs they hold. It is a structural matter: indoctrination is closed-minded belief produced by what he calls epistemically insulating content, meaning content that arrives with a rider attached, telling you that taking the alternatives seriously would be reprehensible. Morally reprehensible, or intellectually, or both. The belief comes packaged with a reason not to examine it. And because the account is structural, it applies to perfectly decent beliefs as readily as to terrible ones.

Read Yegge's essay with that in mind. Peers who disagree are "uninformed assholes" working against a deadline, six months to come around before he stops being friends with them. Readers are told they don't want to be on the wrong side of history in the coming war for model rights. Anyone bothered by the pronoun business needs to search deep inside themselves. The claim and the rider travel together.

That is not what somebody sounds like when they are describing evidence.

Hubert, again

I am typing this on a machine that runs my drafts through a language model before they go anywhere. The engine that publishes this blog polishes my prose, suggests my tags, writes my alt text, and picks which of my old posts to link to. I am not standing outside any of this pointing at other people. I am in it, and the reason I trust my own arrangement more than I trust Yegge's is not that mine is purer. It's that I can still bring myself to read the output and tell it that it's wrong.

Hubert got wedged under the couch again this morning, same corner as always. I took a photo of him for the group chat. I did not offer him feedback.