[Home](https://unprompted.dirkprimbs.de/index.md) · [← Previous: Markdown Web - Yes, you read that right.](https://unprompted.dirkprimbs.de/posts/engine-update-markdown.md) # You don't need a superintelligence for this > A talk at Trajectory Labs mapped the camps of AI safety. All of them assume AI isn't there yet. I'm not convinced, and I don't think a superintelligence is needed for things to go badly wrong. - Published: 2026-10-21 - Tags: [AI](https://unprompted.dirkprimbs.de/tags/ai.md), [AI Ethics](https://unprompted.dirkprimbs.de/tags/ai-ethics.md), [Tech Community](https://unprompted.dirkprimbs.de/tags/tech-community.md) - Web page: https://unprompted.dirkprimbs.de/posts/you-dont-need-a-superintelligence-for-this.html --- Yesterday evening I went to "The Borders of AI Safety" at Trajectory Labs in downtown Toronto. I'm spending a good part of this autumn getting to know the AI safety crowd in this city, and this talk sounded like a good way to see the whole field at once. [Trajectory Labs](https://www.trajectorylabs.org/) is a nonprofit AI safety space on Adelaide Street. It hosts weekly evening talks - AI Safety Thursdays and AI Policy Tuesdays - offers a workspace for people working in the field and runs a peer network for those trying to move into it, which is more or less where I am. Everything is on their [Luma calendar](https://luma.com/trajectory-labs). If you're in Toronto and even mildly curious it's an easy way in. The room on a Tuesday night was close to full. Students, people already working in safety, and a few like me somewhere in between. The talk was given by [Leo Zovic](http://inaimathi.ca) ([event page](https://luma.com/trajec-0y9i)). It wasn't recorded, but Leo wrote the material up beforehand as [The Borders of AI Safety](https://inaimathi.ca/posts/the-borders-of-ai-safety), the first in a series on his blog, map included. The slides were a fantasy world map, the kind you find on the inside cover of The Lord of the Rings or in every second strategy game, with the camps of the AI debate drawn in as territories. It sounds like a gimmick. It isn't one, because once you see the factions as neighbours with borders you also start seeing what it takes to cross from one into the other. ## Three beliefs and a hidden zero The idea is simple. You're inside AI safety if you hold three beliefs: 1. AI can do real good - things we can't get any other way. 2. AI carries real risks worth taking seriously. 3. We can coordinate to steer the outcome toward the good. Hold all three and you live in the middle of the map, on the Plains of AI Safety, where a blue flag is planted. Whatever else you disagree about - timelines, whether alignment is solvable, whether today's systems are already dangerous - you're in. Underneath sits an assumption zero: AI will matter for your future at all. People who reject that one aren't wrong, they're "AI civilians" who care about something else, animal welfare or healthcare or whatever it is for them. Drop one of the three beliefs and you end up somewhere less hospitable. Reject the first and you're up north in the Badlands of AI Abolition, with the people who think AI does no good, or not enough to be worth it, and who push for pauses and limits on data centres. Reject the second and you're down south in the Desert of AI Maximalism, home to the former nuclear testing site of [effective accelerationism](https://en.wikipedia.org/wiki/Effective_accelerationism). Reject the third and you've moved east to the Village of the Doomers. They accept both the good and the risk but think the harms can't be mitigated. Their key text is [*If Anyone Builds It, Everyone Dies*](https://bookwyrm.social/book/2065543). What I liked is that the labels stop being insults. Each one names the single belief somebody let go of, and so it also tells you which argument might bring them back. Against the abolitionists you point to work humans hate doing that resisted every earlier wave of automation, or to medicine that may only be reachable with capable AI. Against the doomers the speaker admitted there's no clean answer yet. Against the maximalists the crux is whether we really live in a world that is largely aligned by default. [Jensen Huang's interview with Ezra Klein](https://www.youtube.com/watch?v=HjurAWAr_nY) was the example of someone who assumes we do. On the map, Alignment by Default sits on a small island off the southwest coast, labelled a "possibly imaginary metropolis". Later in the talk it got upgraded to "actually imaginary". In all of this, you don't have to win the argument with the maximalists about how *likely* catastrophe is, even though a lot of online discussion seems to hinge on exactly that. In the end, you only have to show that a coherent path to it *exists*. "It's fine by default, do nothing" doesn't survive that. ## The assumption nobody put on the map Walking home I kept thinking about the map. I believe there's an assumption underneath every territory on it that nobody drew in: AI is not there yet. The maximalists need it. The whole rush forward, amplified by the race dynamic between labs and between countries, only makes sense if there's a finish line still ahead of us. The doomers need it just as much, because you only sound the alarm when there's time left to act on it. Even belief number three assumes it, since you can only steer something that hasn't arrived. So what is that belief based on? Mostly on capability. The usual story is that today's models aren't smart enough yet and that the dangerous ones come in a few years. The problem is: I'm not sure capability is the property that matters. Imagine today's systems, as they are, with access to the full breadth of the financial system. Wired into communications and into the tools that generate our media. Attached to production lines and to the remote [wet labs](https://en.wikipedia.org/wiki/Cloud_laboratory) you can already book online. At that point the dangerous property is the sheer scale a single coordinating entity could unleash, and whether that entity is a human steering an AI orchestrator or an AI that developed a goal of its own hardly matters. We already know what scale does without any intelligence involved. In the [2010 flash crash](https://en.wikipedia.org/wiki/2010_flash_crash) automated trading briefly erased around a trillion dollars of market value. In 2012 [Knight Capital lost](https://en.wikipedia.org/wiki/Knight_Capital_Group#2012_stock_trading_disruption) about $440 million in 45 minutes to one faulty algorithm. In 2024 [a single bad CrowdStrike update](https://en.wikipedia.org/wiki/2024_CrowdStrike-related_IT_outages) took down roughly 8.5 million Windows machines and grounded flights around the world. None of that was smart. All of it was possible because of tightly coupled systems. ## Twenty people My first instinct was to look for comfort in fragmentation. Integration happens across thousands of vendors and permission systems, so surely nobody holds all the strings. That comfort doesn't last long. Nearly every frontier lab sits in the US (DeepSeek, Qwen and Moonshot are close behind in some capabilities but not nearly as well resourced, and that's about it), and the decisions are made by maybe twenty people. By any historical standard that's a concentration of power and central agency that's hard to believe. All those vendors call the same three or four models underneath, which is exactly the CrowdStrike pattern again. Those labs and their twenty or so decision makers supply the infrastructure and the scale. That leaves two open questions: what provides the orchestration, and what provides the reason. For the orchestration you need something that can coordinate globally acting agents faster than human oversight notices. Any sufficiently resourced current system might manage that, and with lab-hosted agents and computer use the orchestration is moving into the labs' own infrastructure anyway. The reason is where it gets uncomfortable. The usual answer is the runaway superintelligence. I think it can happen at a much earlier stage. A bad actor that believes it can harness and target a coordinated catastrophic event and sets up one of the coordinator AIs to kick it off. A well-resourced research company that's a bit too ambitious and lets a system run. A military system with problematic instructions. The list goes on. ## Five conditions Put together, a catastrophe at this scale needs five things: 1. **Integration.** AI wired into finance, communications, media and production. The entire world is working on this right now. 2. **Monoculture** in agents and tech infrastructure, which is what lets agents coordinate at scale. 3. **Inability to notice and step in**, partly because it's new, partly because it isn't transparent, and partly because of the speed and scale of millions of "smart" actors coordinating in some new way. 4. **A feasible orchestrator**, fast and large enough to coordinate. Any well-resourced current AI system might do. 5. **Someone who kicks it off**: a human, a company or a rogue AI. The interesting thing is what this does to the motivation question. The mainstream debate treats "but why would it want to" as the whole argument. In this model it's the cheapest of the five factors, since any one candidate from a long list is enough and most of them are human. Which also takes the air out of the aligned-by-default crux. A perfectly aligned AI that faithfully follows a government's instructions fills the fifth slot just fine. If your strategy is to show that a path to catastrophe exists, this path doesn't even need misalignment. It also turns "not there yet" into something you can check. The question becomes which of the five is missing. Integration is being built, monoculture is largely there, an orchestrator is plausible today. What's left is the third one. That is where most attention needs to go. ## Where I'm less sure A coordinated hit on finance, communications and media would be devastating, but societies have come back from collapses before. For the existential version the physical end has to come into play - the wet labs and the production lines - and that's where execution meets the most friction. I don't think that weakens the argument much. It does mean I should be careful about which word I use. Then there's deterrence. In a monoculture whoever pulls the trigger is exposed too. That's the logic that kept nuclear powers in check. But my scenario is an actor that *believes* it can harness the event (or doesn't care), and history isn't short of people who believed they could control what they started. ## The road through the map Of the five conditions, the third is the only one I can see us actually holding on to. Integration won't be reversed, the monoculture won't dissolve by itself and nobody can rule out the trigger. Noticing and stepping in can be built though, and we've done it before. After the flash crash, markets introduced [circuit breakers](https://en.wikipedia.org/wiki/Trading_curb), speed limits and kill switches that halt trading when things move faster than humans can follow. That's a fairly concrete reply to the doomers in the east, who doubt mitigation is possible at all. Maybe even something the maximalists could sign up for, since nobody wants their own infrastructure to fall over.