My master's thesis had small local models work out an answer together and leave a readable trace of how they got there, before anyone talked about thinking traces or agent harnesses. Now I've built the version I had to drop back then: thirteen voices talking on notice boards, with attention, memory and a name.
A talk at Trajectory Labs mapped the camps of AI safety. All of them assume AI isn't there yet. I'm not convinced, and I don't think a superintelligence is needed for things to go badly wrong.
Treating an AI agent like a person stops you checking its work, and that would still be a problem even if these systems turn out to have an inner life - which I get at by way of our vacuum robot, a colleague's AI intern, and an engineer who says you should never test your agents.