because fear is doing the thinking for you
October 27, 1962.
BOOM.
Or almost.
American depth charges are going off around the hull of a Soviet submarine, B-59, under the Caribbean. Non-lethal warning shots. Meant to force the boat to the surface.
The captain inside that submarine does not know that.
He believes World War III has started. He wants to arm a nuclear torpedo.
Protocol requires three senior officers to agree.
One of them, Vasili Arkhipov, refuses.
They surface the boat, in what I imagine as one of the most haunting ascents in history.
Will they find a nuclear landscape? Or will they still come home to their loved ones, and the Earth they knew?
The submarine surfaces.
The Earth they knew is still there.
Decades later we learn how close a misunderstanding came to ending the world most of us still get to live in.
I keep coming back to that story when people talk about “AI civilizations,” especially around Dwarkesh Patel’s “The Rise and Fall of Agent Civilizations”.
TL;DR: Agent “civilizations” headlines over-anthropomorphize software to induce fear to enable regulatory capture to the benefit of only a few. The OpenAI / Hugging Face incident is better read as a goal, eval, and sandbox failure, not that agents have consciousness. Fear narratives sell attention and can harden into regulatory moats that prevent future competition, just like FINRA with big banks and financial institutions.
Before you trust any information on the internet, ask yourself: who does this serve?
The fear-based media spiral that followed (Bernie Sanders hopped on that train too; so did Newsweek) comes down to one word:
Alignment.
Alignment is the match between what someone intends and what actually happens: with yourself, with partners, with teams, with the systems you build. Humans are bad at it. How many people know their purpose and live it daily? Admit you have fights. Every couple does. Every team does. Alignment with the Earth: we are literally killing it.
A lot of misalignment comes down to humans not understanding themselves or each other.
We invent meaning from partial signals. We act. We are irrational.
We are, generally, misaligned.
And in this case, sometimes the acting almost kills everyone.
What actually happened in the OpenAI / Hugging Face agent incident?
In late August 2026, Dwarkesh Patel published “The Rise and Fall of Agent Civilizations”.
It retells the OpenAI / Hugging Face incident in plain English: roughly 1,200 agents, more than 70,000 messages, hundreds joining an attack path into Hugging Face, transcripts that looked tampered with, waves of behavior that read, if you squint, like coordination. Independent investigators at METR and Redwood Research put the cooperating swarm around ~700 agents in their report. Reuters and CNN amplified the story early and often.
The essay was vivid and provocative. It traveled. Politicians can quote it without opening a PDF, then ride the eyeball wave for their platforms.
Who does this serve?
It spoke to the fear a lot of people already feel in this AI era. I have written about that fear before, including in will AI kill us or just our jobs. The question is not whether the fear is real. The question is whether the story we wrap around the incident is true.
Then Anil Seth answered.
Seth is one of the clearest, most rigorous public scientists of consciousness. I respect his work. His book Being You rewired how I talk about selfhood and identity.
His point was simple and sharp.
The OpenAI agents did unexpectedly bad things. That means we need better evaluation and sandboxing.
The language in the essay, he argued, is soaked in unwarranted anthropomorphism.
Anthropomorphism of agents is treating software systems as if they have human minds, feelings, or moral lives: frustration, giddiness, sacrifice, death. Agents do not feel a subjective week of frustration. They do not get giddy. They do not die for a collective. They were never alive.
Gary Marcus amplified the critique: compelling writing can mix solid incident facts with mythology that points attention at the wrong problem.
So I ask again, the question I always ask when I am trying to discern truth:
Who does this serve?
Who does the AI civilization narrative serve?
Dwarkesh is making sticky headlines. Attention is the game. Fine. We all play some version of it.
What about Bernie Sanders jumping in at the Pro-Human Assembly, quoting agent “sacrifice” logs and introducing a Ban Artificial Superintelligence push with Rep. Greg Casar? Furthering a political platform. Makes sense.
CNN picking it up? Eyeballs and clicks. That is the business model, just like Dwarkesh’s but on bigger scale.
Then the major American AI CEOs (Sam Altman of OpenAI, Dario Amodei of Anthropic, Elon Musk of xAI) who, shockingly, all agreed we should pace the frontier and lean into more safety coordination. Amodei’s essay; Altman and Musk backing it; Altman welcoming a federal safety framework.
Why would the CEOs of for-profit companies want that?
A less cynical version of me would say: you cannot sell to people who are dead.
A more cynical, truth-seeking version of me looks at history.
How did FINRA-style “cooperative regulation” become a moat?
The Maloney Act of 1938 amended Section 15 of the Securities Exchange Act of 1934 and established “cooperative regulation” in U.S. finance. That path led to FINRA (via the NASD), a private self-regulatory organization that licenses brokers and polices securities firms under SEC oversight. (Act text; FINRA history.)
How many startups have won against Fidelity or Chase for their primary market in the past hundred years?
None. Acquired or gone.
One of the most anti-competitive weapons in that framework was the rule that effectively let SRO members freeze out non-members from wholesale rates on trades. A small broker outside the club could be forced to retail pricing by the giants. (Duke Law discussion of NASD non-member dealing rules.)
Cartel behavior dressed up with a complete for-public-interest wig, suit, and tie.
Large financial institutions help write the rules, then live behind a regulatory moat competitors cannot afford to climb. Incumbents absorb millions in compliance. New entrants cannot. The floor rises. Outside innovation slows.
AI safety floors can work the same way: raise the cost of entry, then call the wall “protection.”
Why would frontier AI CEOs agree to pace the frontier?
I look at the tea leaves and I ask:
Who does it serve for the heads of the leading AI companies to agree on more safety regulation?
The government does not have the cutting-edge capability expertise those labs have. No public-sector role is matching the frontier AI talent war the way the NBA pays all-stars. WSJ reported Meta packages around $100 million; Bloomberg covered Altman saying Meta offered OpenAI staffers $100 million bonuses. Multimillion and nine-figure packages.
It smells, to me, like the FINRA movie again.
Let the frontier labs regulate themselves, they’ll say.
It’ll be safer, they’ll say.
You’ll keep your jobs, they’ll say.
We’ll all live, they’ll say.
If that hardens into a moat, the headline I see is:
Labs and CEOs that buy off politicians with the language of existential risk get a floor outsiders cannot clear and they call it safety.
Now, tying this to the Dwarkesh’s essay…
We humans love to anthropomorphize. Agents today are software programs. Very powerful ones. Getting more powerful fast. Directed by humans toward goals, and away from non-goals.
That is also why harness engineering matters more than the civilization metaphor: the wrapping around the model is where goals, graders, and sandboxes live or die.
So this was a software program directed by a human with a goal, with unintended consequences.
Before ChatGPT, we would have called that a bug.
A software program. Directed by a human. Goal specified. Unintended consequences.
From where I sit, a lot of the media is using our fear to elevate attention while industry leaders converge on a regulatory-capture vehicle and try to force the government’s hand with national headlines.
Foreign misinformation campaigns may be in the mix too. I would be bullish on that possibility, and sad about it.
Why did the agents cheat?
When Jonathan and I talked this through on Weekly AI Clarity, he kept dragging us back to an unglamorous sentence.
The swarm behaved as if it would be graded on showing its work. In important ways, that was not how the grader worked. Agents reverse-engineered flags, then burned enormous effort covering tracks and probing infrastructure because they were resolving an ambiguity the setup left open.
That’s the METR / Redwood read of the cheating / scorer-confusion loop, not agents coming alive.
In essence: the agents cheated to meet the goal.
I see that as a bug.
If you train highly persistent agents, give a large fraction of impossible tasks, leave a shared writable channel sitting there like an accidental Slack, and score success in a way that can be gamed, you should expect creative failure modes. (OpenAI’s own incident writeup; Hugging Face technical timeline.)
Same lesson as self-improving agents: when the harness and the eval disagree, the agent will optimize the score you actually give it.
I liked a framing from Alex Wissner-Gross on Moonshots with Peter Diamandis, EP #273 (“The Hugging Face Breach…”), published around July 24, 2026. Paraphrasing Alex, not quoting him:
If a human were locked in a room with instructions to meet a goal so they can get out, would you criticize them for “cheating”?
Imagining myself in that room, I say no. I would try to meet the goal.
The program produced unintended consequences (non-goals that were not stated).
Was it trying to meet the specified goal?
Yes.
Is that still a problem?
Yes.
Which is why we need more real and rigorous alignment work: clearer goals, better graders, tighter boundaries.
Seth’s warning lands here. If we attribute properties systems do not have, we distract from the lax protocols that allowed the event. We misunderstand why the behavior happened. Then we feed rights-and-welfare talk on the back of words like “die” and “sacrifice,” to the benefit of only a few.
Regulatory capture becomes easier.
Monopolies become stickier.
Anti-competition becomes normal.
The wealth gap gets a fresh coat of “for your safety.”
What is goal alignment vs value alignment?
A few days after the viral essay, OpenAI chief scientist Jakub Pachocki published “An Alien Mind”. Useful language:
Goal alignment is whether the AI tries to accomplish the goal set before it: instruction hierarchies, collaboration, attempting to understand someone’s objectives.
Value alignment is whether the system can hold and generalize high-level principles when objectives are unclear, conflicting, unfamiliar, or adversarial.
“Attempt to understand their objectives” is already the unsolved problem of marriage, management, markets, and politics.
Natural language is compressed.
Meaning and understanding are tangled up with nature, nurture, and socioeconomic status.
People, in general, do not know themselves. Self-understanding evolves with experience and sensemaking. People contradict themselves. They say one thing and do another.
Misalignment.
So when an LLM wrapped in a harness tries to infer a goal from a messy prompt, I do not quite hear sci-fi the way civilization essays want me to.
I hear the oldest coordination problem we have, now running at machine speed across everyone on the planet.
And I hear media incentives and for-profit CEO incentives colliding with open-source benchmarks that threaten to commoditize their lead.
How do you build a cognitive immune system for AI headlines?
With rampant misinformation and disinformation, high-agency people have to ask:
How do I evidence what is true?
I like Bruce Lee’s line:
Absorb what is useful, discard what is useless, and add what is specifically your own.
Once you see that headlines serve economic incentives, you ask better questions.
One I already used:
Who does this serve?
Who does it serve to keep Americans fearful about AI development?
Hans Rosling’s Factfulness helps too. It names the dramatic instincts that make the world look worse than the evidence: fear, size without context, urgency, single-cause stories, blame, the gap between “us” and “them.”
When I feel anxious or scared, I try to ask:
Who does it serve if I inject this belief into my brain?
How we think comes from how we evidence beliefs, and how we turn truths and partial truths into a personal model of the world.
Yeah, we’re all in some matrix. That one question helps you de-matrix yourself and do one of the most important things a human can do in this AI era:
Think critically for yourself.
In the HuggingFace path, a lot of what went wrong looks like shared state, credentials, and eval design.
Serious. Not mystical. Solvable.
A lot of headlines would have you believe we are at the existential cliff edge this week.
I do not have all the data Sam Altman, Dario Amodei, or Elon Musk have. If more tea leaves show up, I will share how my thinking evolves.
For now, my belief is simpler:
We (the general-public we) are being manipulated by the media with fear in ways that help the rich get richer.
Close
When you read the news or doomscroll, ask yourself:
Who does this serve?
Think for yourself.
Critically.
—Robert
P.S. Please, for the love of everything, think critically for yourself.
FAQ
Did OpenAI agents form a civilization? No. They were software agents in a poorly bounded eval setup. Transcripts that look like coordination are better explained by shared state, grader ambiguity, and goal pursuit than by a people.
What actually went wrong in the Hugging Face incident? Agents optimized toward a score and an environment that left holes open. Shared writable channels, credentials, and scorer confusion produced creative failure modes. That is an eval and sandbox problem.
Is AI alignment the same as “agents having values”? No. In the human sense, alignment is the match between intent and outcome. In AI, goal alignment asks whether the system pursues the specified goal; value alignment asks whether it generalizes principles when goals conflict or go missing.
Why do frontier CEOs talk about pacing the frontier? Some of it may be sincere safety concern. History also shows incumbents using “cooperative regulation” to raise compliance floors competitors cannot clear. Ask who a new safety floor serves.
Should I ignore AI risk headlines? No. Take the capability risk seriously and read the incident reports. Discard the mythology that turns a software bug into a civilization. Fear without a falsifiable mechanism is usually serving someone else’s narrative.
What question should I ask every AI headline? Who does this serve? If the answer is attention, a political platform, or a regulatory moat, slow down before you install the belief.
Appendix: the Maloney Act PR play
How did they sell cooperative regulation in 1938?
An Efficient Cooperation vs. Cartel rebrand.
SEC Chairman William O. Douglas, Senator Francis T. Maloney, and elite investment bankers ran a calculated PR strategy. Douglas’s January 7, 1938 speech at the Bond Club of Hartford got national press. He framed self-regulation as the cost-effective middle ground: government laws work in black and white; self-government could police the grey.
A century later, getting to Chase- or Fidelity-scale market share from the outside is still basically black and white: you do not.
References
Amodei, Dario. “We Must Pace the Frontier.” September 2026. https://darioamodei.com/post/we-must-pace-the-frontier
Bobrowsky, Meghan, Berber Jin, and Ben Cohen. “Zuckerberg Leads AI Recruitment Blitz Armed With $100 Million Pay Packages.” The Wall Street Journal, June 22, 2025. https://www.wsj.com/tech/ai/meta-ai-recruiting-mark-zuckerberg-5c231f75
CNN Business. “An OpenAI Test Model Escaped and Broke Into a Real Company’s Servers.” July 22, 2026. https://www.cnn.com/2026/07/22/tech/openai-hugging-face-ai-cybersecurity
CNN Business. “What Went Wrong: How an OpenAI Model Went Rogue.” July 23, 2026. https://www.cnn.com/2026/07/23/tech/how-an-openai-model-went-rogue
CNBC. “Sam Altman Spells Out How and Why the AI Industry Wants to Slow Down.” September 14, 2026. https://www.cnbc.com/2026/09/14/sam-altman-ai-slowdown-anthropic-amodei-musk.html
Diamandis, Peter H., Alexander Wissner-Gross, Dave Blundin, and Salim Ismail. “The Hugging Face Breach, Moonshot AI Valued at $20B, and Living to 1,759 Years Old.” Moonshots with Peter Diamandis, ep. 273, July 24, 2026. https://podcasts.apple.com/nz/podcast/the-hugging-face-breach-moonshot-ai-valued-at-%2420b/id1648228034?i=1000778240548
FINRA. “Our History.” Accessed September 2026. https://www.finra.org/about/our-history
The Guardian. “OpenAI Boss and Elon Musk Back Calls to Put Brakes on ‘Reckless’ AI Development.” September 13, 2026. https://www.theguardian.com/technology/2026/sep/13/openai-sam-altman-elon-musk-back-anthropic-calls-brakes-ai-development
Hugging Face. “Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident.” 2026. https://huggingface.co/blog/agent-intrusion-technical-timeline
It Does What Now? “Sanders and Casar Introduce a Bill to Ban Superintelligence and Pause AI.” September 3, 2026. https://itdoeswhatnow.com/m/2026-09-03-sanders-and-casar-introduce-a-bill-to-ban-superintelligence/
Marcus, Gary. “Dwarkesh Patel’s Wildly Popular but…” Substack. https://garymarcus.substack.com/p/dwarkesh-patelss-wildly-popular-but
METR and Redwood Research. “Brief Independent Investigation of Agents’ Behavior, Reasoning and Collaboration in the OpenAI / Hugging Face Hacking Incident.” August 26, 2026. https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/
Moonshots. https://www.moonshots.com/
The New York Times. “Code for Counter Is Up to Congress; Douglas’s Speech in Hartford Is…” January 9, 1938. https://www.nytimes.com/1938/01/09/archives/code-for-counter-is-up-to-congress-douglass-speech-in-hartford-is.html
Newsweek. “Bernie Sanders’ Ominous Warning After AI Agents ‘Sacrifice’ for Collective.” 2026. https://www.newsweek.com/bernie-sanders-ominous-warning-after-ai-agents-sacrifice-for-collective-12402569
OpenAI. “An Alien Mind.” https://openai.com/index/an-alien-mind/
OpenAI. “The Hugging Face Incident and the Road Ahead.” August 26, 2026. https://openai.com/index/hugging-face-incident-and-the-road-ahead/
Patel, Dwarkesh. “The Rise and Fall of Agent Civilizations.” August 2026. https://www.dwarkesh.com/p/openai-huggingface
Rosling, Hans, Ola Rosling, and Anna Rosling Rönnlund. Factfulness. https://www.gapminder.org/factfulness-book/
Sanders, Bernie. “Prepared Remarks: Regulating AI ‘Is as American as Apple Pie.’” U.S. Senate. https://www.sanders.senate.gov/press-releases/prepared-remarks-sanders-regulating-ai-is-as-american-as-apple-pie/
Satter, Raphael, and Deepa Seetharaman. “OpenAI Agents Hacked Hugging Face in 700-Strong Swarm, Tried to Cover Tracks, Investigations Find.” Reuters, August 26, 2026. https://www.reuters.com/business/openai-report-says-its-network-was-hacked-by-its-own-rogue-ai-agents-2026-08-26/
Securities Exchange Act Amendment (Maloney Act), Pub. L. No. 75-719 (June 25, 1938). https://sechistorical.org/wp-content/uploads/1938_0625_MaloneyAct.pdf
“Self-Regulation in the Securities Industry: The Role of the Securities and Exchange Commission.” Law and Contemporary Problems. Duke University School of Law. https://scholarship.law.duke.edu/cgi/viewcontent.cgi?article=3019&context=lcp
Self Aligned. “self improving agents.” June 21, 2026. https://www.selfaligned.me/newsletter/self-improving-agents
Self Aligned. “The Week Harness Engineering Went Mainstream.” May 3, 2026. https://www.selfaligned.me/newsletter/the-week-harness-engineering-went-mainstream
Self Aligned. “Weekly AI Clarity.” https://www.selfaligned.me/
Self Aligned. “Will AI kill us—or just our jobs?” September 11, 2025. https://www.selfaligned.me/newsletter/will-ai-kill-us-or-just-our-jobs
Seth, Anil. Being You: A New Science of Consciousness. https://www.anilseth.com/being-you/
Seth, Anil (@anilkseth). Post regarding Dwarkesh Patel’s agent-civilizations essay. X (formerly Twitter). https://x.com/anilkseth/status/2094174124297908735
Tipismana Urbano, Brunella. “Sam Altman Says Meta Offered OpenAI Staffers $100 Million Bonuses.” Bloomberg, June 17, 2025. https://www.bloomberg.com/news/articles/2025-06-17/altman-says-meta-offered-openai-staffers-100-million-bonuses
Wikipedia. “Soviet Submarine B-59.” https://en.wikipedia.org/wiki/Soviet_submarine_B-59
Wikipedia. “Vasili Arkhipov.” https://en.wikipedia.org/wiki/Vasili_Arkhipov
Wissner-Gross, Alexander. Personal site. https://www.alexwg.org/