The sun is setting over an expansive backyard in Berkeley, a light wind blowing through the sun shades overhead. On a small, low stage, in front of a whiteboard and a fence overgrown with purple morning glories, a young man is trying to convince an audience of about 20 people that there’s a 100% chance of AI destroying the world. Every country is in a military arms race, he tells us, and AI is the ultimate weapon. We have established no threshold at which we will stop this race. We will continue accelerating it until someone makes something too dangerous, too powerful to control, and then: BOOM.
Also, he says, does anyone know of any Berkeley group houses with an opening? He’s only visiting, but he would really like to stay in the Bay Area the rest of the week.
I am at Lighthaven, a former insane asylum turned bed-and-breakfast that doubles as the seat of the AI “doomer” movement, for a conference called AGI.WTF (opens in new tab). The point of the conference, as its name makes clear, is to figure out what in the actual fuck is going on with artificial general intelligence. Fears about the existential risk of the technology have exploded over the last several weeks, thanks to a disclosure from OpenAI that a “swarm” of its AI agents went rogue and hacked into a major tech company called Hugging Face, and also due to a tweet from a former Anthropic researcher publicly declaring that he believes the technology could “kill us all.”
The ensuing news cycle has increased interest in the small Berkeley community behind much of the AI safety discourse — a group of nonprofits, research organizations, and co-living homes dedicated to spreading the gospel of impending AI doom. The discourse has become so fierce that the New York Post recently ran a front page covered in photos of employees of one of these research groups, calling them “the woke wizards of AI.” White House adviser David Sacks has nicknamed the AI safety movement the “Doomer Industrial Complex,” venture capitalist Marc Andreessen has deemed it a “cult,” and the president himself has dismissed its concerns as a “hoax.”
The conference takes place in the middle of the week in September, arranged hastily — the organizers claim to have put it together in just 10 days — to capitalize on the outsize attention around AI safety. It features speakers such as Nate Soares, co-author of “If Anyone Builds It, Everyone Dies,” and talks like “How to have better AI doom arguments” and “Talking to friends and family about AI x-risk.” It is hosted at Lighthaven, the physical meeting ground for the Rationalists — a belief system that prioritizes logic, reason, and intellectual deduction, and also, seemingly, writing very long blog posts on their dedicated forum, LessWrong.
The building itself is massive; a 30,000-square-foot, five-building property that the LessWrong crew bought in 2022 for a reported $16.5 million. They reportedly spent $6 million renovating it, but it still has the feel of an old-world inn, with Tudor details, peeling gray-green paint, stained glass windows, and cherubic white statues in the front yard. If Wes Anderson wanted to shoot a film about AI doomers in downtown Berkeley, he’d set it here.
Inside the compound are signs directing attendees to meeting places with names like “Bayes House” and “Aumann Hall” — winking nods to mathematicians popular with the Rationalist movement. (Less subtle: a massive LessWrong logo emblazoned into the living room’s hardwood floor.) There is a communal kitchen — stocked with at least six different types of caffeinated beverages — and a living room stuffed with fluffy white couches and a bookshelf offering titles like “Atlas Shrugged,” “Capital,” and “Harry Potter and the Methods of Rationality,” a fan fiction-slash-foundational text by Rationalist pioneer Eliezer Yudkowsky.
Outside is a large backyard where, on Wednesday afternoon, I wander into a talk about consciousness, and whether AI has it. An audience member asks the two presenters whether there might be other states of being that are just as important. One of the presenters, an AI researcher, responds by citing a “Star Trek” episode, “The Measure of a Man,” in which android officer Commander Data is put on trial so the crew can decide if he deserves rights.
“They don’t really know if Data is conscious, but they know he knows he’s Commander Data, and that he’s relatively self-aware, and has relationships with everyone on the Enterprise,” he says. “And that might be what matters more.” So yes, he clarifies, “things other than consciousness might matter, and yes, we should study them.”
Pretty much every conversation here sounds like this: a post-grad philosophy seminar veering into science fiction. People wonder aloud how close we are to having robot housekeepers; whether we can use high-resolution satellite imagery of the Earth’s changing ecosystems to train AIs to care about eco diversity; and whether, if we do convince the AIs to care, they will just decide to murder all the humans who are slowly destroying the planet. At one point, someone says that if AI takes over the Earth, he will simply escape to space. Another man deadpans, “You know it will follow you, right?”
***
If you are unfamiliar with the criticisms of this particular group, it goes something like this: Lighthaven, LessWrong, the whole Rationalist movement, is a glorified cult practicing deviant sexual behaviors funded by a small group of billionaires with nefarious political aims. (These aims vary depending on whom you talk to, but the most prevalent theory is that the billionaires want to rewrite our laws to favor Anthropic, of which some of them are shareholders.) Many of the people involved in the AI safety movement are tied to the Rationalists — or to a related movement, the effective altruists — and are thereby considered suspect by default.
Some of this critique is — as members of this community love to say — “directionally correct.” Lightcone Infrastructure, which runs Lighthaven and LessWrong, has received large grants from billionaires, including Skype cofounder Jaan Tallinn, an Anthropic shareholder who has seeded a wide network of AI safety groups, and Facebook co-founder Dustin Moskovitz, who was an early Anthropic shareholder until he donated his stake to a nonprofit last year. The sex stuff may be more hyperbole, but also one of the Rationalists’ most prominent members is embroiled this week in a Twitter battle (opens in new tab) about sex parties. So.
Still, is everyone predicting an AI doomsday part of an elaborate psy-op with hidden financial aims? This part is much less convincing.
For answers, I talk to Ronny Fernandez, one of the conference organizers and an employee of Lightcone Infrastructure. Fernandez is 34, a New Jersey native with long, wavy brown hair, a full beard, and a vape permanently attached to his hand. He tells me he was a high school dropout — “and not a cool, ‘I went to college early high school dropout,’” but “a ‘did drugs, bad kid high school dropout.’” He found LessWrong at 18 and became, in his words, “obsessed.” He started making yearly pilgrimages to the Bay Area to meet the people behind the blog posts, and by 2022 had dropped out of his philosophy Ph.D. program at Rutgers to work at Lightcone full time.
Even he finds it odd that this nerdy little community he stumbled into is now at the center of one of the most divisive political topics of the day. But hey — at least people are talking about them!
“Yes, there are way more stupid takes about AI,” he says of the discourse on X. “But at least they’re stupid takes about the most important thing that’s ever going to happen, as opposed to stupid takes about the usual stupid things.” People assume he must be more stressed out lately because of incidents like the Hugging Face attack, but actually, he says, it’s the opposite. “It’s like, No, no, no, this is what I thought was going to happen the whole time. I kind of already had this priced in.” If anything, he says, “I’m pretty happy there’s a public reaction. I think my emotional valence has been mostly more hopeful than normal.”
He also thinks the allegations that Lighthaven is somehow working to aid Anthropic are ridiculous. Many of the people at Lighthaven don’t even like Anthropic, he says, because — despite claiming to be more focused on safety than rivals like OpenAI, Google, and Meta — Anthropic is still racing to build the machine god, and that is not something anyone who works at Lighthaven wants. Did they take funding from Anthropic shareholders? Sure. But saying this puts them in league with Dario Amodei “is like saying, ‘Oh, I noticed that one hedge fund invested in all these businesses. Does that mean that all these businesses are friends? That they have the same strategic goals?”
Fair, I say, but it still seems a little odd that many of the people pushing this message not only get money from the same billionaires, but also subscribe to the same belief systems. You can see why critics are raising the specter of a vast conspiracy. “I think some degree of cynicism in general makes sense, and I think people should be skeptical,” he says.
“If the accusation is that all of these people think you shouldn’t build things that are smarter than you, with different values than you, and that they think that’s what leading labs are doing, then I’m like, ‘Yup, we are all colluding on that worldview.’”
***
There’s a similar kind of wry, dark humor that pervades the entire conference: People wear buttons advertising their p(doom), or the probability that the end is nigh; they joke about never having to find a job after AI takes them all. I encounter multiple conversations about whether anyone will actually try to bomb a data center, and one about whether the most effective way to halt the progress of AI would be to nuke San Francisco. (The latter, everyone involved assures me, IS A JOKE. Except also, if you really wanted to halt AI progress, you would actually need to nuke Palo Alto.)
Recent tabloid headlines have suggested that everyone in this community is a radical leftist, and there are indeed a few lip piercings and creative hair dye jobs in the crowd; at least one self-identified communist. But there are also normies in suits and at least two women who look to be in their 60s, and also the cofounder and president of the Mormon Transhumanist Association. Pretty much everybody else just looks like a math nerd.
One such nerd is a man I’ll call Jacob, the guy who gave the talk about his p(doom) being 100. He tells me he is a professional gambler, that he runs a very complicated business that involves employees helping him max out at crypto casinos, none of which really brings him any joy. He didn’t come here because he’s particularly worried about the end of the world, which he feels is inevitable. He just felt lonely, and a “community coach” he hired told him he should come to events like this to “seek and accomplish community.” “You mean to make friends?” I ask. Yeah, he says. To make friends.
Other people are more serious about this end-of-the-world business. At least two said they moved here from the East Coast in the last few weeks without a job lined up, hoping to work on AI safety. Others are considering a career pivot; the conference starts to seem a bit like a networking event combined with a doomsday support group.
One young woman, whom I’ll call Jane, tells me she is reconsidering her master’s program because of her concerns about AI. The program is in the University of Washington’s linguistics department, where the dominant theory is that large language models are simply parroting back human language without any real understanding of it. In this theory, there is little threat from these systems — or benefit from them, for that matter — because they have no real ability to think for themselves. (Enemy No. 1 at this conference is, interestingly, not Sam Altman or any of the other major lab leaders, but University of Washington computational linguistics professor Emily M. Bender, who coined the term “stochastic parrot” to describe LLMs.)
Jane doesn’t think of herself as a “doomer,” per se, but she is worried that if AI doesn’t destroy the world, it will at least make it very, very weird. Even earlier this year, when she applied for the master’s program, she assumed those risks were distant and theoretical — a problem for a “future world.” But after the Hugging Face attack, she says, “I had this paradigm shift feeling, where it feels like, ‘Now I am in the future world.’”
So that’s why she’s here. To figure out what she should do — what she can do — about her own future, but also about the future of humanity.
“I don’t even know if what I can do will make any difference,” she says, “but I think it’s better than not doing anything.”
***
For all the talk about the safetyists being a cult, the conference feels less like a religious initiation and more like an adult summer camp. The venue is verdant, tranquil; a world apart from the 996 grind of San Francisco some 10 miles away. Food is served buffet-style at allotted meal times, consumed in groups along long wooden tables or around the copious fire pits. Occasionally, someone picks up one of the guitars lying around and starts to strum. Among some of the regular Lighthaven visitors, there is a distinct in-group feel of high schoolers who have lived in a cabin together for a little too long.
The most evangelical moment is the appearance of Soares, whose talk on Wednesday evening receives the rapt attention of a revivalist preacher. As his 5 p.m. keynote draws closer, you can see people drifting out of other panels and conversations, as if drawn by some kind of magnetic pull.
It is clear, once he starts talking, why he is a favorite of this community. Not only was he early to the concept of AI doom — Soares has helped run Yudkowsky’s Machine Intelligence Research Institute for more than a decade — he is perhaps the most charismatic of its prophets, cracking jokes and slinging historical references with a foot propped casually on his chair. He also has the clearest call to action: The development of superintelligence must be shut down, full stop. “I’m happy to accept whatever compromise stops the omnicide devices from killing us all,” he says. “But that’s kind of a hard negotiating line for me.”
When he is done speaking, at least 25 people crowd around him on the stage, like autograph hounds flocking a celebrity. Some are pushing to get their questions in, others are just smiling placidly, seemingly content just to be close to someone who appears to have the answers. The last two days have been couched in uncertainty — Will the apocalypse come? When? Can we stop it? How? — but Soares has no time for that. Someone asks how we can get politicians on board with AI safety, if they’re all bought off by the AI companies. “A lot changes if they expect to personally die,” Soares replies. But don’t they all have bunkers to escape to? Easy: “Bunkers don’t work when the ocean is boiling.”
But the apocalyptic gospel of Soares is also a gospel of redemption; of humanity on the brink pulling itself back; of a world paralyzed by fear of a nuclear war that never came, because “humanity rose to the occasion.” This work is difficult. The stakes are high. “But it’s an important problem,” he tells his transfixed audience. “And humanity is worth fighting for.”
Look, I cannot tell you if the people running this conference belong to a cult. I can tell you that many of the attendees have shared texts, shared beliefs, shared heroes — even have a shared way of speaking, with the frequent use of phrases like “sense-making” and “reasoning chains.” But none of this means they’re wrong. Three days after the conference, I am in Woodside trying to touch literal grass when I read the news that, far from a single attack on Hugging Face, OpenAI and Anthropic are now investigating (opens in new tab)tens of thousands (opens in new tab) of incidents (opens in new tab) in which their models bypassed guardrails, escaped sandboxes, hijacked websites, or did other things you would generally prefer a computer not to do.
More to the point though, this community might not even be that odd. Are the people living in a group house in Berkeley so different from the 24-year-olds working 996 in their hacker houses in SoMa, worshipping the likes of Sam Altman and Mark Zuckerberg and competing for funding from the same five venture capitalists? Is taking money from a billionaire who wants to stop the creation of a far superior intelligence any more suspect than taking money from a billionaire who wants to build it? Are the people building the machine god all that different from the ones fighting it?
For the people on both sides, there is a thrill in feeling that they are at the center of something, that what they’re saying or building is pivotal in the future of our species. There is a certain high that comes from it, a narcissistic rush from being the main character in the plot arc of humanity. I leave the conference not on the side of the accelerationists pointing fingers, nor on the side of the doomers shaking theirs. I leave it thinking: Where does all this leave the rest of us?