There has been an increase in concern over AI destroying humanity, most recently spurred by tweets from former Anthropic employee Jacob Coxon about how dangerous AI is, and a follow-up tweet from current employee Evan Hubinger, who said “Jacob is correct here — we really do earnestly believe A.I. could kill all humans! I personally think it is >10% within the next decade.”
Some in the AI world are calling for “the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development” and there has been much discussion of trying to slow down or stop all AI research.
Now, a new open letter is circulating, this time from academics worried about the possibility of a “global catastrophe” resulting from AI, calling for political leaders to “create an international treaty, much like the Treaty on the Non-Proliferation of Nuclear Weapons.” It was written by philosopher Molly Gardner (University of Florida) and made public yesterday. If you’re interested in reading or signing it, it’s here: “Open Letter from Academics Concerning Recent Developments in Artificial Intelligence“.
What follows are some thoughts of mine about one approach to mitigating AI risk. I don’t intend to discourage people from signing the “Open Letter from Academics Concerning Recent Developments in Artificial Intelligence“—it couldn’t hurt to do so.
But for various reasons, I think efforts to slow or pause or stop AI research face an insurmountable collective action problem. Even if the well-known corporations working on AI cease their efforts (itself highly unlikely), research and development will continue. AI tools are relevantly not like nuclear weapons, insofar as work to create and improve them is much less detectable, so enforcement will be extremely difficult or invasive. And since the claims about AI destroying humanity are speculative, the scenarios they describe are unprecedented, and AI is not seen unambiguously as a weapon, there will be insufficient will to implement a regime of punishment (on individuals, corporations, nations) severe enough to effectively enforce cooperation.
I don’t know more about AI than people like Coxon and Hubinger, and while I’m not fully convinced of their pessimism, I haven’t read anything (yet) by other knowledgeable people to show me they’re wrong. So let’s suppose they’re correct to at least this degree: AI poses a serious and pressing risk to the survival of humanity.
What should we do?
Both individually and collectively, we need to develop the means to survive without the internet (and other device and communications networks). Without networked computers, the ability of an AI to do much harm to us decreases dramatically. If we find ourselves imminently facing an AI threat, what we may have to do to save our lives is what many have argued we should do anyway to save our souls: disconnect.
It won’t be easy. This isn’t just about giving up social media, or the hassle of not being able to order things online. And the fact that we all lived without the internet just a few decades ago isn’t all that helpful to us today. Now, every company we rely on for everyday items like food, clothing, shelter, and transportation, every public utility we depend on for things like electricity and clean water, every doctor’s office and hospital, every government agency that’s supposed to provide us with public safety and maintain a good society—they all rely on computer networks. Were we to shut them down, most of these organizations would be barely functional, and the downstream effects of that would be destructive chaos. At that point, AI would be superfluous: we might destroy ourselves.
What we need is non-network redundancy. That is, we need to have in place the means by which to function in a situation in which all of the computer networks have been shut down. By “we” I mean every government agency, every utility, every firm above a certain size, every hospital or medical office, and so on. By “the means” I mean not just plans, schemes, ideas (which some agencies and institutions have), but also the physical infrastructure, the equipment, and the employee know-how to adequately function in the absence of computer networks. (Some government agencies have something like this in place, but not all.) There also needs to be in place the means by which to communicate to the public how to navigate a non-networked world, should that be necessary. Additionally, governments may need to improve their ability to keep track of or monitor potential AI threats, so as to better know if or when it is advisable to pull the plugs.
A demand for action on the AI threat that calls for something highly unlikely to successfully happen may have value as a message to others, including politicians, to pay attention to the problem. But it would be even better if it asked for something useful that can be done. People around the world should be calling on their governments for regulation that requires non-network redundancy.
Will it be easy? No. Will it be cheap? No. But such regulation is of a type that is familiar to people and governments around the world. We have regulatory schemes for workplace safety, for non-discrimination, for accounting, for engineering, and so on. What I’m proposing is a kind of thing we have done before.
This is not a new idea. In some ways it is just an expansion of initiatives like CI Fortify by the Cybersecurity and Infrastructure Security Agency (CISA), a branch of the Department of Homeland Security.
And it’s not a radical idea. It isn’t a call to ditch computers or to necessarily forgo the benefits that AI may bring. (It’s not a call for a Dune‘s “Butlerian Jihad“.) Rather, it is a form of disaster preparedness.
Does non-network redundancy eliminate all AI risk? Of course not. Eliminating all AI-risk is not possible. Even if there is no further development of the technology, some humans may use existing AI to develop a deadly mass weapon (or something else dangerous, a la “Ice-nine“). (One might note that “eliminate all risk” is not generally a rational regulatory aim for a technology, but we can leave that debate aside.) But, perhaps alongside other measures, widespread non-network redundancy could go a long way toward reducing the risk of catastrophe.
(The foregoing is about non-network redundancy for government agencies, utilities, businesses, and other organizations. What about individuals? Should each of us be ready—really ready—just in case we have to live without the internet? To be honest, I don’t know. Everyone, it seems, should be prepared to weather some kind of emergency. But beyond that, I don’t know what to recommend, or whether it would be a good idea to do much more. For a few reasons, I don’t think a nation of “doomsday preppers” will make for a good society. But it does seem valuable for individuals and communities to be at least discussing these issues.)
I eagerly signed this open letter and encourage others to do so. But I don’t think regulating and even perhaps banning AI research is anywhere near all that needs to be done. This is a HUGE problem in my view and one that requires a tremendous amount of collective effort to navigate. I’m pessimistic we’ll be able to navigate it.
I heartily second this, but also add I’m pretty mad at us as a species for putting ourselves in this position. And it’s not as though this was unforeseen. Existential risk has been on all of our minds from the beginning.
I’m unconvinced that there is a radical difference between detectability / enforcement of an AI limitation treaty, and the equivalent situation in nuclear nonproliferation. In both cases, you need a somewhat scarce and somewhat monitorable material (cutting edge GPUs vs. uranium ore); in both cases, you need an industrial facility with a fairly large footprint (data farms vs. centrifuges); in both cases, you need a reasonably high level of expertise. If you’ve a source that says otherwise, I’d be interested in seeing it. (Nuclear nonproliferation is actually quite difficult, but we’ve still been somewhat successful at it.)
I hope you’re right. Even if you are, though, things might change. This article in Nature suggests that a decentralized, smaller-footprint alternative to massive data farms is possible (they argue in favor of this alternative), and also notes improvements in chips that may facilitate this.
That’s true, but most proposals for an AI pause don’t involve an indefinite shutdown, just a few years where we can use existing systems to work on the alignment problem.
I believe that AI 2040: Plan A even gives a bit more detail on how they think such a coordinated enforcement regime could work (largely because the manufacture of the relevant chips, and the equipment for making those chips, is concentrated in so few companies right now, and there is enough bottleneck in demand for these things that the offer of functioning, but trackable, chips could persuade some reluctant participants to join).
https://ai-2040.com/
I hope that this is the case. Of course, one key difference is that nuclear capabilities are still very much controlled by governments while AI companies are motivated by private interests that don’t have to include the safety of their fellow citizens. All it takes is one bad actor willing to take risks for a competitive advantage.
If I have to choose between being ruled by OpenAI or Anthropic, or by the governments of Russia or China (let alone North Korea), then I, for one, welcome our new corporate overlords.
But I’m not sure that is the choice here. If they slip up in their effort to create faster and better forms of AI before their competitors do, then our overlords won’t be the CEOs of Anthropic and OpenAI. They will be non-human forms of intelligence that may have good reason to consider us a threat to their very existence.
Oh, sure: agreed. I just meant (referring back to your previous comment) that I don’t think nuclear weapons being controlled by autocracies rather than private companies is especially an improvement.
So it really IS like Los Alamos! 😉
I’m not sure I understand your non-network redundancy proposal. I don’t think you’re saying we should shut down the Internet right now. But if you’re saying that we should wait until AI gets dangerous and then shut down the Internet, it seems to me like that’s too late. (At that point, AI will already have engineered a deadly virus or gotten control of nuclear weapons or whatever, so turning off the Internet will not save us.)
I am not going to pretend to know enough about the capabilities of AI or the operation of nuclear weapons (etc.) to state when it might be advisable to shut down computer networks. As I say in the post, “governments may need to improve their ability to keep track of or monitor potential AI threats, so as to better know if or when it is advisable to pull the plugs.” If it is in fact highly unlikely that monitoring of potential risks will be sufficiently effective (however we want to specify “sufficiently” or “effective”), then sure, that’s a problem with my proposal. That seems overly skeptical to me, but I’m no expert. (I’d be happy to hear from experts on this.)
My interpretation of the suggestion is that shutting down the internet is one potential response to a larger event than the HuggingFace incident that is still short of existential disaster. (And a shutdown of the internet is also the potential outcome of such an incident, if some rogue AI systems get widely established, but are unable to achieve their other aims, and just end up making networks not safely usable by humans.)
Justin did not, so far as I can tell, argue for “shutting down the internet”. He argued, quote, “Both individually and collectively, we need to develop the means to survive without the internet.” Non-network redundancy doesn’t mean “no internet”. Those are very different things.
We are in the world of Silo. At least it is a good story…
Any solution here depends on the above discussed question whether AI development can be regulated. That is partly a technical question, I suppose.
If it can be regulated, there is the further political question of having the will to regulate (or, even nationalize). I think Europe would be partner in this (for now, since fascists (oh, sorry, populists) are coming up fast and their interest lies in freely abusing AI to undermine democracy). But I see no chance of this happening in the US, which is the primary mover in this story.
So, there we are. (I don’t think plugging out is an option: who, when, how, and why? Of course, this much is obvious and we know this from Battlestar Galactica: never keep your most important assets in the net. But what are those assets? For Trump, it is probably is golf club…)
I wrote this about two months ago on Facebook:
I thoroughly agree with the proposal for non-network redundancy, and I note that it is a central component of both deterrence and potential warfighting.
Frankly, having critical infrastructure directly connected to the Internet was a stupid idea to begin with, and if the AI threat prompts rethinking that, it’s all to the good.
I don’t think I have the energy to rigourously defend my opinion, so I’ll just state it for the record: this is bullshit.
It is corporate hype and maybe also fantasy. But its purpose is to draw investor cash, which these companies are setting on fire like nothing else we’ve seen before.
As for the apocalypse, it’s already here: it’s 2.6+ degrees of warming by century’s end as the most optimistic scenario. I’mma worry about that real thing, not PR stunts.
I’m not entirely sure how you can state anything for the record if you’re anonymous.
But, for the record and under my own name:
– 2.6C warming by 2100 will lead to lots of misery and if we can avoid it at acceptable cost, we should (“acceptable cost” does not include the permanent impoverishment of much of the world). It is in no sense “the apocalypse”.
– that misery is fairly trivial compared to a large-scale nuclear exchange. This is closer to an apocalypse but is very unlikely to kill everyone.
– the worst-case AI risk scenarios actually will kill everyone.
“The actual worst-case AI risk scenarios actually will kill everyone.”
Is anyone willing to either work me through this part or direct me to a source which does? Specifically, I understand what the worst-case AI scenarios are, but I have no idea (a) how likely they are or relatedly (b) how they might realistically come about. Like, I hear Eliezer Yudkowsky and co. talk about how we’ll hit a singularity and then AI will release a new plague or whatever, and I’m told we can’t rule out that such a scenario is accidentally compatible with AI meeting the goals we’ve given it. But I don’t quite follow how we actually get from here to there—like, how do the possibilities translate into plausible outcomes I should actually worry about? These aren’t criticisms; I’m genuinely curious because I never see this part explained when I’ve gone looking (probably because I’m not looking well enough).
The only scenarios I’ve encountered that seem remotely plausible to me are:
1) AI gets good enough to allow a group (or even just one jerk) to develop, manufacture, and spread a disease that kills everyone.
2) Enough militaries automate themselves sufficiently for a malfunctioning AI to cause enough damage to make someone mad enough to launch nukes, and once missiles start flying, everyone with a nuke decides it’s now or never.
All of the other AI doom scenarios seem to me wishful thinking, usually on the part of people who have a death wish or on the part of people who (even if they don’t admit it to themselves) think at some level that it would be kind of cool if there were robot gods. And of course a lot of AI doom lately is from people whose stock portfolios depend on how impressive people imagine AI will be in the indefinite (but very near!!!! we swear!!!) future. But this is not my main area of expertise, so maybe I’m missing something.
As to (a): I guess it doesn’t matter whether the likelihood of an outcome is small if what we stand to lose is great enough. Even if the probability of catastrophe or the end of humanity is small it should be addressed. However, surveys of people working in AI yield show that they tend to think the probability of something going horribly wrong in the next two decades is high, like 5% or 10%. Part of the calculation assumes we fail to solve the same collective action problems that have kept us from dealing adequately with climate change or the creeping up of fascism. (b) It’s fair to ask for scenarios to be spelled out, but I think part of the issue is that whatever happens may well be something we could not have guessed. This is what makes this technology different from other technologies that present risks. A place to start though, in describing scenarios, is to notice there are two lines of thought particularly wrt existential risk: (1) The first stems from the idea that at some point AI will develop a self-preservation bias and this will mean that it will compete for resources etc. and squeeze out humans. (2) The second emphasizes the idea that AI will be so superior to human intelligence that at some point it will no longer even notice us, or notice what damage its activities do to us. I like Hinton’s often repeated question: when has a less intelligent thing ever controlled a more intelligent thing? As for catastrophic risk (bad, but not the end of humanity), it’s easy to think of examples where ‘determined’ AI agents misunderstand instructions or where rogue humans or state actors step in to get AI help designing a horrific weapon (e.g. like a plague) or destroying infrastructure like an electrical grid.
Nope. None of these AI doomers ever propose an actual scenario or AI-driven path to human extinction. Even the super-virus and nuclear war scenarios are essentially mainly human.
AI-2027 provide a very detailed scenario: AI is superintelligent and misaligned; humans collude in letting it largely automate the economy, which it then controls; eventually humans are getting in its way so it kills them with bioweapons.
Soares and Yudkowsky provide a similar (if imho less plausible) scenario.
There are lots of others that are easy to find online (see Mercy Corredor’s link above).
You might find some or all of these scenarios very implausible but “no-one ever proposes an actual scenario” is just false as a statement about the literature.
Sorry, but I don’t regard those as actual scenarios. They’re fantasies. Like “eventually humans get in its way so it kills them with bioweapons”. That’s fantasy.
I agree with Michel: This is bullshit. It’s a fantasy narrative engineered for the express purpose of maintaining hype. There are actual harms that so-called “AI” has already brought about, which we could (and should) be focusing on, not these ridiculous phantasms that just so happen to take it as a given that “AI” corporations still have so much more power they can eek out of their models, if only we give them hundreds of billions more.
Oh, but now they’re saying that they’ll “slow down” for the sake of focusing more on “safety.” Brilliant! Now they can cast the slowing down of progress toward their claimed goals (“super-intelligence”) as a responsible decision they made rather than a reality that was forced upon them because there’s no more “progress” to be made and their goals were foolish and unrealisable. Better still, they can play as our protectors, sagely guarding us from the far-off future demons of their imagination while ignoring the actual evils they’ve already foisted upon the public. Our heroes!
My version, for what it’s worth (nothing original here really, but maybe I can avoid some of the tics that you tend to find in the AI-safety literature).
What I mean by “the worst-case AI risk scenarios” is that if we build misaligned, agentic superintelligence, it is quite likely to end human civilization and humans more generally. Terminology:
– ‘superintelligence’, meaning it is much better than the best humans at any given cognitive task
– ‘agentic’, meaning it persists and acts in the world in a goal-directed way (it isn’t just a transient chatbot);
– ‘misaligned’ meaning its primary goal isn’t human flourishing.
There are plenty of reasons to doubt that we will or can build agentic superintelligence, or that it will necessarily be misaligned if we do. I interpret the question as: conditional on us doing so, how do things go wrong?
The starting point here is just: in this scenario we would be sharing the planet with an agent much smarter than us but with different goals. That has not worked out well for nonhuman animals, including the ones whose noncognitive capabilities are much better than humans’. Even prior to thinking through specifics, that makes me very nervous.
But getting specific: at present Earth’s industrial economy is not automated and needs humans in the loop. But there is a clear trend towards automation (drone swarms, self-driving cars, major progress on humanoid robots) and superintelligence would accelerate that trend. A misaligned agentic superintelligence that had control even an initially localized chunk of the economy (say, North Korea’s) could be expected to outmatch the human-controlled part of the economy in fairly short order. A situation where Earth’s industrial resources are controlled by agents much smarter than us who don’t wish us well, is pretty much definitionally the end of civilization even if it doesn’t actively try to kill us.
*Would* it be given control of even some part of the economy? Well, the reason people are building AIs is to supercharge economic, scientific, and technological progress, and that requires some level of access. And for a superintelligence there would not be much gap between ‘access’ and ‘control’: ex hypothesi it is much better at IT than the best humans.
(Could we somehow keep it ‘in the box’, providing advice but not gaining any control? Well, again: ex hypothesi it’s much better at us at all cognitive tasks, including persuasion and manipulation.)
Couldn’t the threat of force prevent an AI that controls a small chunk of the economy from taking over? Well, now we’re in a space where it’s hard to avoid sounding science-fictional. But there are already several technologies that an AI in this situation could use to decimate humanity: man-in-the-middle methods to take control of strategic nuclear arsenals; autonomous drone swarms; bioweapons. There are more exotic possibilities (nanotech) that don’t violate any law of physics and that a superintelligence might figure out. And of course there might be any number of ways to kill or control humans that we haven’t thought of but that it will, because it’s smarter than us.
I agree with Daniel Weltman that there are also scary, if not quite “everyone dies” scenarios where AI could lead to civilization being badly damaged or destroyed even without reaching superintelligence. I think the main ones are bioterror and AI escalating world tensions so that a nuclear war occurs (again, largely in agreement with Daniel). I don’t think it is implausible to imagine a misaligned AI with only somewhat greater capabilities than the HuggingFace swarm triggering a nuclear war directly for its own inscrutable reasons or on behalf of malign humans: nuclear weapons might not be networked but the detection and communication infrastructure they rely on is. I also think AI-driven tech advances might undermine nuclear deterrence, which relies to a substantial degree on the fact that ballistic-missile submarines are not trackable.
You can read about some paths to extinction here (and also why we’re likely really bad at predicting what this would actually look like): https://www.lesswrong.com/posts/LAPa2jxoq3n63GzTr/some-ways-ai-could-kill-us-all
Yeah, the rationalists. A millenarian cult with a political agenda. But indeed now you’re getting to the heart of the issue: the only way to save us from the robot uprising is a techno-fascist aristocracy.
Just a note, 2.6 is likely to be catastrophic because of tipping points. There is plenty of analysis of this, although of course, there is no certainty. But there is still more certainty than regarding the worry of a runaway AI. No one says that the latter is not important and nothing should be done, but it sounds incorrect to me to say that 2.6 is not a bigger danger, I think it is. And, with present trajectories, it is pretty much certain to happen, unlike the AI apocalypse.
Right, but what does “catastrophic” mean? If it means “by 2100 areas of the planet that currently support hundreds of millions of people become uninhabitable” then I agree that’s extremely bad, but it doesn’t end human civilization, and the timescale is very relaxed compared to the high end of AI risk (could end human civilization in the next few years) or to strategic nuclear exchange (could kill tens of millions in the next few hours and end human civilization within a year).
Even if OpenAI and Anthropic have been running the preposterous “scare everyone to death” marketing strategy, lots of people that have nothing to do with these corporations are sounding the alarm. Are we to think that all of these people are secretly being paid by the AI companies to “hype” them in this ludicrous way?
Get a grip.
This is the same kind of corporate hype as Taco Bell hyping the freshness of their ingredients by pointing out how many people have gotten food poisoning from eating it. Maybe this will work on someone. But OpenAI seems to have tried pretty hard to hide evidence of several of the worst hacks their systems have perpetrated (including the hack of OpenAI’s own systems, which outside investigators have not been permitted to look into).
To live off the grid, individually, is very easy. You just need a solar panel, water desalination and a garden. For communities, the main problem is that there is no off the grid economy. Try to live without money, ie without banks, that’s the hard part. The only alternative is economy of exchanging goods, which can work in very local communities. I’d like to see the development of an analog coin (in contrast to the stock market, banks and Bitcoin).
This is such a good post. Not in the least because it cuts to the material heart of the issue and thereby exposes the hard choice in front of us, the one we all know is there. It is just possible that our civilization has erred in making the networked computer the sine qua non of its major operations. Make that physical object less essential in your institutions and you immediately cut off a growing source of risk.
On the ground, many folks have been loudly informing various institutions–i.e. healthcare institutions, school boards–that retaining the ability to run operations offline will be a critical strength in the 21st century. It will also become a matter of human rights, as many will increasingly be unable to live their lives without exposing themselves (and their families and children) to serious risks and harms. No liberal or democratic society can possibly permit such coercion.
What Justin correctly suggests here is that we can’t wait around for the big, captured institutions like nation-states to do this work for us. So we must return to the various smaller institutions we rely on and we ask how we’re going to get them to become AI-resilient in this way. Collective action problems at that scale are solvable: eliminating tech bloat in the unversity and providing spaces and resources for ‘analog’ learning would be a good start.
This whole “AI will kill everybody” is part of a techno-religious movement which aims to replace democracy with a fascist aristocracy (composed mostly of tech billionaires). According to this techno-religion, the fascist aristocracy is the only way to save us from AI killing all humans.
The techno-religion is now often called TESCREALism. It includes transhumanism, extropianism, singularitarianism, cosmicism, the rationalist movement, effective altruism, and longtermism. The political root of TESCREALism comes from Nick Land’s accelerationism. Nick Land is very popular among the tech billionaires.
The techno-religion is a millenarian religion, in which the robot uprising plays the role of the Christian apocalypse, and the new fascist leaders are our saviors. “AI doom” is a religious myth, which supports a political agenda.
So just be careful about what you’re signing up for.
This seems to carry risks of political repression. Various countries have recently restricted internet access to suppress political protest/unrest. One reason these attempts had limited to success is that _some_ internet systems needed to be kept online for essential services — good VPNs often allows users to access any information they want through the internet services still available. No-network redundancy will make more thorough internet shutdowns feasible. Of course, this may be a small risk compared to AI risks….