Various parties concerned about the risks that artificial intelligence (AI) pose to humanity have called for a pause in its development. But such a call faces a difficult collective action problem.
Can it be solved?
I have been skeptical (and suggested an alternate type of approach to AI safety here). But a new report argues that a pause is indeed possible.
Drafted by the Working Group on AI Pause Feasibility, an interdisciplinary group led by statistician William Fithian and philosopher Wesley H. Holliday (both of Berkeley), the report‘s focus is:
a scenario in which world leaders would prefer pausing to not pausing if they could be confident that others were also pausing or that defections from a pause would be detected within a reasonable timeframe. In such a pause-willing scenario, is a pause for at least a decade feasible?
Their answer is that a “hardwired pause” is feasible. What is a hardwired pause?
A hardwired pause would stop the production of training-capable AI chips, which are needed to develop more powerful AI models. Over time a hardwired pause would also phase out most or all of the pre-pause stock of such chips. It would replace them with inference-only AI chips, which are capable of fast, energy-efficient AI inference on approved models but are not practically useful for training new frontier AI models, even if stolen or seized, due to the way the chips are constructed.
Such a pause is feasible, they argue, mainly because of two sets of facts. The first concerns chip production:
The supply chain for AI chips is highly concentrated with several chokepoints, bolstering governments’ abilities to enforce rules on the production of such chips. In addition, bypassing existing supply chains is extraordinarily difficult.
The second concerns the demands of training new models:
Because training new models requires large numbers of training-capable chips, the feasibility of verification is an inherently quantitative question: while it is next to impossible to detect an evader who is using a single computer to train a small AI model, it is far easier to detect an evader who is training a frontier-scale AI model using hundreds of thousands or millions of chips... To research and train a new model that would significantly advance the frontier after a hardwired pause, absent an unforeseen methodological breakthrough, an evader would need to marshal a significant fraction of the world’s total computing capacity for months or years at a time.
The above quotes are from the executive summary of the report. The full report is here, and a press release from Berkeley about it is here.
Discussion welcome.
(via Wesley Holliday)
So people talk about this as a collective action problem. But it isn’t actually clear that’s what’s going on. Based on their AI policy, I think China just doesn’t take the more extreme predictions seriously.
An additional problem. From what I can tell—though others should correct me if I’m wrong—the proposal would also make it impossible for people to fine tune their own open-weight models. It would also create a large bureaucratic obstacle for fixing alignment problems in existing models.
From what I can tell the proposed policy would also lock in US dominance of AI. I’m not sure why other countries would go along with this.
Thanks for engaging. Our working paper doesn’t assess the probability that leaders in the U.S. and China will want to pause, only the feasibility of their achieving it if they decide they want to. We argue it can be done, if they decide they want to do it.
Chinese leaders have expressed concern about serious AI risks, but not (yet) a desire to pause; e.g., https://www.insurancejournal.com/news/international/2026/09/14/884930.htm
It’s not unusual for leaders to take new risks more seriously once they begin to materialize. Political discourse shifted considerably after the HuggingFace attacks, even though those had no severe consequences; see, for example, Ro Khanna’s change of heart: https://x.com/RoKhanna/status/2098074525313216742?s=20
I also recall reading in early 2020 that restrictions on international travel were an overreaction to a new respiratory virus (e.g., https://newsinfo.inquirer.net/1224016/who-chief-says-widespread-travel-bans-not-needed-to-beat-virus). Opinions can change quickly, and policy preferences with them.
Whether, or under what circumstances, fine-tunes of near-frontier models would be allowed, is a design option not central to the proposal. Fine-tunes can of course be useful (as can training new frontier models), and the question of whether the benefits outweigh the risks would be a matter of discussion.
Presuming that other countries would oppose a pause on the grounds of “locking in U.S. dominance” neglects the possibility that they could expect, on the present trajectory, either (1) that American companies would become more dominant over time, or (2) that continued American dominance in AI would place other countries at a greater disadvantage as models became more powerful.
Regardless, a possible reason why other countries would go along with a pause would be a shared interest in avoiding catastrophic risks.
Thanks for the reply.
China does express some concern, but their AI policy so far seems to indicate that they don’t regard the more apocalyptic scenarios as that likely. I agree that that might change in response to evidence. But they do not seem to find the present level of evidence convincing.
I may have misread what was going on in section 3.2 of the paper, but it seemed to be placing the threshold on the size of training or developing news models at 13B parameters, which isn’t even close to near-frontier. (I think someone could run a 13B parameter model on their phone.) And the maximum computing threshold seems like it might be 3 orders of magnitude lower than the estimates I’ve seen for training GPT Astra. Again, I’m pretty much an amateur at all this, so correct me if I’m wrong. But the proposed numbers seem really restrictive and like they go way beyond just pausing the frontier. This seems like it is intended to completely freeze the industry in its current state for a decade.
I think there are obvious reasons why US dominance via increasing value of US products might be more appealing to other countries than US dominance via treaty obligations. Obviously the existential risk worries could overcome this, but I think it’s reasonable to note that if X-risk really is a legitimate worry, the risk is mainly coming from the US. “Hey, the rest of you need to stop building your own AI or we’ll blow everything up” does not sound like a friendly posture to take to the rest of the world.
> the proposed numbers seem really restrictive and like they go way beyond just pausing the frontier. This seems like it is intended to completely freeze the industry in its current state for a decade.
In § 6.2.4, we discuss the question of how far below the frontier to take the compute ceiling, suggesting frontier/1000 as “illustrative,” but saying “a wider range of values could likely support a treaty depending on leaders’ judgments.”
While I recognize there would be commercial and recreational benefits of relaxing this threshold, my personal recommendation would be to keep it low, precisely because we want to push the industry away from trying to compete commercially by training better models (even if smaller than the present frontier) and toward other economically productive but less risky activities, like productizing current models. In short, we want the industry to reduce its efforts toward training efficiency gains, not redouble them. We don’t assume the rate of training efficiency gains would slow, but the agreement would be even more durable if it did.
The current leaders would need to be regulated as state-created monopolies (or oligopolies), the way, e.g., power companies are. How to do so efficiently and fairly would be a project that I’d love to see others work on.
> “Hey, the rest of you need to stop building your own AI or we’ll blow everything up” does not sound like a friendly posture to take to the rest of the world.
Definitely not. Companies’ primary responsibility is not to cause catastrophes through their own actions, and states are also responsible to prevent their own companies from doing so. But a domestic ban in the U.S., even if it proves to be a necessary first step, would not be enough on its own. An international solution will be needed at some point.
Why assume that other countries want to build AI so powerful that they can’t control it? So far that’s really just been the US frontier labs, and it seems like it’s explained by the ideological commitments of those labs. Again, this is why I’m questioning the collective action problem framing of all this. People say that’s what’s going on, but the only evidence of a collective action problem is that the big labs say they think their product might kill everyone but they have to build it anyway. This is so manifestly irrational that everyone assumes there must be some collective action problem leading them to act like this. But what if it’s just that they really do believe the things they tell us they believe?
I guess I’m not understanding why more traditional forms of regulation–e.g., safety standards, mandatory testing, financial liability, at the limit criminal liability for companies put dangerous products on the market–is insufficient. Companies can compete, but with hard penalties if they violate certain rules in that competition.
If by possible we mean metaphysically possible, then we could do a lot more than is laid out in this proposal. If by possible we mean practicable or likely, then I fear not, but only because we lack the collective willpower.
Also, even if 95% percent of folks agreed and had collective willpower, Trump (with his band of tech oligarchs) are cheering ‘build baby build!’, and so far no one has been able to stop Trump and the oligarchs. We’ll see what happens in November.
They will try their best to defeat science, careful reasoning, good sense and loving kindness, every step of the way. For what? killing and braggadocio around it, oil, dominance, cruelty, ridiculous out-of-this world personal wealth. AI can def. help them with all that, esp. the first and the last.
.
Exactly. A just world is possible, if only everyone decides to behave justly. Alas, they won’t.
Let’s assume that this kind of hardwired pause can work.
Can someone help me feel a bit better about our ability to prevent a malicious actor from using an “uncensored” model (in the spirit of PewDiePie’s recent efforts) to provide bespoke, easy-to-follow instructions for clandestinely constructing bombs, spreading pathogens, hacking critical infrastructure, and the like?
I don’t know about hacking but I would have thought building bombs and designing pathogens getting your hands on the materials is by far the hard part rather than the instructions
I’ve long been able to google instructions for repurposing the green fan wire on my home furnace to act as the common wire with a smart thermostat, but I only felt comfortable following through with it when I received bespoke, easy-to-follow instructions from Gemini.
I was imagining something similar in the bad case, including with respect to the hard(er) part of securing materials.
But maybe I’m giving the models too much credit?
Everyone seems to agree the cyber/hacking threat from AI is real and is a cause for serious concern. At the same time, this also means AI can be super helpful for cyber defense, since it can be used to find vulnerabilities in your own system and fix them.
My guess with bomb-making is that it has probably increased people’s access to that information, but so did the internet. I think it’s a question of how much, and how many people use that. I think it hasn’t really been that hard to make bombs for a while. It’s just most people have no interest in making bombs.
Also part of why the AI is getting much better is extensive reinforcement learning. I don’t know why the big labs or Google would be hiring terrorists to help reinforce their models in the quality of advice they give to aspiring terrorists. So I’m guessing they’d mostly help by doing an extensive web search and explaining the information already online.
Not as hard as you might think
https://www.nature.com/articles/s41598-024-53944-x
It certainly serves the interests of the extant leaders in the field to pause development before the competition can catch up, and before they can collapse under the weight of their own financial obligations. This way, they can blame the poor finances and the lack of progress on “the pause” rather than assuming responsibility for themselves, and hopefully win almost a decade of continued existence.
It may or may not also be in our interests, in terms of consumer electronics pricing (i.e. if the fab space gets freed up–I would hope so, but I have no idea), and for the more ephemeral “wrestling with the dangers”. But I can’t help but see this as a very cynical exercise.
Michel, I share your cynicism about the motives of industry leaders who warn about dangers even while continuing to exacerbate them by racing ahead. Unfortunately, the best reasons to believe that the AI race is reckless don’t rest on appeals to the authority of tech executives. I won’t rehash them here, but I will point out that our working group includes no employees of AI companies.
When governments don’t act, they in effect trust the industry to self-regulate. I don’t think that’s been working out.
Yes; “yes, a thousand times, yes.”
Approaching the global threat of AI, as have previously understood nuclear arms escalation threats, is precisely the right kind of model to both acknowledge and address the widespread significance of the problem.
Just as with nuclear disarmament, it will require a limitation of the resources that create the global threat, and mutual agreements across nations to cease development.
A 10 year pause seems a lovely start, though likely not sufficient.
In the age in which nuclear technology came to fruition, political systems, even in democracies, were vastly different than today and there was no social media.
If that happened today, people would demand for the permission to hoard uranium in their backyard or put it in their body or something
I think they would just deny that the bomb is real and would just eat uranium for breakfast. I am sure there would be competitions of the sort on tik tok or something.
What ‘worked’ with nuclear weapons was the threat of mutual destruction (i.e you stockpile as many weapons as possible and then never use them). So I guess, after the AI age had its first Hiroshima, we might have a go at something like a truce (i.e. continuous development of more and more dangerous AI systems that we will never use). Cheerful thoughts. And prospects. We really do need some alien ass to kick asap.
This is the only point at which I wish I was 80, and not middle-aged.
Arms control ain’t what it used to be. The main difference for cyberweapons and AI is that there’s no plausible verification regime.
They’re not like bombs which can be counted and physically inspected, and they don’t require rare materials (like highly enriched uranium) that are tightly controlled. They can exist anywhere there’s a capable enough computer system.
And if you don’t have verification, good luck with enforcement. I don’t even see how trust and confidence-building measures can be promoted, as the next-best thing, given the state of world and AI competitors…
This is precisely the issue that the report being discussed here is addressing.
Oops, thanks for clueing me in, David. I hadn’t read their report since the premise sounded so implausible. After skimming the report now, my assessment and comment still stand.
For one thing, frontier AI models don’t need “AI chips”, but they can work and be trained with ordinary computer chips like GPUs, though more slowly, less efficiently, etc. That’s how Nvidia got into the AI game, via its graphics processors.
Then what–lock down the supply of regular computing chips? That’s a non-starter.
To be sure, the report does recognize that ordinary chips can power and train AI, but it seems to suggest that they would be so energy-intensive and impractical that it wouldn’t happen. But impractical is not the same as impossible or even difficult.
So, that’s a big assumption holding the argument together, on my quick read. And lots of other things in the report are implausible, even if they’re fun to think about…
Patrick, thanks for taking a look at the report.
GPUs are an example of what we call “AI chips”. We distinguish between datacenter-grade GPUs like H100s, H200s, etc. and consumer-grade GPUs like GeForce RTX series cards.
Concerning consumer GPUs, we do not make the assumption that “they would be so energy-intensive and impractical that it [frontier training with consumer GPUs] wouldn’t happen.” Instead, we take a quantitative approach to estimating the extent to which consumer GPUs could contribute to the training of a beyond-frontier model.
The quantitative analysis is in Section 6.4.5 – Consumer GPUs. The conclusion is that “the consumer GPU base does not appear to represent a significant hurdle to the treaty’s durability.” Of course, there are assumptions of this analysis, which we state. Feedback on Section 6.4.5 would be welcome.
Thanks for the clarification, Wes. Ok, I’ve now read sec. 6.4.5. Just so I understand:
Are you saying that “they would be so energy-intensive and impractical that it [frontier training with consumer GPUs] wouldn’t happen” isn’t an assumption because it’s a conclusion? Is that what you’re objecting to?
It sounds like the report is saying the energy demands and (either covert or overt) coordination needed to secure the chips explain why consumer GPUs aren’t much of a threat, and your conclusion is supported by assumptions and math.
Or if you’re saying there’s another reason why you think consumer GPUs aren’t much of a threat, that’s not clear to me yet.
Either way, I wouldn’t argue down at the level of quantitative methodology, since it’s not my area of expertise. But I might say this is ultimately an empirical question, and a priori accounting can’t settle it, esp. if you’re helping yourself to big assumptions.
Good luck with your project–if you can help to bring about a pause of any kind, that might be better than the runaway trainwreck we’re seeing now…
You can verify the destruction of GPUs. I’m not saying that’s the best option, but clearly it is an option.
Indeed, although not because I was thinking of these points, my comment was intended to be satirical (since I don’t expect any kind of pause at all).
The fact that we are having this conversation, no matter its outcome, is a welcome change from the Mr.-Magoo-walking-into-traffic/Jesus-take-the-wheel approach of many.
yes. at last.
For those concerned with regulatory capture, Alex Tabarrok has a great post explaining why this doesn’t fit calls to slow down AI progress:
https://marginalrevolution.com/marginalrevolution/2026/09/what-regulatory-capture-actually-looks-like.html
A concern about any temporary pause is that it creates an artificial bottleneck that, once released, will likely result in an overwhelming tsunami of developments. For example, suppose we stop producing new training-capable GPUs. It’s likely that the designs for the next generation of chips will still be developed. And once the pause is lifted, what comes out of production are GPUs capable of training far, far more powerful AIs than what were created prior to the pause. Moreover, a hardware pause further encourages investment in improving algorithmic efficiency. So, when new GPUs come online, the AI training regimens and models may be even better poised to take full advantage. A worrisome possible scenario is that, once the pause is lifted, the takeoff of new capabilities is so tremendously fast that alignment research cannot keep up and highly capable misaligned AI takes over.
So, while these occasional couple week pauses seem silly, I think they might actually part of a decent approach to pacing the frontier. What matters is giving alignment research some time to catch up before capabilities advance again without building up too much pressure behind a temporary bottleneck.
Im afraid that by now NVIDIA and the whole AI ecosphere is too big to fail. if anyone in gov’t seriously entertained this, then we would have such a massive crash in the economy that the human costs of the downturn would be greater than any realistic AI harm. In Ca. alone, the entire higher ed system is funded right now by a bet on future ai earnings.
If I thought the upper bound of “realistic AI harm” was below “major economic downturn”, I would sleep easier at night.
the upper bound will have lower likelyhood than the lower bound and the lower bound of economic disaster is far higher than the lowerbound of ai harm. the ripple effects of the ai crash will impact everything, all the way to the ukrainian frontline.
The lower bound of both is presumably zero (AI might be harmless; for any economic crash, there is another one that causes less harm). The upper bound of harm from a nuclear exchange has lower likelihood than the lower bound, but again, that doesn’t help me sleep at night.
I know this isn’t the focus of your report, but I’d be curious if Will or Wes have thoughts on the political feasibility of this proposal, compared to other plans to pause or slow down frontier AI? Nividia seems like the big loser here, along with other companies involved in making high-end GPUs.
AI 2040 Plan A, for example, wants to halt algorithmic progress and allow total compute to grow. They call for total research transparency, which has always struck me as an incredible ask that would wipe out much of the value of Open AI and Anthropic. So this plan has parties that lose in the short-term as well.
Just so its clear, I think we should be pretty open to the idea that plans which seem incredibly difficult politically might become politically viable in the near-term future. I think a lot can and will change. But any serious proposals will likely create enemies in the near-term.