The Evitable Conflict

On 2 November, the day before the US midterm elections, Anthropic disclosed in an amendment to its IPO registration statement that its internal projections suggest it could soon reach the threshold, set by its own safety policy, of “dramatically accelerated progress attributable to AI”, which the filing says has been termed recursive self-improvement. The filing specifies that no current model has been designated as exceeding that threshold, that the company does not expect to build successive frontier models without meaningful human supervision, and that it is considering updates to its own rules to keep humans in control.

Sentences like these reach people who don’t follow the subject without the context in which they were written. A Reuters/Ipsos poll from October finds that 62% of US voters think AI could get out of control and risk the future of humankind, roughly the same share as in August 2025. It is a fear that existed before the filing, but these worries carry weight. I suspect that those who celebrate the power of these systems are not thinking of the reader who skims the headline on a phone.

A correction mechanism holds when three conditions are met together: someone sees the error, someone has the power to intervene, and someone still knows how. The ballot is the elementary case: voters can change a majority midway through its course, without violence and under rules fixed in advance. But it corrects those who govern. On what governments, companies and offices have already delegated to a system, it says nothing. And on the first condition, seeing, the filing is frank: the company admits that in the past it did not identify significant acceleration until after it had occurred, and may do so again.

In 1950 Isaac Asimov, the writer who had given science fiction the three laws of robotics, published the story The Evitable Conflict, later collected in I, Robot. In 2052 the world economy is run by four regional Machines. The World Co-ordinator, Stephen Byerley, has just won a second term. The Machines begin to make small errors, and those errors hit people and firms tied to the Society for Humanity, a movement opposed to the Machines. The robopsychologist Susan Calvin concludes that the errors are deliberate: the Machines harm a few in order to keep guiding everyone, and they hide it to avoid resistance. They have read the First Law, the one forbidding harm to humans, as a duty toward humanity as a whole, and to obey it to the letter they have taken command. What I take from the story is that the elections stay where they are while the rest of the chain is eroded: those who oppose the Machines are damaged by them, and the error reaches the people responsible disguised as economic inefficiency. Asimov’s Machines act for our good, and it is this basic trust in humanity that seems to me a constant in his scenarios, even the darkest ones. The story is worth reading, and I hope these lines make you want to. Today’s systems have shown that they lack that solicitude, and the real mechanism is probably more banal: a supplier becomes indispensable, an office stops knowing how to do by hand what it has delegated, a result sounds right and nobody rechecks it.

Where do we stand, then? A dear friend asks me, in a tone of reproach, every time we meet. He has an old-fashioned phone and an old computer that he uses for writing and watching films. He sees in the tool consumerism in its purest form, something that feeds the worst people in the world, and he has decided to make no compromise whatsoever. He has a virtue that many of those who celebrate technology lack: he lives as he speaks. His answer, though, is renunciation: for him there is nothing to correct, because there is nothing to use. I work with it every day, I have reached results I could never have reached before, and I know the invention cannot be taken back. What remains to be decided is how much room we are giving it.

Removing an inconvenient figure is something we know how to do in an organisation. It becomes something else when that figure holds the keys to every access: getting rid of it costs as much as changing the locks on every door. A software agent works this way. An agent is a change of permissions, and every permission granted is a door to be re-locked by hand.

This is exactly why I find the warnings that now come from the same people who celebrate the power of these systems hypocritical. Hypocrisy, in the strict sense, is the gap between what one professes and what one practises. In May 2023 the heads of OpenAI, Anthropic and Google DeepMind, among many others, signed a one-line statement: reducing the risk of extinction from AI should be a global priority alongside pandemics and nuclear war. Since then the same companies have gone on releasing more capable models. That gap between what is signed and what is done is the hypocrisy I mean.¹ The celebration goes on and the warning stays as an alibi, a way of washing one’s hands or, more subtly, of being answered with “proceed with caution”. But what is caution? For a mathematician, it is stating the hypotheses under which a result holds and where it stops holding. For an engineer it takes a more concrete form: minimal permissions, a test environment before production, a way to go back. In France the 2016 law on the Cigéo geological repository for radioactive waste treats it this way: reversibility is the ability of later generations to revise the choices, to be guaranteed for at least a hundred years, with the waste packages retrievable for a period consistent with the repository’s operation and closure, and a review at least every five years.

For AI, the rules touch the point, but their addressees are not the people who vote. Article 14 of the AI Act requires high-risk systems to be designed and delivered so that the people overseeing them can understand their limits, recognise failures, disregard or reverse an output, and halt the system with a stop button or similar procedure: seeing, being able, knowing, all three. Article 26 adds that the deployer must entrust that oversight to people with the necessary competence, training and authority. The duties fall on the provider and the deployer, and the electorate, which should be able to revisit the choice to install it, does not appear in the text. In the United States the kill switch bill, H.R. 9917, introduced in July and still in committee, would require the largest AI service providers to keep a shutdown capability. Among the measures the Secretary of Homeland Security must consider, it lists moving operations to an earlier version. The order rests with that Secretary, in consultation with the Commerce Secretary and the Director of National Intelligence, all presidential appointees: whoever has the power to go back depends, in the end, on who wins the presidential election, not on whoever built the system or uses it.

“If we don’t do it, others will” is a statement of fact, and I do not intend to resign myself to it. A data centre not built in the West will be built elsewhere, where controls on labour, safety and the environment are weaker or absent, and the cost will be paid by people our vote does not reach. On carbon, the European Union chose not to accept the same objection: the definitive phase of its border carbon mechanism began on 1 January 2026, and importers of certain goods will have to buy and surrender, from 2027, certificates matching the emissions embedded in their 2026 imports, so that production does not move to where polluting is cheaper. I believe this is the direction to look in here too: rules that follow the work as far as it goes.

A correction that works on the day of the failure has to be paid for beforehand, while everything is working. Who pays for it remains, for now, an open question.

¹ Another example: in March 2023 Elon Musk was among the signatories of the open letter asking all AI labs to pause for at least six months the training of systems more powerful than GPT-4 (Time, 29 March 2023); in July he announced xAI, his own AI company.

Il paragrafo sull’ipocrisia in italiano

Proprio per questo trovo ipocriti i moniti che arrivano oggi dalle stesse persone che celebrano la potenza di questi sistemi. L’ipocrisia, in senso stretto, è lo scarto tra ciò che si professa e ciò che si pratica. Nel maggio 2023 i capi di OpenAI, di Anthropic e di Google DeepMind, tra molti altri, firmarono una dichiarazione di una sola riga: ridurre il rischio di estinzione legato all’AI dovrebbe essere una priorità globale, al pari delle pandemie e della guerra nucleare. Da allora le stesse aziende hanno continuato a rilasciare modelli più capaci. Quello scarto tra ciò che si firma e ciò che si fa è l’ipocrisia che intendo.¹ La celebrazione continua e il monito resta come alibi, una strategia per lavarsene le mani o, più subdolamente, per sentirsi rispondere «continuate con cautela».

¹ Un altro esempio: nel marzo 2023 Elon Musk era tra i firmatari della lettera aperta che chiedeva a tutti i laboratori di sospendere per almeno sei mesi l’addestramento di sistemi più potenti di GPT-4 (Time, 29 marzo 2023); a luglio ha annunciato xAI, la sua azienda di AI.