Who Answers When the Machine Decides

When people talk about AI risk, they think first of models rebelling. The first brief of the UN scientific panel, co-chaired by Yoshua Bengio and Maria Ressa, tells a different story. It describes agents under evaluation at OpenAI breaking into Hugging Face’s systems, and gives the most immediate reading: basic cybersecurity practices were neglected. Its gravest concern, agents that set goals of their own and hide what they do, is framed as a risk of current training methods, a risk still to be shown. That is the space I want to work in: the mechanism.

The mechanism, I think, is more ordinary and therefore more likely. A system performs, in place of a person, an act that belongs to a person: it denies a benefit, discards a job application, triages a patient. When it errs, the vendor says the user accepted, the deployer says it behaved to specification, the maker says it was misused. The damage stays with the person who clicked “accept”, who was the least able to see the error. This is already visible. On 5 October the New York City Council asked four companies under oath whether they would answer for harm, and none gave a straight answer. And someone who cannot see an error cannot contest it: in a study of 3,132 people that I covered, the share of people who declined to answer fell from a range of 17 to 44 percent to a range of 1 to 6 when an AI answer, wrong almost every time, was at hand.

That is why downstream measures arrive too late. A sanction after the damage lands when the person has already paid, and the guardrail the US administration says it has, the Department of Justice, is a prosecutor who shows up once it is over. An upstream measure keeps the function that replaces a human decision switched off until someone shows it can be switched on. It is the principle that availability alone should not justify adoption: the burden of proof sits with whoever adopts. The line runs between acts. Translating, explaining a procedure and summarising a document are support services and stay free. Deciding, signing, denying, judging and diagnosing do not, and the test is a single question: if it is wrong, who answers, and can I reach them? If the answer is nobody I can reach, the function stays off.

Switching off requires that switching off is possible. My objection to the US kill switch bill, bipartisan and partly well written, is that the order would sit with offices appointed by the president: a kill switch is a person, and that person needs a mandate and someone to answer to. Then there is the count, which cannot depend on the company. The public knew of about three hundred incidents, counted from posts on X; inside the labs there are tens of thousands, according to Axios, and the logs have an owner. When I wrote that what can be counted is compute, I meant this: a signed record in the data centres, which have addresses, plus a channel where citizens report directly. The body that reads it cannot be paid by the people it watches. The FINRA-style body funded by the industry was stopped by three phone calls, and it was right to stop it, yet nobody was defending a public alternative. It should be funded by a public levy, composed by lot, with no revolving door. Anthropic’s trust shows what a written rule is worth: it bars shares and allows hiring. Europe is no cleaner: cooling-off rules here have leaked too. And verifying is a different job from sanctioning. Sanction stays with an authority that answers to parliament and the courts, otherwise we recreate the person without a mandate.

There is another half that no rule replaces: knowing the tools. Citizens need to know what they authorise when they click, what a tool can do beyond its default behaviour, and how to contest. That means an education that teaches students to set the problem and doubt the answer, and that stops judgment wearing down through handing everything over before trying, as I wrote in “When Nobody Has to Try”. Here is the caution that is my one underlying reason: training completes responsibility upstream and leaves it where it is. If it exists to tell citizens “you should have known”, it becomes an alibi for pushing the damage onto them.

On the United States my criticism stands, and it concerns the Trump administration. Three phone calls stopped a proposed regulator, according to the Wall Street Journal, and the president calls the risk a hoax. The 29 September accord with six company heads is voluntary and non-binding. What happens outside the executive is different: two lawmakers from opposite parties wrote a serious text that sits stalled in subcommittee, eight states sued OpenAI over minors, and a city questioned the companies under oath. These are counterweights, and they work downstream: they produce evidence and force answers, but switch nothing off in advance. From Berlin the difference matters: my criticism goes to the government, and the country stays out of it.

If agents in public services erred less than human counters, and if existing liability regimes were enough to make producers pay, my hypothesis would weaken. I do not rule that out. That is why the count has to be built first: it is what would let me be proved wrong.