Under Oath in New York, Three Yes-or-No Questions About AI Went Unanswered

On Monday 5 October the New York City Council, the city's legislature, sat all 51 of its members as one committee and put representatives of Anthropic, OpenAI, Google and Meta under oath. The questions fitted in a word each. Does a model that fails independent validation get blocked from release? Does the company answer in law for the harm if a model slips its leash? What probability does it give to a catastrophe? None of the four answered plainly. Julie Menin, who presides over the Council, put it this way: "A simple yes or no would instill more confidence in the public."

A city council can compel a company to show up, and it cannot compel an answer. Menin issued a subpoena, a court-backed summons, to SpaceXAI, the company formed by the merger of SpaceX and xAI. It did not appear, and the Council has announced legal action. Meta agreed at once. OpenAI and Google agreed only once the same summons was threatened, and Anthropic confirmed a few hours before its deadline. Where the only power is to compel presence, the company keeps the choice of who sits in the chair: the witness is the answer.

Anthropic sent Logan Graham, who runs the team that attacks the company's own models to find dangerous capabilities. Pressed on catastrophic risk, he declined to put a number on it and said his expertise was on the technical side. He made no commitment on kill switches, or on third-party validation blocking a release. Sure, a head of red-teaming is no lawyer, and saying so is honest. Then again, none of the four witnesses carries a legal title in the reports, and Meta's representative also pleaded missing legal expertise. The cheapest explanation is that the four were chosen for that; it remains an inference, since no company has explained the choice.

Three weeks earlier an Anthropic co-founder had asked the question in public. Jack Clark told the BBC that society might eventually want to mandate a kill switch that a third party can verify, and that Anthropic would bring in outside evaluators, METR among them, within weeks. A question is not a commitment, and Clark spoke of rules for some later date. Put a bill and an oath in front of the same company and the idea moved to a desk that was not in the room.

The bill at the centre of the hearing, introduced by Menin, bars deploying an AI model in the city without outside validation, requires a shutdown that a human can operate and independent validators can verify, and sets a penalty of $25,000 per violation. Companion bills add a right to sue over harm from jailbroken systems, a bounty for whistleblowers and an incident-response plan run through the city's cybersecurity command. The text names no validators. BetaNYC, a civic group that supports outside validation, flagged two gaps: open-weight models, whose files anyone can download and run, leave no operator to switch off, and nothing says who commissions a validation. The second gap is the old question about independent third parties, already hanging over a Florida lawsuit against OpenAI, where nobody has yet said independent of whom.

The same Monday, in Oxford, the UN human rights chief Volker Türk said that the time for relying on goodwill is over, and noted that the pledge the companies signed at the White House in late September carries no penalties. One day, three layers: a city that can summon but not make anyone answer, a federal pledge nobody can be fined for breaking, and an international body that can only warn. The authority able to compel is the smallest of the three, and the one with the widest reach holds the weakest tools. In New York the witness is the answer; in Oxford there is no witness at all.

The one kill switch actually used so far was pulled by Washington. On 12 June the Commerce Department ordered Anthropic to disable its two most advanced models, Fable 5 and Mythos 5, for every customer, invoking export controls and national security, on a legal basis a Lawfare analysis called uncertain. The bill that would write the rules for such a switch, the AI Kill Switch Act, was still sitting in a House subcommittee in mid-September, where it had gone on 24 July. Every kill switch still turns out to be a person, and the rule about when that person may press the button is being drafted in a city hall.

Anyone who wants to know whether a failed model gets stopped has to settle first who fails it, who pays when that verdict is wrong and before whom the company answers. Until those three have an owner, the cost of the rule is set by whoever picks the witness.