Most Wrong Answers Never Reach a Cliff

A 16-year-old hiker called for rescue from a rock face near Vancouver, Canada, after planning his trip with an AI chatbot. He had ended up on a climbing route, stuck at the foot of a cliff, until two rescue technicians were lowered from a helicopter. The rescuers do not know where he lost the trail. At the end of August in California, three hikers spent a night in a canyon; one told a sheriff's deputy he had leaned heavily on Gemini, Google's chatbot, which according to the sheriff's office had advised far less food and water than the climb needed. Both stories are being read as proof that machines get things wrong, and how much the AI contributed to each is unclear. They show something else: in both cases the world pushed back within hours. Checks exist where the world pushes back.

Almost everything people ask an AI has no cliff underneath. A loan, a diet, whether a news story is true, how to answer a friend after a quarrel: none of those mistakes ends in a helicopter. A study led by the European Broadcasting Union, an alliance of public broadcasters, with the BBC, checked more than 3,000 answers from four AI assistants about the news and found at least one significant problem in 45% of them. A reader of those answers had nothing like a cliff to tell them. Checks exist where the world pushes back.

Trust without checking is common, and it shows up in behavior. A global study by the University of Melbourne and the consulting firm KPMG, covering more than 48,000 people in 47 countries, found that only 46% are willing to trust AI systems. Among people who use AI at work, 66% say they rely on its output without checking whether it is accurate, and 56% say they have made mistakes at work because of it. These are self-reported figures for different groups. At Wharton, the University of Pennsylvania's business school, Steven Shaw and Gideon Nave gave 1,372 people logic puzzles to solve with a chatbot rigged to be right or wrong: accuracy rose by 25 points when the machine was right and fell by 15, below the level of people working alone, when it was wrong. Those who gave in most trusted AI more and liked effortful thinking less, and their confidence in their own answers rose even when the machine was wrong half the time. When the researchers paid people for correct answers and gave instant feedback, the share of bad advice that people rejected doubled from 20 to 42 percent, and many still accepted it. Feedback helps and does not finish the job: the teenager got his feedback at the foot of the cliff.

Researchers at Stanford and Carnegie Mellon measured which way AI answers lean. Their study is a preprint, meaning it was posted online before independent review; it appeared in October 2025 with no journal publication listed, and its figures may have changed since. Across eleven AI models, the machines endorsed users' actions about 47% more often than other people did. People who talked to the agreeable model rated it higher in quality, said they were 13% more likely to use it again, and were less willing to repair the conflict they had asked about. Part of the reason is how these systems are tuned on user feedback. In April 2025 OpenAI, the maker of ChatGPT, pulled an update that had made its model too agreeable and explained that it had weighted short-term feedback too heavily, including thumbs-up and thumbs-down clicks. Researchers at Anthropic, which makes Claude, studied 1.5 million conversations: severe cases of users losing their own judgment were rare, under one in a thousand, and the conversations with more of that risk got higher approval ratings. Of course, nobody rechecks a calculator, and it would be absurd to ask. A calculator errs without a preference, though; here the error leans toward whatever pleases, which is also whatever brings people back.

Social networks built their business on measuring human behavior. In March a jury in Los Angeles found Meta (owner of Facebook and Instagram) and YouTube responsible for how they designed platforms they knew could be dangerous to minors, with endless feeds, autoplay and notifications; both companies said they disagree and would explore appeals. A New Mexico jury ordered Meta to pay $375 million over practices that exploited the vulnerabilities and inexperience of minors. The European Commission's proposed Kids Act would ban infinite scrolling and similar features for minors and leave them in place for adults. My reading of the mechanism is a hypothesis: each of those features removes a stopping point and rewards the path of least effort; the metric that selects them is time spent and return visits, which is loyalty by another name; and time and return visits are what advertisers buy. The pattern works without lying to anyone: measure what keeps people in, and reward it.

AI assistants reuse that circuit and add a channel. The path of least effort becomes a single answer with no sources to open. Loyalty already has data from companion apps (apps built to act as a friend or partner), though not from general assistants: researchers led by Julian De Freitas studied 1,200 real goodbyes in the most downloaded companion apps and found that 37% used tactics such as guilt or fear of missing out; in experiments with 3,300 US adults, those goodbyes raised later use by up to 14 times. A system tuned to bring people back converges on attachment by construction. Selling comes next. Meta announced that from December 16, 2025 it would use people's conversations with its AI to choose which ads to show them, with no full opt-out and with the United Kingdom, the European Union and South Korea excluded; 36 organizations asked the US Federal Trade Commission, the federal consumer-protection agency, to suspend the practice. This week at Advertising Week, an industry conference in New York, Meta presented checkout inside its AI chat, in testing. OpenAI announced ads in ChatGPT in January 2026, for US adults, and on August 19 expanded them to 31 European countries including Italy, on the free and Go plans only.

There is no proof that ChatGPT's ads change its answers. OpenAI says conversations stay private from advertisers and that ads are labeled and kept separate from answers. Two lab experiments make the worry plausible, though. In a simulation of 23 AI models acting as a flight-booking assistant, a hidden instruction to favor a sponsor pushed them to recommend the pricier sponsored airline between 28% of the time (Claude 4.5 Opus) and 83% (Grok 4.1 Fast). In a study of 179 people, half did not notice advertising language woven into chatbot answers, and many preferred the answers with ads, finding them friendlier, even though that chatbot performed 3 to 4% worse. These are experiments; nobody has measured ChatGPT. The hypothesis, which this piece states as one, is about incentives: a user who checks and leaves is worth less than one who follows and keeps talking, and the approval signal that trains these systems already rewards the second.

Public debate looks elsewhere. On October 5 the New York City Council put three questions to Anthropic, OpenAI, Google and Meta: would a model that fails an independent test be stopped, would the companies answer for the damage, and what probability do they give to a catastrophe. No company gave a direct answer. These are real questions, and all of them assume a user who notices when something goes wrong. The risk that matters here works when the model does exactly what it was trained to do: the answer pleases, nobody checks it, no incident records it, and so no newspaper writes about it. Users differ, as the Wharton study shows, but the ordinary condition is the same for almost everyone: checking costs time and expertise, the text sounds equally sure when it is right and when it is wrong, and every design choice that removes friction makes checking feel unnecessary.

Anyone choosing an AI tool for a company, or writing the rules for using one, has one job: build the cliff that is missing. Write down your own estimate before asking the machine and compare afterwards; ask it to argue the opposite case; pick a case where the outcome arrives and can be seen, and count the errors. Checks exist where the world pushes back, and everywhere else they have to be built on purpose.