A Machine That Is Wrong Half the Time Can Still Pay Off, If You Can Tell When
Averaging the data from a Wharton experiment, an assistant that was wrong half the time produced more right answers than working alone; the signal that says when to trust it was missing, and the only remedy tested left almost six in ten people following the error.
In the Wharton experiment that gave us “cognitive surrender”, an assistant built on ChatGPT was wrong at random half the time. Shaw and Nave, at the Wharton School (the business school of the University of Pennsylvania, in the United States), gave 1,372 people, across three experiments, seven questions built to trip up the first instinct. The classic case: a bat and a ball cost €1.10 together, and the bat costs €1 more than the ball. Instinct says the ball costs ten cents; the arithmetic says five. Participants could consult an assistant that, question by question and at random, gave either the right answer or exactly the wrong one that instinct produces, with the same confidence. Alone, people got about 46% of the questions right. With the right assistant they got 71%, with the wrong one 31%. The study is a preprint, meaning other researchers have not yet checked it.
Cognitive surrender starts where you stop building an answer of your own and adopt the machine’s; with a calculator, your reasoning stays switched on. The authors write that surrender is not irrational by nature: deferring to a statistically superior system can pay. In a city you do not know, a satnav that gets one turn in ten wrong still gets you there more often than your sense of direction would. I do the sum on their experiment myself, because the paper does not report it: if the assistant is right half the time, the average of 71% and 31% is about 51%, five points above the 46% of people working alone. It holds only if right and wrong answers come in equal numbers (and the assistant was opened a little over half the time), but it shows the threshold: a source pays when it is right more often than you are on your own, and here it only had to beat 46%. An assistant that is right half the time is no statistically superior system, though, so the authors’ claim that surrender can be adaptive is out of reach of their data.
What does not pay is not knowing which half is the wrong one. Among those who opened the chat, the right advice was followed 93% of the time and the wrong advice 80%: thirteen points apart, a weak form of discernment. Confidence rose by almost twelve points out of a hundred for people with the assistant, and did not fall as wrong answers piled up. Probably the tone matters, a fluent and always confident text that lowers the bar for checking; the authors propose this as the mechanism but did not test it, because the tone was the same in every trial. Still, confidence kept some value: in one of the three experiments, ten points more confidence went with a slightly higher chance of being right.
The share of people who follow the error overstates real surrender. The assistant’s error was always the instinctive answer, so someone who followed it may simply have recognised their own intuition without giving up a correct calculation. I cannot see the authors discussing this overlap in the main text, and the list of questions sits in an appendix I could not read. The tightest measure is the loss against oneself: 46% alone, 31% with the wrong assistant, fourteen points lower. On average, those are people who would have reached the right answer alone and stopped reaching it with the assistant (the comparison is between different groups, so it is indicative).
The one remedy tested is a package: twenty US cents for each right answer, a ticket in a $20 lottery and a correction after every question. Rejection of wrong advice rose from 20% to 42%, so almost six in ten people kept following it. Reward and correction also improved people working alone, from 42% to 64%, and with the wrong assistant the figure went from 31% to 45%: measured against their own level, the gap is about 12 points without the remedy and about 19 with it (with one trial per person measuring the starting level, the difference is indistinguishable from noise). The remedy raised the level of the person receiving it and left the distance from their own level roughly where it was. The other remedies the authors propose, such as uncertainty signals and explanations in the interface, or digital education, were not tested.
The study is worth what a laboratory is worth: seven questions, one session, one model, an assistant wrong by design. The preprint also has figures that do not add up (samples summing to 1,375 instead of 1,372, two different measures of the same effect), so rounded numbers are the safe ones to quote. What happens with repeated use is a different question and stays outside it, as does the fate of the “don’t know” when an answer is within reach.
Cognitive surrender pays when the person surrendering can see the error coming. In the Wharton data that signal was missing: confidence went up, following the error stayed close to 80%, and the one remedy tested, which acts on the person answering, raises the level without shortening the distance. Probably only whoever builds the assistant can put the signal in place, deciding what an uncertain answer looks like, but nobody has yet measured whether it works. Until someone does, a company or a school that adopts an assistant is accepting an average gain whose cost falls on whoever lands in the wrong half.