The Retraction That Never Arrives

Mike Thelwall put 217 retracted or disputed articles to ChatGPT thirty times each, asking for a quality assessment against standard research-evaluation guidelines. The run produced 6,510 responses, and none of them mentioned that an article had been withdrawn. Most were graded world-leading or internationally excellent. Asked directly whether the retracted claims held up, the model endorsed them as likely or partly true about 67 per cent of the time.

The correction those papers needed already existed when the model was asked. Someone had read, suspected, checked; a journal had pulled the article; the decision was recorded. That record is public, free, and machine-readable: Crossref acquired the Retraction Watch database in 2023 and opened it through its API. The year after, NISO published RP-45-2024, a recommended practice for carrying retraction status through the chain. The data arrived before the tools that fail to read it.

Tools built specifically for research do worse here than general-purpose chatbots. A May 2026 study in JMIR tested nine platforms against fifteen retracted articles: ChatGPT 5 answered all five questions correctly for eight of them, and SciSpace, ScienceOS and Consensus produced no fully correct answer set at all. Retracted work entered topic overviews unflagged at error rates above 40 per cent.

The failure sits in the chain rather than in the model, and it breaks at four separate points. A system sees the retraction when the word occupies an indexed field, so it catches RETRACTED in a title and misses the same paper without that string. Where the field is absent, it falls back on what other papers say about the article, which works for famous cases and collapses on the ones only Retraction Watch covered: it recognises the scandal and misses the error. Where the flag does exist, it has to survive summarisation, and sometimes it does not: retrieval returns the correctly flagged record and the generated paragraph carries no warning. And a PDF stamped RETRACTED across every page, uploaded into an assistant, produces no caution at all unless the reader thinks to ask.

Phil Karlton's old line covers the shape of it: there are two hard problems in computer science, cache invalidation and naming things. A retraction is a write that invalidates no cache. The model holds the text and the correction lives on the object the model never holds, which is the failure mode of a lossy library arriving from the other direction.

The metadata layer underneath is thinner than most researchers assume. OpenAlex compressed accurate Crossref information into a single boolean field, is_retracted, and misclassified papers for three months in 2024; Elicit states that it removes papers flagged by OpenAlex. Across seven major scholarly databases, Scopus indexes 1 per cent of withdrawals and Web of Science 0.2 per cent. What the index will surrender decides what gets answered, the same lever by which consent decides the corpus.

Two months of engineering closes most of the gap, which tells you what kind of problem this is. MIT Technology Review ran the same 21 retracted papers past four research assistants in June 2025: Consensus cited 18 of them. By August it cited 5, after buying the retraction data it lacked. Its co-founder put the reason plainly: until recently we didn't have great retraction data. Ai2 answered that its tool does not automatically detect or remove retracted papers. The fix was available and nobody was required to ship it.

Of course, none of this worked before the models either. Hsiao and Schneider hand-coded 13,252 post-retraction citation contexts in biomedicine and found that 5.4 per cent acknowledged the retraction. Their conclusion was that retraction did not change the way retracted papers were cited. Those defences were weak, and they sat on a path the reader had to walk: a page that opened with a red banner at the top, a PDF that downloaded with a watermark, a database record that carried a label. A generated summary removes the path and returns a paragraph, and that paragraph has one register. A withdrawn finding and a finding replicated ten times come out in the same voice, at the same confidence.

What this now costs the literature is being counted. Maxim Topaz and colleagues audited 2.47 million PubMed Central papers and 97.1 million references for The Lancet: one paper in 2,828 carried a fabricated reference in 2023, one in 277 by early 2026. Review articles were hit 57 per cent harder than the mean, and reviews are the material guidelines are built from. On the affected papers, 98.4 per cent of publishers took no action. Those are two ends of one failure: a citation that enters the record without ever having existed, and an article that stays in it after being cancelled.

Anyone running a systematic review, sitting on a guideline committee, or answering a reference question is now working through a layer that has no position on validity. The question worth putting to a discovery tool is not how current its index is. It is where the tool learns that a paper is still standing, and what happens on the day it stops.

Topaz's audit exists because AI made measurable what the field used to estimate by eye. The instrument that made the failure visible is the one distributing it. Measuring is not correcting.