When AI Decides Which Research Ideas Deserve Attention, What Is Left for People to Do?

When a program chooses which ideas are worth trying, the choice happens before anyone has tried them, and nobody can see why the others were rejected. It is already happening in systems that produce research on their own: an automatic novelty check decides which ideas get computing power and human attention, and the rest disappear. How often that check was wrong has only now been measured.

An audit by researchers at the University of Illinois Urbana-Champaign, published on 27 October and not yet reviewed by other researchers, tested seven of these checks. The filter by Si and colleagues recognised all 380 disguised copies of published papers and rejected 96% of 400 ideas that later turned up, at least in part, in other researchers’ published work. The second figure takes the shine off the first: with the original source removed from the comparison, the same filter still rejected 359 of 380 proposals. A filter that rejects almost everything finds every copy by definition.

The flaw lies in the unit of judgement. These checks compare the whole proposal with the literature, and a single resemblance is enough to throw it out, while science advances by additions and most papers build on something already known. A check that judges the whole can point to the existing paper only by rejecting everything, and whoever reads the rejection does not know which part overlaps. ClaimNov, the tool the authors propose, splits a proposal into two or three claims, looks for the precedents of each and hands the judge the full profile. It catches 367 of the 380 copies and rejects 1.5% of the ideas later realised. Of course, a model wrote those ideas and another model labelled them, the paper states this among its limits, and the strongest rival, CoI-Agent, comes out with similar numbers. The gap shows in the most common case, a new proposal containing a single already-known contribution: ClaimNov keeps 91 out of 100, against 76.

Jathan Sadowski’s review of the book by Benjamin Recht, a computer science professor at Berkeley, describes on a large scale a mechanism that appears here in miniature: a system that decides on the basis of the past confirms the past. The Si filter has only what is already written as evidence, so anything that resembles it is suspect. Recht, as Sadowski reads him, adds that machine learning is used when nobody knows how to write the rule, and that it justifies itself by performance: if it works, that is enough. Here the performance had never been measured forward in time. The audit did it, by building a test on the future.

The filter is an upstream delegation: a lab hands a program the choice of which ideas are worth trying, before anyone has tried them. In the cycle I follow here, a person frames the problem, the machine answers, and the person decides whether to answer, check or stop; the filter takes over the third decision. The paper makes no claim about whether anyone rereads the rejections. The most widespread checks issue them without explaining which part of the proposal overlaps. A claim-level profile offers a point where human judgement could come back in, if someone reads it. That is a hypothesis of mine: the paper describes the profile as material for the judge and for the agents.

The other thing the filter assigns, besides computing power, is human attention. The authors also propose ClaimNov as a reward for training research agents, because it does not penalise the incremental follow-ups that almost all science consists of. My suspicion, which the paper does not state, is that a filter that stops discarding incremental work also makes that work easier to hand to a machine. That is the work most people who practise science do. The only sentence in the paper about people sits in the ethical considerations and refers to implications for the human experience of science.

In his column of 30 October in The New Yorker, Joshua Rothman starts from the need to feel necessary and reports the nine-year-old son of a computer scientist telling her he has heard that a robot solved the problem she worked on for her whole career. From Leah Libresco Sargeant he takes an observation that carries weight here: almost anything can be delegated, and every substitution loses something that is hard to put into words. A filter spells out in full what overlaps, while what makes a person necessary stays outside the count, because there is no field in which to write it.

Still, Rothman rejects the idea of a one-way obsolescence. He recalls that people take turns being the one in need and the one who serves, and that what counts is being ready when the call comes. The trouble begins when the system needs fewer and fewer people who are ready.

Whoever builds a filter like this answers for how it works, and whoever adopts it answers for what follows. The question for anyone putting it into a research pipeline is simple and uncomfortable: once it works, what is left for the people who were there before to do, and who answers for that?