Google’s AI writes scientific papers and takes its grades from its own reviewer

Of 107 research problems drawn from machine-learning conferences, ScientistTwo completes 86. That is the claim of an arXiv paper from Google Cloud AI Research and the University of Waterloo, released on 17 September: given a problem chosen by a person, the system looks up the best known solutions, forms hypotheses, writes the code, runs the experiments, performs ablation studies and revises its draft with a simulated reviewer. The average gain on successful cases over the strongest human solutions is 25.2%, the median 7.7%. The experiments run on Gemini 3.6 Flash and Claude Opus 4.8, with Claude Code as the coding agent, and each problem costs on average about $3,800 and two to three days.

The numbers need reading next to the average. The 21 failed problems do not enter the 25.2%. The references were checked by a language model with live web search, which flagged none of 1,814, and as far as I can tell nobody checked them by hand. A person defines the initial challenge and, as far as I can tell, none of the generated papers has been submitted to a conference: the evaluation is entirely by automated reviewers.

Still, the result is serious. A system that closes 86 of 107 problems with runnable code, and beats the average of papers accepted at NeurIPS 2025 and ICLR 2026 according to two automated reviewers, does work that until recently few would have called automatable, and the authors themselves state that it does not reach Spotlight level. How it is built also shows where the person remains: the job is to set up the problem, the stage where much of the reasoning happens, while the whole cycle downstream (writing, testing, reviewing) passes to the machines. Lisanne Bainbridge described in 1983, for industrial plants, how automation takes the ordinary work away from the operator and leaves the rare, hard cases just when the operator has stopped practising on the ordinary ones. If that holds here, and I only suggest it might, the person who sets the problem loses the chances to understand what happens next, and each turn of the cycle leaves fewer tools for noticing a wrong turn.

The in-house reviewer sets the grade. The main reviewer, ScholarPeer, shares two authors with this paper and is also used to improve the drafts before the final evaluation: the paper itself calls it an “in-distribution” evaluation. With that reviewer ScientistTwo scores 7.0, 7.3 and 7.6 against 6.8, 6.2 and 6.9 for the human papers (ICLR, NeurIPS and ICML Spotlight, in that order). With the external reviewer, the Stanford Agentic Reviewer, the scores are 5.4, 5.6 and 5.7 against 5.2, 5.5 and 6.1: the lead shrinks to 0.2 and 0.1 points, and on the Spotlight papers the system loses. An automated reviewer can be a good tool. What counts is who built it, what it is used for and whether it has the power to say no, and a yardstick that also serves to refine the text stops measuring it independently.

Another gap lies between promise and perimeter. The paper’s title speaks of the frontier of knowledge; the text admits that the system works on local algorithmic improvements, on problems where the state of the art is already known and chosen by people. Then there is responsibility: as far as the paper and the project site show, no person answers for the content of any generated paper, and in this field the producer always answers, together with whoever decides to adopt the system. The authors themselves name the cost of about $3,800 per problem as a limit for university labs.

What would change the picture is ordinary, which makes it realistic. Reviewers from outside the lab, sharing no authors with the system under review and with the power to halt publication, because a check that cannot stop the line does little. A person with a name under every result, who can be summoned to defend it in person. And a perimeter declared in full: the failed problems, the median next to the mean, which reviewer shaped the drafts, and what was done to rule out that the problems were already in the models’ training data, a topic the paper, as far as I have read, does not address. An in-house reviewer, paired with an external one that has no power to stop anything, is not enough.

As long as the reviewer stays in-house, the grade stays in-house.