Researchers at Stanford University have built a system that turns published computational biology papers into executable tool sets an AI assistant can use with new data. In a study published September 16 in Nature, the team reported converting 74 of 100 papers tested; the remaining 26 could not be converted because of missing code, missing data, or software that would not run [3].
How Paper2Agent works #
Paper2Agent takes a paper that comes with associated code, methods, and data. It packages the code into reusable tools, tests those tools against reference outputs from the paper, and writes step-by-step instructions for an AI assistant to follow when a user asks a question or provides new data [3]. This differs from a general-purpose assistant that generates new code on the fly, said Peter Clark, a senior research director at the Allen Institute for Artificial Intelligence who was not involved in the work [3].
Performance on converted papers #
For the 74 papers that were successfully converted, the Paper2Agent tool sets ran analyses of the published work and answered questions about the results more accurately and more than twice as fast as the same AI assistant working from the original papers and accompanying code [3]. The comparison applies only to those converted papers. James Zou, a computer scientist at Stanford and a co-author of the study, described the result as “a virtual author of the paper,” available around the clock [3]. Zou said his team has already created over 100 paper-agents from their own work and that these have helped him find other scientists to collaborate with [3].
Limitations noted by the researchers and outside experts #
The 26% failure rate reflects what the papers themselves provide. Papers that lacked code, lacked data, or included software that would not run could not be turned into tool sets [3]. The system does not generate missing pieces; it packages what is there.
Clark noted that Paper2Agent specializes in helping scientists reuse methods and code, not in answering general questions a “true virtual author” should be able to handle [3]. For open-ended questions—precisely the things authors did not put in the paper—Clark said he would still contact the corresponding author [3].
Clark also raised a longer-term concern: if millions of low-quality papers are turned into agents capable of pestering scientists, the result could make things “worse rather than better,” recalling Microsoft’s Clippy [3]. He said he would welcome a “non-annoying agent” that flags work worth paying attention to.
The Science News report gives only “more than twice as fast” without a precise multiplier or confidence interval. The original Nature paper may contain more detail. Performance on papers outside computational biology or with partial code and data was not described in the Science News coverage, and no independent replication has been reported.
Sources #
1. [Researchers reportedly converted 74 of 100 computational ...](https://digg.com/ai/uzzglf7b) (digg.com)
2. [AI for Science timeline - AI4S.fyi](https://ai4s.fyi/timeline) (ai4s.fyi)
3. [AI could turn scientific papers into research assistants](https://sciencenews.org/article/ai-scientific-papers-research-assistant) (sciencenews.org)