DeepInstructor: An Agentic AI Instructor for Experience-Driven Idea Evaluation

Researchers proposed DeepInstructor, an agentic framework that evaluates research ideas by reasoning over an Experience Graph built from 58,607 peer reviews using a ReAct-based agent, according to the arXiv paper 2609.22104v1. The team also introduced DeepInstruct, a dataset of controlled pairwise comparisons across novelty, significance, and feasibility. DeepInstructor improved Hit@1 and Hit@2 alignment with human judgments by 24.4% and 29.7%, respectively, over existing baselines.

arXiv:2609.22104v1 Announce Type: new Abstract: As automated scientific discovery advances, Large Language Models LLMs can now generate research ideas at an unprecedented scale, shifting the bottleneck from idea generation to idea evaluation. Existing evaluators mainly rely on parametric LLM knowledge or unstructured retrieval, producing judgments that lack the experience-grounded reasoning used by human instructors. To address this, we propose DeepInstructor, an agentic framework that formulates idea evaluation as reasoning over structured scholarly experience. DeepInstructor constructs an Experience Graph from 58,607 peer reviews and employs a ReAct-based agent to retrieve dimension-specific evidence for traceable evaluation. We further introduce DeepInstruct, a dataset with controlled pairwise comparisons across novelty, significance, and feasibility. Experiments show that DeepInstructor substantially outperforms existing baselines, improving Hit@1 and Hit@2 alignment with human judgments by 24.4% and 29.7%, respectively. Our findings suggest that scientific idea evaluation can be grounded in explicit reasoning over structured scholarly experience