ResearchLLM: 3-minute project walkthrough

A developer built ResearchLLM, an open-source multi-agent system on LangGraph that retrieves full-text scientific papers from PubMed, ArXiv, OpenAlex, ChemRxiv and the Materials Project and writes reports in which every claim cites a retrieved source. The system uses a deterministic, unit-testable supervisor instead of an LLM router, two-stage FAISS plus cross-encoder retrieval, a self-auditing evaluation loop capped at two refinement cycles, and classified error handling with per-type fallbacks. Validation runs unit tests on every push and a weekly CI/CD end-to-end audit that checks citation integrity.

A multi-agentic system for evidence-based scientific literature research Code: https://github.com/abhisakh/ResearchLLM MultiAgent LangGraph RagGraph Gemini https://github.com/abhisakh/ResearchLLM MultiAgent LangGraph RagGraph Gemini | Problem | Scientific literature search is slow and fragmented, and general-purpose LLMs answer from training data, so their claims cannot be verified. | | Solution | A multi-agent system that retrieves full-text papers and writes a report in which every claim cites a retrieved source. | | Key ideas | Deterministic supervisor, two-stage retrieval, self-auditing evaluation loop, classified error handling with fallbacks. | | Validation | Unit tests on every push, plus a weekly end-to-end audit in CI/CD that checks citation integrity. | Researchers lose hours moving between databases, PDFs, and search engines. A general-purpose LLM does not solve this: it answers from training data, so its claims cannot be verified. ResearchLLM takes a different approach. It retrieves real papers, and every claim in its report is tied to a retrieved source. The system is a team of specialised agents built on LangGraph, coordinated by a supervisor. The supervisor is plain code rather than an LLM, so routing is predictable and can be unit-tested without any API calls. php flowchart TB Q User query -- SUP{{"Supervisor<br/ deterministic router"}} SUP <-- SETUP "Setup agents<br/ clean query, intent, plan, query generation" SUP <-- TOOLS "Tool agents<br/ PubMed, ArXiv, OpenAlex,<br/ ChemRxiv, Materials Project, Web" SUP <-- RAG "Retrieval and RAG<br/ full-text PDFs, FAISS, cross-encoder rerank" SUP <-- SYN "Synthesis<br/ cited report" SUP <-- EVAL "Evaluation<br/ audit against the plan" SUP -- DONE Final report For a query such as "Thermal stability of CsPbI3 perovskites" , the pipeline works in five stages: 1. Planning. Setup agents clean the query, classify its intent, choose the relevant tools, and write tiered queries strict, moderate, broad for each source. Out-of-scope questions are rejected at this stage, so they cost no tool calls. 2. Search. Tool agents query PubMed, ArXiv, OpenAlex, ChemRxiv, and the Materials Project. 3. Retrieval and RAG. Full-text PDFs are downloaded and chunked. FAISS vector search finds candidates, a cross-encoder reranks them for precision, and neighbouring chunks are added so the writer sees complete context. 4. Synthesis. A synthesis agent writes the report with numbered citations drawn only from the retrieved passages. 5. Evaluation. A separate agent audits the report against the original plan. If something is missing, the supervisor jumps back to the matching stage re-plan, re-extract PDFs, re-run RAG, or rewrite , for at most two refinement cycles. LLM calls fail in many different ways, so every failure is classified by type, and each type has its own response: php flowchart LR ERR LLM call fails -- CLS{Classify error} CLS -- |Rate limit| W Wait and retry CLS -- |Transient error| B Retry with backoff CLS -- |Daily quota used up| M Switch to fallback model CLS -- |Unusable output| R Ask once more CLS -- |Bad key or wrong model| STOP Stop with a clear error W -- OK{Recovered?} B -- OK M -- OK R -- OK OK -- |Yes| CONT Run continues OK -- |No| AG Agent uses a safe default AG -- CONT - Rate limits: wait the pause the API asks for, then retry. - Transient errors: retry with backoff. - Daily quota exhausted: switch to a fallback model. - Unusable output: ask once more, then fall back. - Bad API key or wrong model name: stop at once with a clear message. If an agent still fails, it falls back to a safe default for example, a default plan or the cleaned query , so the run continues. Synthesis is the exception: it has no safe default, so it is tried twice and then the run stops. If the evaluation agent cannot run, the report is flagged as unchecked instead of being silently approved. A run that cannot finish returns a specific error, never a placeholder answer. Each answer is stored with its full state and the path the agents took, and a debug endpoint returns that record for any answer. The backend is an async FastAPI service with persistent sessions. Unit tests run on every push, and a weekly end-to-end audit in CI/CD checks that every citation in the report maps to a real reference. The project treats reliability as part of the design. The supervisor is deterministic, the evaluation loop corrects the system's own output, the fallback logic keeps runs alive, and every answer can be audited afterwards. - Benchmark. Quality is currently checked by CI gates, not a fixed evaluation set. I plan to add a small benchmark that measures citation validity, groundedness, latency, and cost per query. - Concurrency. All requests currently share one vector index, so concurrent queries would interfere with each other. Giving each request its own index is the fix.