{"slug": "semantic-scholar-vs-valyu-which-research-api-should-your-agent-call", "title": "Semantic Scholar vs Valyu: Which Research API Should Your Agent Call?", "summary": "A developer compared the Semantic Scholar and Valyu research APIs for building research agents, noting that Semantic Scholar provides metadata and citation graphs across 214 million papers, while Valyu offers full-text retrieval with structured citations across academic, clinical, and regulatory sources. The choice depends on whether the agent needs paper records or passage-level evidence.", "body_md": "If you are building a research agent, the first real decision you make is what comes back from the retrieval call. A paper record, or the paragraph inside the paper that answers the question. Those are different products, and picking the wrong one shows up three weeks later as a RAG pipeline you did not plan to build.\n\n[Valyu](https://www.valyu.ai) is a search and DeepResearch API that retrieves full text with structured citations across academic, clinical, scientific, patents & regulatory sources. [Semantic Scholar](https://www.semanticscholar.org) is an academic search engine and metadata API from the Allen Institute for AI.\n\nPick **Valyu** when the unit of work is a *passage of evidence*: search inside papers at query time, pull a clinical trial protocol and an FDA label alongside the literature, get a cited answer or a full multi-step report without operating your own retrieval stack.\n\nPick **Semantic Scholar** when the unit of work is a *record*: find papers by topic, walk a citation graph, rank by influence, pull author profiles, get recommendations. It is free, the corpus is enormous, and nothing else gives you that citation graph as cleanly.\n\nThe short version in code:\n\n```\n# Valyu returns the text that answers the question, with the citation attached\n{\"title\": \"...\", \"content\": \"In the phase 3 cohort (n=847), median PFS was...\",\n \"citation\": {\"doi\": \"10.1056/...\", \"authors\": [...], \"fragment\": \"#:~:text=median%20PFS\"}}\n\n# Semantic Scholar returns records you then have to go and read\n{\"paperId\": \"649def34...\", \"title\": \"...\", \"abstract\": \"...\", \"citationCount\": 1893}\n```\n\nThe Semantic Scholar Academic Graph covers **214 million papers**, **2.49 billion citations**, and **79 million authors**. Coverage spans every field, assembled from publisher feeds, preprint servers, and web crawling.\n\nThe API is organised as three services:\n\n| Service | What it does |\n|---|---|\n| Academic Graph | Paper search, bulk search, title match, autocomplete, snippet search, paper details, batch lookup, citations, references, author search |\n| Recommendations | Papers similar to one paper, or to a positive/negative example set |\n| Datasets | Bulk corpus downloads, including S2ORC |\n\nFull text is not a query-time product here. It lives in **S2ORC**: 8 million-plus full-text papers, alongside 81 million paper nodes and 73 million abstracts, distributed as a bulk download. There is also a snippet search endpoint over open-access papers, which returns short extracts rather than the retrieval-depth passages a RAG pipeline usually wants.\n\nSo: metadata and abstracts at query time, full text as a corpus you host yourself.\n\nValyu full-text-indexes roughly 4 million open-access papers, plus licensed journal content:\n\n| Source | Full-text coverage |\n|---|---|\n| PubMed | 2.5M+ |\n| arXiv | 1M+ |\n| bioRxiv | 350K+ |\n| medRxiv | 80K+ |\n| ChemRxiv | 8K+ |\n\nPubMed's complete **37 million-record abstract corpus** is available as an opt-in via `include_abstracts`\n\n. More on that flag below, because it is the single most commonly misreported detail about this API.\n\nBeyond the literature, the index covers ClinicalTrials.gov (500K+ trials), FDA drug labels from DailyMed (150K+), SEC filings (3M+), USPTO patents (8M+) and EPO patents (6M+) with full text and figures, plus genomics and chemistry sources. One query can span several of those at once, which is the part that matters if your agent needs to go from a mechanism in a paper to the trial testing it to the label of the approved drug.\n\nBoth do discovery. Here is Semantic Scholar, rewritten from the docs example as something you would actually put in an agent rather than an interactive prompt loop:\n\n``` python\nimport os\nimport requests\n\nS2_BASE = \"https://api.semanticscholar.org\"\nHEADERS = {\"X-API-KEY\": os.environ[\"S2_API_KEY\"]}\n\ndef search_papers(query: str, limit: int = 10):\n    r = requests.get(\n        f\"{S2_BASE}/graph/v1/paper/search\",\n        headers=HEADERS,\n        params={\n            \"query\": query,\n            \"limit\": limit,\n            \"fields\": \"title,abstract,year,citationCount,externalIds,url\",\n        },\n        timeout=30,\n    )\n    r.raise_for_status()\n    return r.json().get(\"data\", [])\n\ndef recommendations(paper_id: str, limit: int = 10):\n    r = requests.get(\n        f\"{S2_BASE}/recommendations/v1/papers/forpaper/{paper_id}\",\n        headers=HEADERS,\n        params={\"fields\": \"title,year,citationCount,url\", \"limit\": limit},\n        timeout=30,\n    )\n    r.raise_for_status()\n    return r.json()[\"recommendedPapers\"]\n\nfor p in search_papers(\"chimeric antigen receptor T cell exhaustion\"):\n    print(f\"{p['citationCount']:>6}  {p['year']}  {p['title']}\")\n```\n\nThe `fields`\n\nparameter is the thing to learn first. Ask for nothing and you get almost nothing back, and every extra field is a join on their side, so keep the list tight.\n\nFor anything above a few thousand results, use bulk search with its continuation token instead of paging the relevance endpoint:\n\n``` python\nimport json\nimport requests\n\nurl = \"https://api.semanticscholar.org/graph/v1/paper/search/bulk\"\nparams = {\"query\": \"(cold -temperature) | flu\", \"fields\": \"title,year\", \"year\": \"2023-\"}\n\nretrieved = 0\nwith open(\"papers.jsonl\", \"a\") as f:\n    while True:\n        r = requests.get(url, params=params, timeout=60).json()\n        for paper in r.get(\"data\", []):\n            print(json.dumps(paper), file=f)\n        retrieved += len(r.get(\"data\", []))\n        if \"token\" not in r:\n            break\n        params[\"token\"] = r[\"token\"]\n\nprint(f\"Retrieved {retrieved} papers\")\n```\n\nNote the query syntax on the bulk endpoint: `|`\n\nis OR, a leading `-`\n\nnegates, and parentheses group. It is not the same syntax as the relevance search endpoint, which trips people up.\n\nValyu does discovery too, over a smaller but full-text index:\n\n``` python\nimport os\nfrom valyu import Valyu\n\n# Reads VALYU_API_KEY from the environment when no key is passed\nvalyu = Valyu(api_key=os.environ[\"VALYU_API_KEY\"])\n\nresponse = valyu.search(\n    \"Phase 3 melanoma immunotherapy trials\",\n    search_type=\"proprietary\",\n    included_sources=[\"valyu/valyu-pubmed\", \"valyu/valyu-clinical-trials\"],\n    max_num_results=15,\n    response_length=\"large\",  # full methodology and results, not just abstracts\n)\n\nfor result in response.results:\n    print(result.title, result.url)\n    # content is str | list | dict; structured sources return objects\n    print(result.content)\n```\n\nThere are TypeScript and Rust SDKs if Python is not your stack, and the REST endpoints are there if you would rather not add a dependency at all.\n\n**Semantic Scholar.** To search inside papers you download S2ORC, chunk it, embed it, store it, and query your own index. That is a real pipeline: object storage, an embedding job, a vector database, and a refresh strategy when the corpus updates. Perfectly reasonable if you want control over chunking and embeddings, and genuinely the right call for some teams. It is just not a thing you get from an API call.\n\n**Valyu.** Full text is the default return value:\n\n```\nresponse = valyu.search(\n    \"mechanisms of acquired resistance to KRAS G12C inhibitors\",\n    search_type=\"proprietary\",\n    included_sources=[\"valyu/valyu-pubmed\"],\n    max_num_results=10,\n    response_length=\"large\",\n)\n\nfor r in response.results:\n    print(r.title)\n    print(r.content[:500])          # relevant full-text chunks, not the abstract\n    print(r.citation.doi, r.citation.fragment)  # fragment deep-links the passage\n```\n\nThe `fragment`\n\nfield is worth calling out separately. It is a text-fragment deep link to the exact cited passage, so a reviewer can click a citation in your agent's output and land on the sentence rather than the paper. If anyone is ever going to audit what your agent claimed, that field is the difference between \"reviewable\" and \"take my word for it\".\n\n`include_abstracts`\n\nGotcha\n`include_abstracts=true`\n\n:Full-text-first is the default. The complete abstract corpus is the opt-in. If you have read the opposite somewhere, that is the error.\n\n```\n# Deep evidence, narrower net (default)\nvalyu.search(query, included_sources=[\"valyu/valyu-pubmed\"])\n\n# Wide net, shallower evidence for the papers that lack full text\nvalyu.search(query, included_sources=[\"valyu/valyu-pubmed\"], include_abstracts=True)\n```\n\nUse the default for evidence synthesis. Flip the flag for coverage sweeps and systematic-review-style screening, where missing a paper is worse than only having its abstract.\n\nThe Semantic Scholar API has no answer-generation endpoint. Ai2 does ship **Ai2 Scholar QA** separately: an open-source cited-synthesis system over 11 million-plus full-text papers and 100 million-plus abstracts, available as a Docker app, an async API, or a Python package. You bring your own Semantic Scholar, Anthropic, and OpenAI keys, and you host it. It is a good piece of software. It is also infrastructure you now operate.\n\nValyu ships synthesis as managed API surface. The Answer API returns a cited answer in one call. DeepResearch runs autonomous multi-step investigation: planning, searching, extraction, fact verification, and report writing.\n\n``` python\nimport os\nfrom valyu import Valyu\n\nvalyu = Valyu(api_key=os.environ[\"VALYU_API_KEY\"])\n\ntask = valyu.deepresearch.create(\n    query=(\n        \"What is the current evidence that GLP-1 receptor agonists reduce \"\n        \"major adverse cardiovascular events in patients without diabetes? \"\n        \"Cover trial design, effect sizes, and where the evidence conflicts.\"\n    ),\n    mode=\"standard\",\n    search={\n        \"search_type\": \"proprietary\",\n        \"included_sources\": [\"academic\"],  # arXiv, PubMed, bioRxiv/medRxiv, ChemRxiv\n        \"start_date\": \"2021-01-01\",\n    },\n    research_strategy=(\n        \"Prioritise randomised controlled trials and systematic reviews over \"\n        \"observational studies. Separate primary endpoints from secondary and \"\n        \"post-hoc analyses. Flag any conflicting or null results explicitly.\"\n    ),\n    report_format=(\n        \"Structured review with: evidence summary table (trial, n, population, \"\n        \"endpoint, effect size, CI), narrative synthesis, conflicting findings, \"\n        \"and evidence gaps.\"\n    ),\n    output_formats=[\"markdown\", \"pdf\"],\n)\n\nresult = valyu.deepresearch.wait(task.deepresearch_id)\n\nif result.status == \"completed\":\n    print(result.output)\n    print(\"cost:\", result.cost)\n    for s in result.sources:\n        print(f\"{s.title} | {s.doi or s.url}{s.fragment or ''}\")\n```\n\n`research_strategy`\n\nand `report_format`\n\nare the two parameters that do the most work. They are where you encode the methodology a domain expert would apply, and they are the difference between a report you can hand to someone and a wall of summarised abstracts.\n\nPoint it at a single dataset when you know exactly where the evidence lives, and ask for a spreadsheet instead of prose:\n\n```\ntask = valyu.deepresearch.create(\n    query=(\n        \"Summarise reported mechanisms of acquired resistance to KRAS G12C \"\n        \"inhibitors in non-small-cell lung cancer, with supporting evidence \"\n        \"for each mechanism.\"\n    ),\n    mode=\"fast\",\n    search={\n        \"search_type\": \"proprietary\",\n        \"included_sources\": [\"valyu/valyu-pubmed\"],\n        \"start_date\": \"2022-01-01\",\n    },\n    deliverables=[\"xlsx\"],           # mechanism-by-evidence table\n    tools={\"code_execution\": True},  # required for xlsx/pptx/docx\n)\n```\n\nDeepResearch also supports webhooks and human-in-the-loop checkpoints, which is what you want when a run takes minutes rather than milliseconds and you do not want to hold a request open.\n\n**Valyu** is usage-based, priced per thousand results by source type, with `max_price`\n\nas a per-query spend cap. Sources priced above the cap are excluded and the response returns a `206`\n\npartial-success warning rather than a surprise on the invoice. Free credits are available to start, $10 without a card and $20 with a work email.\n\n**Semantic Scholar** is free, which is good. The limits are the catch, and they are widely misread:\n\nOne request per second is a hard ceiling on any fan-out design. Batch endpoints and bulk search exist precisely because of it, and you should build around them from day one rather than discovering the limit in production.\n\n| Feature | Semantic Scholar | Valyu |\n|---|---|---|\n| Base URL | `api.semanticscholar.org` |\n`api.valyu.ai` |\n| Auth | Optional key; key gives dedicated quota |\n`x-api-key` , required |\n| Corpus size | 214M papers, 2.49B citations, 79M authors | ~4M full-text papers, 37M PubMed abstracts opt-in, plus non-academic sources |\n| Full text at query time | No (S2ORC bulk download, plus a snippet endpoint) | Yes, default for PubMed and arXiv |\n| Citation graph traversal | Yes, first class | No dedicated graph endpoints |\n| Author profiles | Yes, `/author` endpoints |\nNo dedicated author endpoint |\n| Recommendations | Yes, dedicated service | No |\n| Cited answer generation | Not in the API (Ai2 Scholar QA is separate and self-hosted) | Answer API |\n| Multi-step research agent | No | DeepResearch, plus templated Workflows |\n| Deliverables | JSON | JSON, markdown, PDF, xlsx, docx, pptx, csv |\n| Provenance | IDs, DOIs, citation counts | Title, URL, DOI, venue, authors, plus passage-level `fragment` links |\n| Rate limit | 1 req/s with a key; 1,000 req/s shared unauthenticated | Usage-based, `max_price` per-query cap |\n\n**Semantic Scholar** fits work where the graph is the point:\n\n**Valyu** fits work where the evidence is the point:\n\nIdentical input, different object types, which is the clearest way to see the design split.\n\nA topic search for \"melanoma immunotherapy\" on Semantic Scholar returns paper records: titles, abstracts, citation counts, author links, external IDs. The same query on Valyu returns full-text passages with citations attached to each one.\n\nAsk \"which papers cite this one\" and Semantic Scholar answers directly through `/paper/{id}/citations`\n\n. Valyu has no equivalent, because it is not a graph.\n\nAsk \"what does the evidence say about X\" and Valyu answers with passages, or with a cited answer if you call the Answer API. Semantic Scholar returns candidate records for you to go and read.\n\nThis is a difference in design goal, not in quality. One is built for graph discovery. The other is built for evidence retrieval.\n\nSemantic Scholar is a REST API returning JSON, with community client libraries in most languages. That is the whole integration story, and for many teams it is enough.\n\nValyu ships a [hosted MCP server](https://github.com/valyuAI/valyu-mcp), a CLI, a Claude Code plugin, agent skills, and framework integrations for the Vercel AI SDK, LangChain, LlamaIndex, AWS Bedrock AgentCore, and n8n, plus tool definitions for Anthropic, OpenAI, and Google. If you are wiring retrieval into an existing agent framework, that is usually a config change rather than a client to write.\n\nFor a sense of scale in production: RevisionDojo uses Valyu to deliver academic research to more than 450,000 students, integrating the JavaScript SDK and the Valyu AI SDK across search, citation discovery, and structured literature reviews for IB Extended Essays.\n\nNot at query time. Full text ships as the S2ORC bulk dataset, 8 million-plus full-text papers alongside 81 million paper nodes and 73 million abstracts, which you download and index yourself. There is also a snippet search endpoint over open-access papers that returns short extracts.\n\nNot in the API. Ai2 separately ships Ai2 Scholar QA, an open-source cited-synthesis system over 11 million-plus full-text papers and 100 million-plus abstracts, available as a Docker app, async API, or Python package. It requires your own Semantic Scholar, Anthropic, and OpenAI keys, and you host it.\n\nNo, and this is commonly stated backwards. PubMed search defaults to papers with full text available, returning the abstract plus relevant full-text chunks. Set `include_abstracts=true`\n\nto expand to the complete 37 million-record abstract corpus, where papers without full text return their abstract.\n\nRoughly 4 million full-text open-access papers: PubMed 2.5M, arXiv 1M, bioRxiv 350K, medRxiv 80K, ChemRxiv 8K, plus licensed journal content. PubMed's complete 37 million-record abstract corpus is available via `include_abstracts`\n\n.\n\nSearch returns structured results. Contents extracts clean content from URLs. Answer adds cited answer generation on top of search. DeepResearch runs autonomous multi-step investigation with file deliverables. Workflows are templated, versioned DeepResearch runs for repeatable work.\n\nCoverage currently centres on US trials and FDA-approved drugs, with 24 to 48 hour update delays on trial data.\n\nYes, and it is usually the right answer. Semantic Scholar identifies which works matter through the citation graph, and Valyu retrieves the full-text passages that support specific claims. See the code above.\n\nNo. Preprints from arXiv, bioRxiv, medRxiv, and ChemRxiv are not peer-reviewed, and both services index them alongside peer-reviewed articles. Filter on source or publication type when evidence quality matters.", "url": "https://wpnews.pro/news/semantic-scholar-vs-valyu-which-research-api-should-your-agent-call", "canonical_source": "https://dev.to/valyuai/semantic-scholar-vs-valyu-which-research-api-should-your-agent-call-oc7", "published_at": "2026-08-28 13:39:03+00:00", "updated_at": "2026-08-28 13:50:45.522787+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "artificial-intelligence"], "entities": ["Semantic Scholar", "Valyu", "Allen Institute for AI", "PubMed", "arXiv", "bioRxiv", "medRxiv", "ClinicalTrials.gov"], "alternates": {"html": "https://wpnews.pro/news/semantic-scholar-vs-valyu-which-research-api-should-your-agent-call", "markdown": "https://wpnews.pro/news/semantic-scholar-vs-valyu-which-research-api-should-your-agent-call.md", "text": "https://wpnews.pro/news/semantic-scholar-vs-valyu-which-research-api-should-your-agent-call.txt", "jsonld": "https://wpnews.pro/news/semantic-scholar-vs-valyu-which-research-api-should-your-agent-call.jsonld"}}