If you are building a research agent, the first real decision you make is what comes back from the retrieval call. A paper record, or the paragraph inside the paper that answers the question. Those are different products, and picking the wrong one shows up three weeks later as a RAG pipeline you did not plan to build.
Valyu is a search and DeepResearch API that retrieves full text with structured citations across academic, clinical, scientific, patents & regulatory sources. Semantic Scholar is an academic search engine and metadata API from the Allen Institute for AI.
Pick Valyu when the unit of work is a passage of evidence: search inside papers at query time, pull a clinical trial protocol and an FDA label alongside the literature, get a cited answer or a full multi-step report without operating your own retrieval stack.
Pick Semantic Scholar when the unit of work is a record: find papers by topic, walk a citation graph, rank by influence, pull author profiles, get recommendations. It is free, the corpus is enormous, and nothing else gives you that citation graph as cleanly.
The short version in code:
{"title": "...", "content": "In the phase 3 cohort (n=847), median PFS was...",
"citation": {"doi": "10.1056/...", "authors": [...], "fragment": "#:~:text=median%20PFS"}}
{"paperId": "649def34...", "title": "...", "abstract": "...", "citationCount": 1893}
The Semantic Scholar Academic Graph covers 214 million papers, 2.49 billion citations, and 79 million authors. Coverage spans every field, assembled from publisher feeds, preprint servers, and web crawling.
The API is organised as three services:
| Service | What it does |
|---|---|
| Academic Graph | Paper search, bulk search, title match, autocomplete, snippet search, paper details, batch lookup, citations, references, author search |
| Recommendations | Papers similar to one paper, or to a positive/negative example set |
| Datasets | Bulk corpus downloads, including S2ORC |
Full text is not a query-time product here. It lives in S2ORC: 8 million-plus full-text papers, alongside 81 million paper nodes and 73 million abstracts, distributed as a bulk download. There is also a snippet search endpoint over open-access papers, which returns short extracts rather than the retrieval-depth passages a RAG pipeline usually wants.
So: metadata and abstracts at query time, full text as a corpus you host yourself.
Valyu full-text-indexes roughly 4 million open-access papers, plus licensed journal content:
| Source | Full-text coverage |
|---|---|
| PubMed | 2.5M+ |
| arXiv | 1M+ |
| bioRxiv | 350K+ |
| medRxiv | 80K+ |
| ChemRxiv | 8K+ |
PubMed's complete 37 million-record abstract corpus is available as an opt-in via include_abstracts
. More on that flag below, because it is the single most commonly misreported detail about this API.
Beyond the literature, the index covers ClinicalTrials.gov (500K+ trials), FDA drug labels from DailyMed (150K+), SEC filings (3M+), USPTO patents (8M+) and EPO patents (6M+) with full text and figures, plus genomics and chemistry sources. One query can span several of those at once, which is the part that matters if your agent needs to go from a mechanism in a paper to the trial testing it to the label of the approved drug.
Both do discovery. Here is Semantic Scholar, rewritten from the docs example as something you would actually put in an agent rather than an interactive prompt loop:
import os
import requests
S2_BASE = "https://api.semanticscholar.org"
HEADERS = {"X-API-KEY": os.environ["S2_API_KEY"]}
def search_papers(query: str, limit: int = 10):
r = requests.get(
f"{S2_BASE}/graph/v1/paper/search",
headers=HEADERS,
params={
"query": query,
"limit": limit,
"fields": "title,abstract,year,citationCount,externalIds,url",
},
timeout=30,
)
r.raise_for_status()
return r.json().get("data", [])
def recommendations(paper_id: str, limit: int = 10):
r = requests.get(
f"{S2_BASE}/recommendations/v1/papers/forpaper/{paper_id}",
headers=HEADERS,
params={"fields": "title,year,citationCount,url", "limit": limit},
timeout=30,
)
r.raise_for_status()
return r.json()["recommendedPapers"]
for p in search_papers("chimeric antigen receptor T cell exhaustion"):
print(f"{p['citationCount']:>6} {p['year']} {p['title']}")
The fields
parameter is the thing to learn first. Ask for nothing and you get almost nothing back, and every extra field is a join on their side, so keep the list tight.
For anything above a few thousand results, use bulk search with its continuation token instead of paging the relevance endpoint:
import json
import requests
url = "https://api.semanticscholar.org/graph/v1/paper/search/bulk"
params = {"query": "(cold -temperature) | flu", "fields": "title,year", "year": "2023-"}
retrieved = 0
with open("papers.jsonl", "a") as f:
while True:
r = requests.get(url, params=params, timeout=60).json()
for paper in r.get("data", []):
print(json.dumps(paper), file=f)
retrieved += len(r.get("data", []))
if "token" not in r:
break
params["token"] = r["token"]
print(f"Retrieved {retrieved} papers")
Note the query syntax on the bulk endpoint: |
is OR, a leading -
negates, and parentheses group. It is not the same syntax as the relevance search endpoint, which trips people up.
Valyu does discovery too, over a smaller but full-text index:
import os
from valyu import Valyu
valyu = Valyu(api_key=os.environ["VALYU_API_KEY"])
response = valyu.search(
"Phase 3 melanoma immunotherapy trials",
search_type="proprietary",
included_sources=["valyu/valyu-pubmed", "valyu/valyu-clinical-trials"],
max_num_results=15,
response_length="large", # full methodology and results, not just abstracts
)
for result in response.results:
print(result.title, result.url)
print(result.content)
There are TypeScript and Rust SDKs if Python is not your stack, and the REST endpoints are there if you would rather not add a dependency at all.
Semantic Scholar. To search inside papers you download S2ORC, chunk it, embed it, store it, and query your own index. That is a real pipeline: object storage, an embedding job, a vector database, and a refresh strategy when the corpus updates. Perfectly reasonable if you want control over chunking and embeddings, and genuinely the right call for some teams. It is just not a thing you get from an API call.
Valyu. Full text is the default return value:
response = valyu.search(
"mechanisms of acquired resistance to KRAS G12C inhibitors",
search_type="proprietary",
included_sources=["valyu/valyu-pubmed"],
max_num_results=10,
response_length="large",
)
for r in response.results:
print(r.title)
print(r.content[:500]) # relevant full-text chunks, not the abstract
print(r.citation.doi, r.citation.fragment) # fragment deep-links the passage
The fragment
field is worth calling out separately. It is a text-fragment deep link to the exact cited passage, so a reviewer can click a citation in your agent's output and land on the sentence rather than the paper. If anyone is ever going to audit what your agent claimed, that field is the difference between "reviewable" and "take my word for it".
include_abstracts
Gotcha
include_abstracts=true
:Full-text-first is the default. The complete abstract corpus is the opt-in. If you have read the opposite somewhere, that is the error.
valyu.search(query, included_sources=["valyu/valyu-pubmed"])
valyu.search(query, included_sources=["valyu/valyu-pubmed"], include_abstracts=True)
Use the default for evidence synthesis. Flip the flag for coverage sweeps and systematic-review-style screening, where missing a paper is worse than only having its abstract.
The Semantic Scholar API has no answer-generation endpoint. Ai2 does ship Ai2 Scholar QA separately: an open-source cited-synthesis system over 11 million-plus full-text papers and 100 million-plus abstracts, available as a Docker app, an async API, or a Python package. You bring your own Semantic Scholar, Anthropic, and OpenAI keys, and you host it. It is a good piece of software. It is also infrastructure you now operate.
Valyu ships synthesis as managed API surface. The Answer API returns a cited answer in one call. DeepResearch runs autonomous multi-step investigation: planning, searching, extraction, fact verification, and report writing.
import os
from valyu import Valyu
valyu = Valyu(api_key=os.environ["VALYU_API_KEY"])
task = valyu.deepresearch.create(
query=(
"What is the current evidence that GLP-1 receptor agonists reduce "
"major adverse cardiovascular events in patients without diabetes? "
"Cover trial design, effect sizes, and where the evidence conflicts."
),
mode="standard",
search={
"search_type": "proprietary",
"included_sources": ["academic"], # arXiv, PubMed, bioRxiv/medRxiv, ChemRxiv
"start_date": "2021-01-01",
},
research_strategy=(
"Prioritise randomised controlled trials and systematic reviews over "
"observational studies. Separate primary endpoints from secondary and "
"post-hoc analyses. Flag any conflicting or null results explicitly."
),
report_format=(
"Structured review with: evidence summary table (trial, n, population, "
"endpoint, effect size, CI), narrative synthesis, conflicting findings, "
"and evidence gaps."
),
output_formats=["markdown", "pdf"],
)
result = valyu.deepresearch.wait(task.deepresearch_id)
if result.status == "completed":
print(result.output)
print("cost:", result.cost)
for s in result.sources:
print(f"{s.title} | {s.doi or s.url}{s.fragment or ''}")
research_strategy
and report_format
are the two parameters that do the most work. They are where you encode the methodology a domain expert would apply, and they are the difference between a report you can hand to someone and a wall of summarised abstracts.
Point it at a single dataset when you know exactly where the evidence lives, and ask for a spreadsheet instead of prose:
task = valyu.deepresearch.create(
query=(
"Summarise reported mechanisms of acquired resistance to KRAS G12C "
"inhibitors in non-small-cell lung cancer, with supporting evidence "
"for each mechanism."
),
mode="fast",
search={
"search_type": "proprietary",
"included_sources": ["valyu/valyu-pubmed"],
"start_date": "2022-01-01",
},
deliverables=["xlsx"], # mechanism-by-evidence table
tools={"code_execution": True}, # required for xlsx/pptx/docx
)
DeepResearch also supports webhooks and human-in-the-loop checkpoints, which is what you want when a run takes minutes rather than milliseconds and you do not want to hold a request open.
Valyu is usage-based, priced per thousand results by source type, with max_price
as a per-query spend cap. Sources priced above the cap are excluded and the response returns a 206
partial-success warning rather than a surprise on the invoice. Free credits are available to start, $10 without a card and $20 with a work email.
Semantic Scholar is free, which is good. The limits are the catch, and they are widely misread:
One request per second is a hard ceiling on any fan-out design. Batch endpoints and bulk search exist precisely because of it, and you should build around them from day one rather than discovering the limit in production.
| Feature | Semantic Scholar | Valyu |
|---|---|---|
| Base URL | api.semanticscholar.org |
|
api.valyu.ai |
||
| Auth | Optional key; key gives dedicated quota | |
x-api-key , required |
||
| Corpus size | 214M papers, 2.49B citations, 79M authors | ~4M full-text papers, 37M PubMed abstracts opt-in, plus non-academic sources |
| Full text at query time | No (S2ORC bulk download, plus a snippet endpoint) | Yes, default for PubMed and arXiv |
| Citation graph traversal | Yes, first class | No dedicated graph endpoints |
| Author profiles | Yes, /author endpoints |
|
| No dedicated author endpoint | ||
| Recommendations | Yes, dedicated service | No |
| Cited answer generation | Not in the API (Ai2 Scholar QA is separate and self-hosted) | Answer API |
| Multi-step research agent | No | DeepResearch, plus templated Workflows |
| Deliverables | JSON | JSON, markdown, PDF, xlsx, docx, pptx, csv |
| Provenance | IDs, DOIs, citation counts | Title, URL, DOI, venue, authors, plus passage-level fragment links |
| Rate limit | 1 req/s with a key; 1,000 req/s shared unauthenticated | Usage-based, max_price per-query cap |
Semantic Scholar fits work where the graph is the point:
Valyu fits work where the evidence is the point:
Identical input, different object types, which is the clearest way to see the design split.
A topic search for "melanoma immunotherapy" on Semantic Scholar returns paper records: titles, abstracts, citation counts, author links, external IDs. The same query on Valyu returns full-text passages with citations attached to each one.
Ask "which papers cite this one" and Semantic Scholar answers directly through /paper/{id}/citations
. Valyu has no equivalent, because it is not a graph.
Ask "what does the evidence say about X" and Valyu answers with passages, or with a cited answer if you call the Answer API. Semantic Scholar returns candidate records for you to go and read.
This is a difference in design goal, not in quality. One is built for graph discovery. The other is built for evidence retrieval.
Semantic Scholar is a REST API returning JSON, with community client libraries in most languages. That is the whole integration story, and for many teams it is enough.
Valyu ships a hosted MCP server, a CLI, a Claude Code plugin, agent skills, and framework integrations for the Vercel AI SDK, LangChain, LlamaIndex, AWS Bedrock AgentCore, and n8n, plus tool definitions for Anthropic, OpenAI, and Google. If you are wiring retrieval into an existing agent framework, that is usually a config change rather than a client to write.
For a sense of scale in production: RevisionDojo uses Valyu to deliver academic research to more than 450,000 students, integrating the JavaScript SDK and the Valyu AI SDK across search, citation discovery, and structured literature reviews for IB Extended Essays.
Not at query time. Full text ships as the S2ORC bulk dataset, 8 million-plus full-text papers alongside 81 million paper nodes and 73 million abstracts, which you download and index yourself. There is also a snippet search endpoint over open-access papers that returns short extracts.
Not in the API. Ai2 separately ships Ai2 Scholar QA, an open-source cited-synthesis system over 11 million-plus full-text papers and 100 million-plus abstracts, available as a Docker app, async API, or Python package. It requires your own Semantic Scholar, Anthropic, and OpenAI keys, and you host it.
No, and this is commonly stated backwards. PubMed search defaults to papers with full text available, returning the abstract plus relevant full-text chunks. Set include_abstracts=true
to expand to the complete 37 million-record abstract corpus, where papers without full text return their abstract.
Roughly 4 million full-text open-access papers: PubMed 2.5M, arXiv 1M, bioRxiv 350K, medRxiv 80K, ChemRxiv 8K, plus licensed journal content. PubMed's complete 37 million-record abstract corpus is available via include_abstracts
.
Search returns structured results. Contents extracts clean content from URLs. Answer adds cited answer generation on top of search. DeepResearch runs autonomous multi-step investigation with file deliverables. Workflows are templated, versioned DeepResearch runs for repeatable work.
Coverage currently centres on US trials and FDA-approved drugs, with 24 to 48 hour update delays on trial data.
Yes, and it is usually the right answer. Semantic Scholar identifies which works matter through the citation graph, and Valyu retrieves the full-text passages that support specific claims. See the code above.
No. Preprints from arXiv, bioRxiv, medRxiv, and ChemRxiv are not peer-reviewed, and both services index them alongside peer-reviewed articles. Filter on source or publication type when evidence quality matters.