SHERLOCK PDF Query: How does sleep affect hippocampal memory consolidation?
Best evidence: 0.91 | sleep_memory_study.pdf, page 5 | Relevant excerpt…0.87 | hippocampus_review.pdf, page 12 | Another relevant excerpt…
That is what you’ll build in this tutorial: ask a question in plain language, and get back the best-matching passages with the filename and page number. Deerstalker hat optional. 😉
What you’ll need: Python 3.10+, an OpenAI API key, a few text-based PDFs, and about 15 minutes. Embedding a few papers costs only cents.
Scientists consume an extraordinary amount of information: papers, preprints, reviews, supplementary files, protocols, and the occasional 87-page PDF named final_version_REALLY_FINAL_2.4.pdf. Reading it all is hard. The real challenge comes weeks later:
Which paper discussed that result? What page was it on? What was the exact paragraph?
Our brains are great at forming ideas and less great at storing the coordinates of every useful sentence. Reopening ten papers and trying keyword combinations can take longer than the experiment that raised the question.
An LLM application is more than one prompt. Documents must be loaded, cleaned, split into pieces, converted into numerical representations, searched, and traced back to their sources. LangChain offers reusable components for each step (s, splitters, embeddings, vector stores) behind a consistent interface. Less plumbing, more questions.
How does Sherlock decide that a passage resembles a question? A common measure is cosine similarity, which compares the direction of two vectors rather than their length:
Here, A . B is the dot product, ||A|| and ||B|| are the lengths of the vectors, and theta is the angle between them. A score near 1 means that the vectors point in very similar directions; a score near 0 means little directional similarity.
Picture two clue cards. One says “muddy footprints, tobacco ash, and a missing laboratory key.” The other says “soil on a boot, cigar residue, and unauthorized access to the lab.” The wording differs, but the clues point in nearly the same direction. A card about formatting a bibliography points somewhere else entirely.
That is why semantic search can find a relevant paragraph even when it doesn’t repeat your words. And the page number is my favorite detail: a plausible answer is useful, but a passage you can locate, read in context, and verify is far more useful. Scientific retrieval should lead you back to evidence, not ask you to trust a mysterious oracle in the cloud.
This tutorial reproduces Sherlock PDF, a small LangChain application that searches scientific PDFs and returns the most relevant excerpts with their filenames and page numbers.
Clone or download the repository, then open a terminal inside it:
git clone "https://github.com/suvrazastrovision/langchain_demo.git"cd path\to\langchain_demo
The project should look like this:
python -m venv .venv.\.venv\Scripts\Activate.ps1
For macOS or Linux:
python3 -m venv .venvsource .venv/bin/activate
python -m pip install -r requirements.txt
The project uses:
Create a .env file in the project directory:
OPENAI_API_KEY=your_api_key_here
Do not commit this file or share your API key publicly. The repository’s .gitignore already excludes .env.
Sherlock PDF is a small demo and therefore I have chosen text-embedding-3-small. Embeddings convert text into numerical vectors so semantically related questions and passages can be found even when they use different wording. OpenAI bills embedding requests according to the number of input tokens. See the official OpenAI embeddings guide.
Place one or more PDFs inside the docs directory:
docs/├── nature_emm.pdf├── nn_ad.pdf└── nn_sleep.pdf
I have used text-based PDFs for this demo. One can also search images and scanned documents. Scanned documents containing only images will require an OCR step before their text can be searched. Also remember that PDF text is sent to the OpenAI API to create embeddings. Do not process confidential or unpublished documents unless your organisation’s data policy permits it.
Open sherlock_pdf.py and locate the main() function:
def main(): query = ( "How do inhibitory glutamatergic synapses between dorsal fan-shaped " "body neurons generate slow-wave activity and regulate sleep pressure " "in Drosophila?")
Replace the question with something relevant to the content of your papers or documents. I am using couple of scientific journal articles from Nature here:
query = "How does sleep affect hippocampal memory consolidation?"
You can also change how many passages are returned:
results = hunt(query, store, top_k=3)
For example, use top_k=5 to retrieve five excerpts.
Run “python sherlock_pdf.py” or alternatively copy the following code:
from pathlib import Pathfrom dotenv import load_dotenvfrom langchain_community.document_s import Directory, PyMuPDFfrom langchain_core.vectorstores import InMemoryVectorStorefrom langchain_openai import OpenAIEmbeddingsfrom langchain_text_splitters import RecursiveCharacterTextSplitterDOCS_PATH = Path(__file__).resolve().parent / "docs"EMBEDDING_MODEL = "text-embedding-3-small"load_dotenv(Path(__file__).resolve().parent / ".env")def load_notes(folder): """Load PDF research notes as LangChain Document objects.""" = Directory( str(folder), glob="*.pdf", _cls=PyMuPDF, ) return .load()def split_notes(documents): """Split loaded notes while preserving document metadata.""" splitter = RecursiveCharacterTextSplitter( chunk_size=220, chunk_overlap=40, separators=["\n\n", "\n", " ", ""], ) return splitter.split_documents(documents)def build_store(chunks): """Build an in-memory LangChain vector store from document chunks.""" embeddings = OpenAIEmbeddings(model=EMBEDDING_MODEL) return InMemoryVectorStore.from_documents(chunks, embeddings)def hunt(query, store, top_k=3): """Return the strongest evidence chunks for a query.""" if not isinstance(top_k, int) or isinstance(top_k, bool) or top_k <= 0: raise ValueError("top_k must be a positive integer") matches = store.similarity_search_with_score(query, k=top_k) return [ { "score": score, "source": Path(document.metadata["source"]).name, # PyMuPDF stores a zero-based PDF page index. "page": document.metadata["page"] + 1, "text": document.page_content, } for document, score in matches ]def print_evidence(query, results): """Print a compact evidence report.""" print("SHERLOCK PDF") print(f"Query: {query}") print() if not results: print("Best evidence:") print("No evidence found.") return print("Best evidence:") for item in results: print( f"{item['score']:.2f} | {item['source']}, page {item['page']} | " f"{item['text']}" )def main(): query = ( "How do inhibitory glutamatergic synapses between dorsal fan-shaped " "body neurons generate slow-wave activity and regulate sleep pressure " "in Drosophila?") documents = load_notes(DOCS_PATH) chunks = split_notes(documents) store = build_store(chunks) results = hunt(query, store, top_k=3) print_evidence(query, results)if __name__ == "__main__": main()
For my querry output looks like this:
The script performs five main operations:
Conceptually:
PDFs → pages → text chunks → embeddings → similarity search → evidence
The page metadata survives the chunking process. Because PyMuPDF numbers pages starting from zero, Sherlock PDF adds one before displaying the page number.
Ask the same question using different terminology:
query = "Does sleep make hippocampal connections weaker?"
A keyword search might miss a paragraph that says “Synaptic downscaling occurs during slow-wave sleep in CA3.” Semantic search can still connect the two, because they discuss related concepts in different words.
The same pipeline scales from a few PDFs to thousands of documents once they’re indexed in a persistent vector database. Add filters (author, year, journal, topic) and a simple interface, and an entire lab could share one searchable research memory.
For one job, yes: asking a natural-language question and getting a semantically related passage with filename and page number. But EndNote and Mendeley are mature tools for collecting references, annotating PDFs, collaborating, and building bibliographies, and this demo does none of that.
So the real comparison isn’t replacement versus obsolete software. It’s semantic evidence retrieval versus reference management. Watson manages the case files; Sherlock follows the conceptual clues.
We may never memorize every paragraph we read, and perhaps we should stop trying. Sherlock PDF is a retrieval assistant, not a truth machine. It tells you where to start reading; interpretation and verification still belong to you. Even Sherlock needed Watson, and preferably peer review. 😄
Elementary, my dear reviewer.
Thanks for reading! If you try Sherlock PDF on your own papers, I’d love to hear how it goes.
Thank you, and happy hunting!
A Hands-On LangChain Tutorial: Building “Sherlock PDF” for Smarter Scientific Literature Search was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.