A Hands-On LangChain Tutorial: Building “Sherlock PDF” for Smarter Scientific Literature Search

A hands-on tutorial walks developers through building "Sherlock PDF," a LangChain application that searches text-based scientific PDFs and returns the best-matching passages with filename and page number, using OpenAI's text-embedding-3-small model and cosine similarity. The tutorial requires Python 3.10+, an OpenAI API key, and about 15 minutes, and states that embedding a few papers costs only cents. The project is published in the GitHub repository suvrazastrovision/langchain_demo.

SHERLOCK PDF Query: How does sleep affect hippocampal memory consolidation? Best evidence: 0.91 | sleep memory study.pdf, page 5 | Relevant excerpt…0.87 | hippocampus review.pdf, page 12 | Another relevant excerpt… That is what you’ll build in this tutorial: ask a question in plain language, and get back the best-matching passages with the filename and page number . Deerstalker hat optional. 😉 What you’ll need: Python 3.10+, an OpenAI API key, a few text-based PDFs, and about 15 minutes. Embedding a few papers costs only cents. Scientists consume an extraordinary amount of information: papers, preprints, reviews, supplementary files, protocols, and the occasional 87-page PDF named final version REALLY FINAL 2.4.pdf. Reading it all is hard. The real challenge comes weeks later: Which paper discussed that result? What page was it on? What was the exact paragraph? Our brains are great at forming ideas and less great at storing the coordinates of every useful sentence. Reopening ten papers and trying keyword combinations can take longer than the experiment that raised the question. An LLM application is more than one prompt. Documents must be loaded, cleaned, split into pieces, converted into numerical representations, searched, and traced back to their sources. LangChain offers reusable components for each step loaders, splitters, embeddings, vector stores behind a consistent interface. Less plumbing, more questions. How does Sherlock decide that a passage resembles a question? A common measure is cosine similarity , which compares the direction of two vectors rather than their length: Here, A . B is the dot product, ||A|| and ||B|| are the lengths of the vectors, and theta is the angle between them. A score near 1 means that the vectors point in very similar directions; a score near 0 means little directional similarity. Picture two clue cards. One says “muddy footprints, tobacco ash, and a missing laboratory key.” The other says “soil on a boot, cigar residue, and unauthorized access to the lab.” The wording differs, but the clues point in nearly the same direction. A card about formatting a bibliography points somewhere else entirely. That is why semantic search can find a relevant paragraph even when it doesn’t repeat your words. And the page number is my favorite detail: a plausible answer is useful, but a passage you can locate, read in context, and verify is far more useful. Scientific retrieval should lead you back to evidence, not ask you to trust a mysterious oracle in the cloud. This tutorial reproduces Sherlock PDF , a small LangChain application that searches scientific PDFs and returns the most relevant excerpts with their filenames and page numbers. Clone or download the repository, then open a terminal inside it: git clone "https://github.com/suvrazastrovision/langchain demo.git"cd path\to\langchain demo The project should look like this: python -m venv .venv.\.venv\Scripts\Activate.ps1 For macOS or Linux: python3 -m venv .venvsource .venv/bin/activate python -m pip install -r requirements.txt The project uses: Create a .env file in the project directory: OPENAI API KEY=your api key here Do not commit this file or share your API key publicly. The repository’s .gitignore already excludes .env. Sherlock PDF is a small demo and therefore I have chosen text-embedding-3-small. Embeddings convert text into numerical vectors so semantically related questions and passages can be found even when they use different wording. OpenAI bills embedding requests according to the number of input tokens. See the official OpenAI embeddings guide https://developers.openai.com/api/docs/guides/embeddings . Place one or more PDFs inside the docs directory: docs/├── nature emm.pdf├── nn ad.pdf└── nn sleep.pdf I have used text-based PDFs for this demo. One can also search images and scanned documents. Scanned documents containing only images will require an OCR step before their text can be searched. Also remember that PDF text is sent to the OpenAI API to create embeddings. Do not process confidential or unpublished documents unless your organisation’s data policy permits it. Open sherlock pdf.py and locate the main function: python def main : query = "How do inhibitory glutamatergic synapses between dorsal fan-shaped " "body neurons generate slow-wave activity and regulate sleep pressure " "in Drosophila?" Replace the question with something relevant to the content of your papers or documents. I am using couple of scientific journal articles from Nature here: query = "How does sleep affect hippocampal memory consolidation?" You can also change how many passages are returned: results = hunt query, store, top k=3 For example, use top k=5 to retrieve five excerpts. Run “python sherlock pdf.py” or alternatively copy the following code: python from pathlib import Pathfrom dotenv import load dotenvfrom langchain community.document loaders import DirectoryLoader, PyMuPDFLoaderfrom langchain core.vectorstores import InMemoryVectorStorefrom langchain openai import OpenAIEmbeddingsfrom langchain text splitters import RecursiveCharacterTextSplitterDOCS PATH = Path file .resolve .parent / "docs"EMBEDDING MODEL = "text-embedding-3-small"load dotenv Path file .resolve .parent / ".env" def load notes folder : """Load PDF research notes as LangChain Document objects.""" loader = DirectoryLoader str folder , glob=" .pdf", loader cls=PyMuPDFLoader, return loader.load def split notes documents : """Split loaded notes while preserving document metadata.""" splitter = RecursiveCharacterTextSplitter chunk size=220, chunk overlap=40, separators= "\n\n", "\n", " ", "" , return splitter.split documents documents def build store chunks : """Build an in-memory LangChain vector store from document chunks.""" embeddings = OpenAIEmbeddings model=EMBEDDING MODEL return InMemoryVectorStore.from documents chunks, embeddings def hunt query, store, top k=3 : """Return the strongest evidence chunks for a query.""" if not isinstance top k, int or isinstance top k, bool or top k <= 0: raise ValueError "top k must be a positive integer" matches = store.similarity search with score query, k=top k return { "score": score, "source": Path document.metadata "source" .name, PyMuPDFLoader stores a zero-based PDF page index. "page": document.metadata "page" + 1, "text": document.page content, } for document, score in matches def print evidence query, results : """Print a compact evidence report.""" print "SHERLOCK PDF" print f"Query: {query}" print if not results: print "Best evidence:" print "No evidence found." return print "Best evidence:" for item in results: print f"{item 'score' :.2f} | {item 'source' }, page {item 'page' } | " f"{item 'text' }" def main : query = "How do inhibitory glutamatergic synapses between dorsal fan-shaped " "body neurons generate slow-wave activity and regulate sleep pressure " "in Drosophila?" documents = load notes DOCS PATH chunks = split notes documents store = build store chunks results = hunt query, store, top k=3 print evidence query, results if name == " main ": main For my querry output looks like this: The script performs five main operations: Conceptually: PDFs → pages → text chunks → embeddings → similarity search → evidence The page metadata survives the chunking process. Because PyMuPDF numbers pages starting from zero, Sherlock PDF adds one before displaying the page number. Ask the same question using different terminology: query = "Does sleep make hippocampal connections weaker?" A keyword search might miss a paragraph that says “Synaptic downscaling occurs during slow-wave sleep in CA3.” Semantic search can still connect the two, because they discuss related concepts in different words. The same pipeline scales from a few PDFs to thousands of documents once they’re indexed in a persistent vector database. Add filters author, year, journal, topic and a simple interface, and an entire lab could share one searchable research memory. For one job, yes: asking a natural-language question and getting a semantically related passage with filename and page number. But EndNote and Mendeley are mature tools for collecting references, annotating PDFs, collaborating, and building bibliographies, and this demo does none of that. So the real comparison isn’t replacement versus obsolete software. It’s semantic evidence retrieval versus reference management . Watson manages the case files; Sherlock follows the conceptual clues. We may never memorize every paragraph we read, and perhaps we should stop trying. Sherlock PDF is a retrieval assistant, not a truth machine. It tells you where to start reading ; interpretation and verification still belong to you. Even Sherlock needed Watson, and preferably peer review. 😄 Elementary, my dear reviewer. Thanks for reading If you try Sherlock PDF on your own papers, I’d love to hear how it goes. Thank you, and happy hunting A Hands-On LangChain Tutorial: Building “Sherlock PDF” for Smarter Scientific Literature Search https://pub.towardsai.net/a-hands-on-langchain-tutorial-building-sherlock-pdf-for-smarter-scientific-literature-search-be56f5cde245 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.