{"slug": "a-hands-on-langchain-tutorial-building-sherlock-pdf-for-smarter-scientific", "title": "A Hands-On LangChain Tutorial: Building “Sherlock PDF” for Smarter Scientific Literature Search", "summary": "A hands-on tutorial walks developers through building \"Sherlock PDF,\" a LangChain application that searches text-based scientific PDFs and returns the best-matching passages with filename and page number, using OpenAI's text-embedding-3-small model and cosine similarity. The tutorial requires Python 3.10+, an OpenAI API key, and about 15 minutes, and states that embedding a few papers costs only cents. The project is published in the GitHub repository suvrazastrovision/langchain_demo.", "body_md": "*SHERLOCK PDF Query: How does sleep affect hippocampal memory consolidation?*\n\n*Best evidence: 0.91 | sleep_memory_study.pdf, page 5 | Relevant excerpt…0.87 | hippocampus_review.pdf, page 12 | Another relevant excerpt…*\n\nThat is what you’ll build in this tutorial: ask a question in plain language, and get back the best-matching passages with the **filename and page number**. Deerstalker hat optional. 😉\n\n**What you’ll need:** Python 3.10+, an OpenAI API key, a few text-based PDFs, and about 15 minutes. Embedding a few papers costs only cents.\n\nScientists consume an extraordinary amount of information: papers, preprints, reviews, supplementary files, protocols, and the occasional 87-page PDF named final_version_REALLY_FINAL_2.4.pdf. Reading it all is hard. The real challenge comes weeks later:\n\n*Which paper discussed that result? What page was it on? What was the exact paragraph?*\n\nOur brains are great at forming ideas and less great at storing the coordinates of every useful sentence. Reopening ten papers and trying keyword combinations can take longer than the experiment that raised the question.\n\nAn LLM application is more than one prompt. Documents must be loaded, cleaned, split into pieces, converted into numerical representations, searched, and traced back to their sources. LangChain offers reusable components for each step (loaders, splitters, embeddings, vector stores) behind a consistent interface. Less plumbing, more questions.\n\nHow does Sherlock decide that a passage resembles a question? A common measure is **cosine similarity**, which compares the *direction* of two vectors rather than their length:\n\nHere, A . B is the dot product, ||A|| and ||B|| are the lengths of the vectors, and theta is the angle between them. A score near 1 means that the vectors point in very similar directions; a score near 0 means little directional similarity.\n\nPicture two clue cards. One says “muddy footprints, tobacco ash, and a missing laboratory key.” The other says “soil on a boot, cigar residue, and unauthorized access to the lab.” The wording differs, but the clues point in nearly the same direction. A card about formatting a bibliography points somewhere else entirely.\n\nThat is why semantic search can find a relevant paragraph even when it doesn’t repeat your words. And the page number is my favorite detail: a plausible answer is useful, but a passage you can locate, read in context, and verify is far more useful. Scientific retrieval should lead you back to evidence, not ask you to trust a mysterious oracle in the cloud.\n\nThis tutorial reproduces **Sherlock PDF**, a small LangChain application that searches scientific PDFs and returns the most relevant excerpts with their filenames and page numbers.\n\nClone or download the repository, then open a terminal inside it:\n\n```\ngit clone \"https://github.com/suvrazastrovision/langchain_demo.git\"cd path\\to\\langchain_demo\n```\n\nThe project should look like this:\n\n```\npython -m venv .venv.\\.venv\\Scripts\\Activate.ps1\n```\n\nFor macOS or Linux:\n\n```\npython3 -m venv .venvsource .venv/bin/activate\npython -m pip install -r requirements.txt\n```\n\nThe project uses:\n\nCreate a .env file in the project directory:\n\n```\nOPENAI_API_KEY=your_api_key_here\n```\n\nDo not commit this file or share your API key publicly. The repository’s .gitignore already excludes .env.\n\nSherlock PDF is a small demo and therefore I have chosen text-embedding-3-small. Embeddings convert text into numerical vectors so semantically related questions and passages can be found even when they use different wording. OpenAI bills embedding requests according to the number of input tokens. See the [official OpenAI embeddings guide](https://developers.openai.com/api/docs/guides/embeddings).\n\nPlace one or more PDFs inside the docs directory:\n\n```\ndocs/├── nature_emm.pdf├── nn_ad.pdf└── nn_sleep.pdf\n```\n\nI have used text-based PDFs for this demo. One can also search images and scanned documents. Scanned documents containing only images will require an OCR step before their text can be searched. Also remember that PDF text is sent to the OpenAI API to create embeddings. Do not process confidential or unpublished documents unless your organisation’s data policy permits it.\n\nOpen sherlock_pdf.py and locate the main() function:\n\n``` python\ndef main():    query = (    \"How do inhibitory glutamatergic synapses between dorsal fan-shaped \"    \"body neurons generate slow-wave activity and regulate sleep pressure \"    \"in Drosophila?\")\n```\n\nReplace the question with something relevant to the content of your papers or documents. I am using couple of scientific journal articles from *Nature* here:\n\n```\nquery = \"How does sleep affect hippocampal memory consolidation?\"\n```\n\nYou can also change how many passages are returned:\n\n```\nresults = hunt(query, store, top_k=3)\n```\n\nFor example, use top_k=5 to retrieve five excerpts.\n\nRun *“python sherlock_pdf.py”* or alternatively copy the following code:\n\n``` python\nfrom pathlib import Pathfrom dotenv import load_dotenvfrom langchain_community.document_loaders import DirectoryLoader, PyMuPDFLoaderfrom langchain_core.vectorstores import InMemoryVectorStorefrom langchain_openai import OpenAIEmbeddingsfrom langchain_text_splitters import RecursiveCharacterTextSplitterDOCS_PATH = Path(__file__).resolve().parent / \"docs\"EMBEDDING_MODEL = \"text-embedding-3-small\"load_dotenv(Path(__file__).resolve().parent / \".env\")def load_notes(folder):    \"\"\"Load PDF research notes as LangChain Document objects.\"\"\"    loader = DirectoryLoader(        str(folder),        glob=\"*.pdf\",        loader_cls=PyMuPDFLoader,    )    return loader.load()def split_notes(documents):    \"\"\"Split loaded notes while preserving document metadata.\"\"\"    splitter = RecursiveCharacterTextSplitter(        chunk_size=220,        chunk_overlap=40,        separators=[\"\\n\\n\", \"\\n\", \" \", \"\"],    )    return splitter.split_documents(documents)def build_store(chunks):    \"\"\"Build an in-memory LangChain vector store from document chunks.\"\"\"    embeddings = OpenAIEmbeddings(model=EMBEDDING_MODEL)    return InMemoryVectorStore.from_documents(chunks, embeddings)def hunt(query, store, top_k=3):    \"\"\"Return the strongest evidence chunks for a query.\"\"\"    if not isinstance(top_k, int) or isinstance(top_k, bool) or top_k <= 0:        raise ValueError(\"top_k must be a positive integer\")    matches = store.similarity_search_with_score(query, k=top_k)    return [        {            \"score\": score,            \"source\": Path(document.metadata[\"source\"]).name,            # PyMuPDFLoader stores a zero-based PDF page index.            \"page\": document.metadata[\"page\"] + 1,            \"text\": document.page_content,        }        for document, score in matches    ]def print_evidence(query, results):    \"\"\"Print a compact evidence report.\"\"\"    print(\"SHERLOCK PDF\")    print(f\"Query: {query}\")    print()    if not results:        print(\"Best evidence:\")        print(\"No evidence found.\")        return    print(\"Best evidence:\")    for item in results:        print(            f\"{item['score']:.2f} | {item['source']}, page {item['page']} | \"            f\"{item['text']}\"        )def main():    query = ( \"How do inhibitory glutamatergic synapses between dorsal fan-shaped \"    \"body neurons generate slow-wave activity and regulate sleep pressure \"    \"in Drosophila?\")    documents = load_notes(DOCS_PATH)    chunks = split_notes(documents)    store = build_store(chunks)    results = hunt(query, store, top_k=3)    print_evidence(query, results)if __name__ == \"__main__\":    main()\n```\n\nFor my querry output looks like this:\n\nThe script performs five main operations:\n\nConceptually:\n\n*PDFs → pages → text chunks → embeddings → similarity search → evidence*\n\nThe page metadata survives the chunking process. Because PyMuPDF numbers pages starting from zero, Sherlock PDF adds one before displaying the page number.\n\nAsk the same question using different terminology:\n\n```\nquery = \"Does sleep make hippocampal connections weaker?\"\n```\n\nA keyword search might miss a paragraph that says *“Synaptic downscaling occurs during slow-wave sleep in CA3.”* Semantic search can still connect the two, because they discuss related concepts in different words.\n\nThe same pipeline scales from a few PDFs to thousands of documents once they’re indexed in a persistent vector database. Add filters (author, year, journal, topic) and a simple interface, and an entire lab could share one searchable research memory.\n\nFor one job, yes: asking a natural-language question and getting a semantically related passage with filename and page number. But EndNote and Mendeley are mature tools for collecting references, annotating PDFs, collaborating, and building bibliographies, and this demo does none of that.\n\nSo the real comparison isn’t replacement versus obsolete software. It’s **semantic evidence retrieval versus reference management**. Watson manages the case files; Sherlock follows the conceptual clues.\n\nWe may never memorize every paragraph we read, and perhaps we should stop trying. Sherlock PDF is a retrieval assistant, not a truth machine. It tells you *where to start reading*; interpretation and verification still belong to you. Even Sherlock needed Watson, and preferably peer review. 😄\n\nElementary, my dear reviewer.\n\nThanks for reading! If you try Sherlock PDF on your own papers, I’d love to hear how it goes.\n\nThank you, and happy hunting!\n\n[A Hands-On LangChain Tutorial: Building “Sherlock PDF” for Smarter Scientific Literature Search](https://pub.towardsai.net/a-hands-on-langchain-tutorial-building-sherlock-pdf-for-smarter-scientific-literature-search-be56f5cde245) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/a-hands-on-langchain-tutorial-building-sherlock-pdf-for-smarter-scientific", "canonical_source": "https://pub.towardsai.net/a-hands-on-langchain-tutorial-building-sherlock-pdf-for-smarter-scientific-literature-search-be56f5cde245?source=rss----98111c9905da---4", "published_at": "2026-10-19 06:09:51+00:00", "updated_at": "2026-10-19 06:19:59.082341+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "developer-tools", "ai-search"], "entities": ["LangChain", "OpenAI", "Sherlock PDF", "text-embedding-3-small", "GitHub", "suvrazastrovision/langchain_demo"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/a-hands-on-langchain-tutorial-building-sherlock-pdf-for-smarter-scientific", "markdown": "https://wpnews.pro/news/a-hands-on-langchain-tutorial-building-sherlock-pdf-for-smarter-scientific.md", "text": "https://wpnews.pro/news/a-hands-on-langchain-tutorial-building-sherlock-pdf-for-smarter-scientific.txt", "jsonld": "https://wpnews.pro/news/a-hands-on-langchain-tutorial-building-sherlock-pdf-for-smarter-scientific.jsonld"}}