{"slug": "build-semantic-search-for-legal-documents-with-pinecone-and-gpt-4", "title": "Build Semantic Search for Legal Documents with Pinecone and GPT-4", "summary": "A developer detailed a method for building semantic search for legal documents using Pinecone, GPT-4, and LangChain. The system enables law firms to search case law and contracts by meaning rather than keywords, using retrieval-augmented generation to provide accurate, cited answers. The prototype can be built in 4-6 hours, with production deployment taking 1-2 weeks.", "body_md": "You can build a semantic search system for legal documents by combining Pinecone (a vector database), GPT-4 (for intelligent retrieval), and LangChain (to orchestrate the pipeline). This setup lets law firms search case law, contracts, and precedents by meaning rather than keyword matching - so a query like \"wrongful termination without cause\" surfaces relevant statutes even when documents use different phrasing. The result is a retrieval-augmented generation (RAG) system that feeds the most relevant legal context to GPT-4, so it can cite sources and answer with accuracy.\n\nThe tech stack below represents a realistic, minimal setup to ship a working legal document semantic search system. Prices are current as of August 2024; verify each provider's current terms before committing.\n\n| Tool | Plan / Cost | Role |\n|---|---|---|\n| Pinecone (cloud vector DB) | Check current pricing at\n|\n\n**Time to build:** 4-6 hours for a working prototype on 100-500 documents; 1-2 weeks to production-grade (chunking strategy, fine-tuning relevance, auth, audit logging for legal compliance).\n\nLegal documents are too large to embed as a single vector. A 50-page contract or court opinion needs to be split into semantic chunks - typically 300-800 tokens each - so that GPT-4 can retrieve relevant sections without losing context.\n\nStart by uploading your documents (PDFs, Word files, plaintext) to a staging directory. Then use LangChain's document loaders to parse them:\n\n``` python\nfrom langchain_community.document_loaders import PyPDFLoader, UnstructuredPDFLoader\nfrom langchain.text_splitter import RecursiveCharacterTextSplitter\nimport os\n\n# Load all PDFs from a directory\ndocs = []\nfor filename in os.listdir('./legal_documents'):\n if filename.endswith('.pdf'):\n loader = PyPDFLoader(f'./legal_documents/{filename}')\n docs.extend(loader.load())\n\n# Split documents into chunks\nsplitter = RecursiveCharacterTextSplitter(\n chunk_size=800,\n chunk_overlap=100,\n separators=[\"\\n\\n\", \"\\n\", \" \"]\n)\nchunks = splitter.split_documents(docs)\n\nprint(f\"Created {len(chunks)} chunks from {len(docs)} documents\")\n```\n\nThis script loads PDFs, splits them into 800-token chunks with 100-token overlap (so key phrases aren't cut off), and reports how many chunks you've created. The overlap ensures that a search query at a chunk boundary still finds the right content. Adjust `chunk_size`\n\ndown to 400 tokens if your documents are dense with cross-references (e.g., statutes with many citations); go up to 1,200 if they're more narrative (e.g., case summaries).\n\nWhy this matters: **a chunk too large (5,000+ tokens) dilutes semantic relevance across unrelated sections; a chunk too small (<300 tokens) loses enough context that GPT-4 struggles to answer follow-up questions.** Most law firms find 500-800 tokens is the sweet spot.\n\nPinecone is a managed vector database that indexes embeddings for fast similarity search. You'll create an index, then embed each chunk and upsert it with metadata (document name, page number, date) so you can trace answers back to source.\n\nFirst, install dependencies and authenticate:\n\n```\npip install pinecone-client openai langchain langchain-openai langchain-pinecone\n```\n\nThen set your API keys as environment variables:\n\n```\nexport OPENAI_API_KEY=\"your-key-here\"\nexport PINECONE_API_KEY=\"your-key-here\"\nexport PINECONE_ENVIRONMENT=\"us-east-1\" # check your Pinecone console for your region\n```\n\nInitialize Pinecone and create an index:\n\n``` python\nfrom pinecone import Pinecone\nfrom openai import OpenAI\n\n# Initialize Pinecone\npc = Pinecone(api_key=os.environ.get(\"PINECONE_API_KEY\"))\n\n# Create index if it doesn't exist\nindex_name = \"legal-docs\"\nif index_name not in pc.list_indexes().names():\n pc.create_index(\n name=index_name,\n dimension=1536, # OpenAI's text-embedding-3-small produces 1536-dim vectors\n metric=\"cosine\",\n spec={\n \"serverless\": {\n \"cloud\": \"aws\",\n \"region\": \"us-east-1\"\n }\n }\n )\n\nindex = pc.Index(index_name)\nclient = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\"))\n```\n\nThe dimension is 1536 because OpenAI's `text-embedding-3-small`\n\nmodel outputs 1536-dimensional vectors. The `cosine`\n\nmetric measures similarity (0 = completely different, 1 = identical). Serverless means Pinecone auto-scales without you managing pods.\n\nNow embed and upload each chunk:\n\n``` python\ndef embed_and_upsert(chunks, batch_size=100):\n \"\"\"Embed chunks and upload to Pinecone.\"\"\"\n vectors_to_upsert = []\n\n for i, chunk in enumerate(chunks):\n # Get embedding from OpenAI\n response = client.embeddings.create(\n input=chunk.page_content,\n model=\"text-embedding-3-small\"\n )\n embedding = response.data[0].embedding\n\n # Prepare metadata\n metadata = {\n \"text\": chunk.page_content[:500], # first 500 chars for display\n \"source\": chunk.metadata.get(\"source\", \"unknown\"),\n \"page\": chunk.metadata.get(\"page\", 0),\n }\n\n vectors_to_upsert.append((\n f\"chunk-{i}\",\n embedding,\n metadata\n ))\n\n # Upsert in batches\n if (i + 1) % batch_size == 0:\n index.upsert(vectors=vectors_to_upsert)\n print(f\"Uploaded {i + 1} / {len(chunks)} chunks\")\n vectors_to_upsert = []\n\n # Upload remaining vectors\n if vectors_to_upsert:\n index.upsert(vectors=vectors_to_upsert)\n\n print(f\"Complete: {len(chunks)} chunks indexed\")\n\nembed_and_upsert(chunks)\n```\n\nThis batches uploads in groups of 100 to avoid timeout errors. Each vector is stored with metadata so you can show the user which document a result came from. **This step typically costs $20-$60 for 10,000 chunks** (depending on document size), because you pay OpenAI for embeddings.\n\nNow that embeddings are in Pinecone, you'll chain retrieval + GPT-4 reasoning together. When a user asks a question, LangChain will:\n\n``` python\nfrom langchain_openai import OpenAIEmbeddings, ChatOpenAI\nfrom langchain_pinecone import PineconeVectorStore\nfrom langchain.chains import RetrievalQA\nfrom langchain.prompts import PromptTemplate\n\n# Initialize embeddings and vector store\nembeddings = OpenAIEmbeddings(model=\"text-embedding-3-small\")\nvectorstore = PineconeVectorStore(\n index=index,\n embedding=embeddings\n)\n\n# Create a retriever (k=4 means return top 4 most similar chunks)\nretriever = vectorstore.as_retriever(search_kwargs={\"k\": 4})\n\n# Define a custom prompt for legal context\nlegal_prompt = PromptTemplate(\n input_variables=[\"context\", \"question\"],\n template=\"\"\"You are a legal research assistant. Use the following context from legal documents to answer the question. \nIf the answer is not in the context, say 'Not found in provided documents.'\nAlways cite the source document.\n\nContext:\n{context}\n\nQuestion: {question}\n\nAnswer:\"\"\"\n)\n\n# Create the QA chain\nllm = ChatOpenAI(model=\"gpt-4\", temperature=0.1) # low temperature for legal accuracy\nqa_chain = RetrievalQA.from_chain_type(\n llm=llm,\n chain_type=\"stuff\",\n retriever=retriever,\n chain_type_kwargs={\"prompt\": legal_prompt},\n return_source_documents=True\n)\n\n# Test the chain\nquestion = \"What are the grounds for wrongful termination in California?\"\nresponse = qa_chain.invoke({\"query\": question})\n\nprint(f\"Answer: {response['result']}\")\nprint(f\"Sources: {[doc.metadata['source'] for doc in response['source_documents']]}\")\n```\n\nThe `temperature=0.1`\n\nsetting keeps GPT-4 focused and factual - appropriate for legal work where hallucination is costly. The `stuff`\n\nchain type concatenates all retrieved documents and sends them to the LLM at once (fine for 4 chunks; for larger contexts, use `map_reduce`\n\nor `refine`\n\nmode). The `return_source_documents=True`\n\nflag ensures users can verify where answers came from.\n\nLegal document search often needs filtering by jurisdiction, case type, or date. Extend the retriever to support metadata-aware searches:\n\n``` python\nfrom langchain.retrievers.self_query.base import SelfQueryRetriever\nfrom langchain.chains.query_constructor.base import AttributeInfo\n\n# Define filterable metadata fields\nmetadata_field_info = [\n AttributeInfo(\n name=\"source\",\n description=\"The file name or document title (e.g., 'California_Labor_Code.pdf')\",\n type=\"string\",\n ),\n AttributeInfo(\n name=\"page\",\n description=\"Page number in the original document\",\n type=\"integer\",\n ),\n AttributeInfo(\n name=\"jurisdiction\",\n description=\"State or country (e.g., 'California', 'Federal')\",\n type=\"string\",\n ),\n]\n\n# Create a self-querying retriever that can filter\nretriever = SelfQueryRetriever.from_llm(\n llm=llm,\n vectorstore=vectorstore,\n document_contents=\"Legal documents: statutes, case law, contracts\",\n metadata_field_info=metadata_field_info,\n verbose=True\n)\n\n# Query with implicit filtering\nquestion = \"What does California labor law say about at-will employment?\"\nresponse = qa_chain.invoke({\"query\": question})\n```\n\nThe self-querying retriever uses GPT-4 to parse the user's question and automatically extract filters. So \"California labor law\" is interpreted as a filter on `jurisdiction='California'`\n\nplus a semantic search for \"at-will employment.\" This saves law firms from writing complex filter syntax.\n\nWrap the QA chain in a FastAPI server so law firm teams can query via web or integration:\n\n``` python\nfrom fastapi import FastAPI\nfrom pydantic import BaseModel\nimport uvicorn\n\napp = FastAPI(title=\"Legal Document Search\")\n\nclass QueryRequest(BaseModel):\n question: str\n top_k: int = 4\n\nclass QueryResponse(BaseModel):\n answer: str\n sources: list[str]\n\n@app.post(\"/search\")\nasync def search(request: QueryRequest):\n \"\"\"Search legal documents and return answer with sources.\"\"\"\n response = qa_chain.invoke({\n \"query\": request.question,\n \"k\": request.top_k\n })\n return QueryResponse(\n answer=response[\"result\"],\n sources=[doc.metadata[\"source\"] for doc in response[\"source_documents\"]]\n )\n\n@app.get(\"/health\")\nasync def health():\n return {\"status\": \"ok\"}\n\nif __name__ == \"__main__\":\n uvicorn.run(app, host=\"0.0.0.0\", port=8000)\n```\n\nRun it locally with `uvicorn app:app --reload`\n\n, then test with curl:\n\n```\ncurl -X POST http://localhost:8000/search \\\n -H \"Content-Type: application/json\" \\\n -d '{\"question\": \"What are non-compete clause limits in California?\", \"top_k\": 5}'\n```\n\nDeploy to a cloud platform (AWS Lambda, Google Cloud Run, or Heroku) for production. Add authentication (API keys or OAuth) if multiple users will access it.\n\nBuilding semantic search for legal documents introduces several real failure modes. Here's how to handle each:\n\n**Rate limits and embedding costs.** OpenAI's API has rate limits (3,500 requests per minute on the free tier, higher on paid accounts). If you're embedding thousands of documents at once, you'll hit these limits. **Fix:** batch your embedding jobs across multiple hours, or use exponential backoff with a sleep counter between API calls. Also monitor your embedding costs - at ~$0.02 per 1M tokens, 100,000 legal documents (50 pages average) can cost $100-$500 to embed. Budget for this upfront.\n\n**Stale or irrelevant retrieval.** If your top-k retrieved chunks don't actually answer the question, GPT-4 will admit it or make an inference that sounds plausible but is wrong. This happens when your chunk size is too large or too small, or when your documents have inconsistent formatting. **Fix:** test your retriever on 10-20 real legal questions before going live. If top results are off-topic, lower `chunk_size`\n\nto 500 tokens or adjust your prompt to be more strict (\"Answer only from the provided context\").\n\n**Pinecone costs scaling unexpectedly.** Pinecone charges per query for serverless mode. If you have 1,000 law firm users running 10 searches a day, that's 10,000 queries per day. Check Pinecone's current pricing page to estimate monthly cost. **Fix:** cache frequently asked questions (e.g., \"What is California wrongful termination law?\") so you don't re-query for identical questions. Use a Redis cache layer in front of your API.\n\n**Token limits in context.** GPT-4 has an 8,192-token context window (or 128,000 for GPT-4 Turbo). If you retrieve 10 chunks and each is 800 tokens, you're at 8,000 tokens before the user's question even arrives. **Fix:** limit `k`\n\nto 3-5 chunks, or use a reranker (a smaller, faster model that re-scores retrieved chunks for relevance before sending to GPT-4).\n\n**Missing or inconsistent metadata.** If PDFs don't have consistent metadata (page numbers, dates, author), your citations will be vague. **Fix:** add a preprocessing step that extracts and validates metadata from PDF headers, or manually tag documents before uploading.\n\n**Cold-start latency.** The first query after deployment can take 5-10 seconds because Pinecone needs to warm up. Users expect <1 second. **Fix:** warm up Pinecone with a dummy query on startup, or accept the cold-start and communicate it to users.\n\nIf you're building semantic search for legal documents, you have multiple vector database options. Here's how they compare:\n\n**Pinecone** (serverless, managed) is easiest for production because it's hosted and scales automatically. You pay per query and storage, with no infrastructure overhead. Pinecone is ideal if you want to ship fast and don't mind a monthly bill ($0.04-$0.10 per 100 queries, roughly).\n\n**Chroma** (open-source, self-hosted) is free and runs in-memory or on disk. It's great for prototypes and small deployments (<10,000 documents), but you manage scaling and backups. No monthly cost, but you run the server yourself.\n\n**Weaviate** (open-source + managed) is a middle ground: free self-hosted version, or a managed cloud plan. It supports filtering and multi-modal search (text + images). If you need production-grade features without Pinecone's cost, Weaviate is worth evaluating.\n\n**FAISS** (open-source, Facebook) is a bare-metal vector library, not a database. It's extremely fast for similarity search over millions of vectors, but it doesn't handle persistence, filtering, or distributed queries. Use FAISS if you're building a custom search engine with very large-scale data and you have an engineering team to maintain it.\n\nFor a law firm, **Pinecone is the most practical choice** because it handles auth, audit logging, and uptime guarantees - all non-negotiable in regulated industries. If cost is the primary concern and you have in-house DevOps, Weaviate is competitive.\n\nLet's walk through a concrete example: embedding a corpus of California labor law statutes and case summaries, then querying them.\n\nAssume you have three PDFs:\n\n`California_Labor_Code.pdf`\n\n(100 pages)`Wrongful_Termination_Cases_2023.pdf`\n\n(50 pages)`Non_Compete_Agreement_Guide.pdf`\n\n(30 pages)After chunking, you'll have roughly 600-1,000 chunks (at 800 tokens per chunk). Uploading to Pinecone costs ~$1 in OpenAI embeddings (1M tokens * $0.02 / 1M). Then, every search query costs ~$0.05 (embedding the question + a small LLM inference). At 100 queries per month, that's ~$5 in query costs.\n\nA user asks: \"Can a California company enforce a non-compete clause?\"\n\nThe LangChain chain will:\n\nThe end-to-end cost is ~$0.02 per query. Accuracy depends on your chunk quality and metadata tagging.\n\nFor a deeper technical reference, see [OpenAI's docs](https://platform.openai.com/docs).\n\nSet `temperature=0.1`\n\nin your LLM config to keep outputs factual and deterministic. Use a prompt that explicitly says \"Do not infer beyond the provided documents.\" Add a human approval step before any answer is sent to a client - no AI system is 100% accurate in law, and GPT-4's mistakes could be costly.\n\nYes. Open-source models like `Mistral-7B`\n\nor `Llama-2-70B`\n\ncan work, especially if you fine-tune them on legal documents. However, they typically require more prompting to stay on-topic, and they're less familiar with U.S. law than GPT-4. If you self-host them (via Ollama or vLLM), you save on API costs but add infrastructure complexity. For law firms, GPT-4's accuracy usually justifies the cost.\n\nKeyword search looks for exact word matches (\"non-compete\" returns only documents with that word). Semantic search embeds meaning, so \"non-compete clause\" and \"restriction on competition\" return similar results even without exact phrase overlap. For legal documents, semantic search is more powerful because statutes use varied language, but it requires the retriever to be well-trained - hence why you need to test your chunks and prompts carefully.\n\nDon't embed the entire document as one chunk. Split into 500-800 token chunks with 100-token overlap, as shown in Step 1. If documents have clear sections (e.g., \"Article III\", \"Section 2\"), use those as natural split points instead of raw character counts. This preserves semantic boundaries and makes retrieval more precise.\n\nYes. Use Weaviate (check the provider's current pricing and self-hosting options) for a free self-hosted option, or Chroma for rapid prototyping. Both work with LangChain and GPT-4. The tradeoff is infrastructure: Weaviate and Chroma require you to run and scale the vector database, whereas Pinecone is fully managed. For a law firm MVP, start with Chroma locally; migrate to a managed solution (Pinecone or Weaviate Cloud) once you have real users and want production guarantees.\n\nLog every query, result, and source document. Store logs in a database with timestamps and user IDs. Ensure your API is authenticated (use OAuth or API keys). If your documents are confidential (attorney-client privilege), run the system on-premise or in a VPC, and encrypt data in transit and at rest. Consult your firm's legal and compliance team on data handling before going live.\n\nYou now have a working blueprint for semantic search on legal documents. The next stage is tuning for your specific use case: test retrieval accuracy on 20-30 real questions, adjust chunk size and top-k based on results, and add role-based access control if multiple teams will use it.\n\nIf you're building AI automation systems like this for clients, explore the [AI automations you can sell](https://getaab.com/ai-automations-to-sell) guide to understand which automation workflows are most in-demand. And grab [the free guide](https://getaab.com/free) for a deeper dive into RAG pipelines, cost optimization, and avoiding common pitfalls.", "url": "https://wpnews.pro/news/build-semantic-search-for-legal-documents-with-pinecone-and-gpt-4", "canonical_source": "https://dev.to/samchenreviews/build-semantic-search-for-legal-documents-with-pinecone-and-gpt-4-3mde", "published_at": "2026-09-01 16:32:12+00:00", "updated_at": "2026-09-01 16:54:48.938639+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "developer-tools"], "entities": ["Pinecone", "GPT-4", "LangChain", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/build-semantic-search-for-legal-documents-with-pinecone-and-gpt-4", "markdown": "https://wpnews.pro/news/build-semantic-search-for-legal-documents-with-pinecone-and-gpt-4.md", "text": "https://wpnews.pro/news/build-semantic-search-for-legal-documents-with-pinecone-and-gpt-4.txt", "jsonld": "https://wpnews.pro/news/build-semantic-search-for-legal-documents-with-pinecone-and-gpt-4.jsonld"}}