Build Semantic Search for Legal Documents with Pinecone and GPT-4

A developer detailed a method for building semantic search for legal documents using Pinecone, GPT-4, and LangChain. The system enables law firms to search case law and contracts by meaning rather than keywords, using retrieval-augmented generation to provide accurate, cited answers. The prototype can be built in 4-6 hours, with production deployment taking 1-2 weeks.

You can build a semantic search system for legal documents by combining Pinecone a vector database , GPT-4 for intelligent retrieval , and LangChain to orchestrate the pipeline . This setup lets law firms search case law, contracts, and precedents by meaning rather than keyword matching - so a query like "wrongful termination without cause" surfaces relevant statutes even when documents use different phrasing. The result is a retrieval-augmented generation RAG system that feeds the most relevant legal context to GPT-4, so it can cite sources and answer with accuracy. The tech stack below represents a realistic, minimal setup to ship a working legal document semantic search system. Prices are current as of August 2024; verify each provider's current terms before committing. | Tool | Plan / Cost | Role | |---|---|---| | Pinecone cloud vector DB | Check current pricing at | Time to build: 4-6 hours for a working prototype on 100-500 documents; 1-2 weeks to production-grade chunking strategy, fine-tuning relevance, auth, audit logging for legal compliance . Legal documents are too large to embed as a single vector. A 50-page contract or court opinion needs to be split into semantic chunks - typically 300-800 tokens each - so that GPT-4 can retrieve relevant sections without losing context. Start by uploading your documents PDFs, Word files, plaintext to a staging directory. Then use LangChain's document loaders to parse them: python from langchain community.document loaders import PyPDFLoader, UnstructuredPDFLoader from langchain.text splitter import RecursiveCharacterTextSplitter import os Load all PDFs from a directory docs = for filename in os.listdir './legal documents' : if filename.endswith '.pdf' : loader = PyPDFLoader f'./legal documents/{filename}' docs.extend loader.load Split documents into chunks splitter = RecursiveCharacterTextSplitter chunk size=800, chunk overlap=100, separators= "\n\n", "\n", " " chunks = splitter.split documents docs print f"Created {len chunks } chunks from {len docs } documents" This script loads PDFs, splits them into 800-token chunks with 100-token overlap so key phrases aren't cut off , and reports how many chunks you've created. The overlap ensures that a search query at a chunk boundary still finds the right content. Adjust chunk size down to 400 tokens if your documents are dense with cross-references e.g., statutes with many citations ; go up to 1,200 if they're more narrative e.g., case summaries . Why this matters: a chunk too large 5,000+ tokens dilutes semantic relevance across unrelated sections; a chunk too small <300 tokens loses enough context that GPT-4 struggles to answer follow-up questions. Most law firms find 500-800 tokens is the sweet spot. Pinecone is a managed vector database that indexes embeddings for fast similarity search. You'll create an index, then embed each chunk and upsert it with metadata document name, page number, date so you can trace answers back to source. First, install dependencies and authenticate: pip install pinecone-client openai langchain langchain-openai langchain-pinecone Then set your API keys as environment variables: export OPENAI API KEY="your-key-here" export PINECONE API KEY="your-key-here" export PINECONE ENVIRONMENT="us-east-1" check your Pinecone console for your region Initialize Pinecone and create an index: python from pinecone import Pinecone from openai import OpenAI Initialize Pinecone pc = Pinecone api key=os.environ.get "PINECONE API KEY" Create index if it doesn't exist index name = "legal-docs" if index name not in pc.list indexes .names : pc.create index name=index name, dimension=1536, OpenAI's text-embedding-3-small produces 1536-dim vectors metric="cosine", spec={ "serverless": { "cloud": "aws", "region": "us-east-1" } } index = pc.Index index name client = OpenAI api key=os.environ.get "OPENAI API KEY" The dimension is 1536 because OpenAI's text-embedding-3-small model outputs 1536-dimensional vectors. The cosine metric measures similarity 0 = completely different, 1 = identical . Serverless means Pinecone auto-scales without you managing pods. Now embed and upload each chunk: python def embed and upsert chunks, batch size=100 : """Embed chunks and upload to Pinecone.""" vectors to upsert = for i, chunk in enumerate chunks : Get embedding from OpenAI response = client.embeddings.create input=chunk.page content, model="text-embedding-3-small" embedding = response.data 0 .embedding Prepare metadata metadata = { "text": chunk.page content :500 , first 500 chars for display "source": chunk.metadata.get "source", "unknown" , "page": chunk.metadata.get "page", 0 , } vectors to upsert.append f"chunk-{i}", embedding, metadata Upsert in batches if i + 1 % batch size == 0: index.upsert vectors=vectors to upsert print f"Uploaded {i + 1} / {len chunks } chunks" vectors to upsert = Upload remaining vectors if vectors to upsert: index.upsert vectors=vectors to upsert print f"Complete: {len chunks } chunks indexed" embed and upsert chunks This batches uploads in groups of 100 to avoid timeout errors. Each vector is stored with metadata so you can show the user which document a result came from. This step typically costs $20-$60 for 10,000 chunks depending on document size , because you pay OpenAI for embeddings. Now that embeddings are in Pinecone, you'll chain retrieval + GPT-4 reasoning together. When a user asks a question, LangChain will: python from langchain openai import OpenAIEmbeddings, ChatOpenAI from langchain pinecone import PineconeVectorStore from langchain.chains import RetrievalQA from langchain.prompts import PromptTemplate Initialize embeddings and vector store embeddings = OpenAIEmbeddings model="text-embedding-3-small" vectorstore = PineconeVectorStore index=index, embedding=embeddings Create a retriever k=4 means return top 4 most similar chunks retriever = vectorstore.as retriever search kwargs={"k": 4} Define a custom prompt for legal context legal prompt = PromptTemplate input variables= "context", "question" , template="""You are a legal research assistant. Use the following context from legal documents to answer the question. If the answer is not in the context, say 'Not found in provided documents.' Always cite the source document. Context: {context} Question: {question} Answer:""" Create the QA chain llm = ChatOpenAI model="gpt-4", temperature=0.1 low temperature for legal accuracy qa chain = RetrievalQA.from chain type llm=llm, chain type="stuff", retriever=retriever, chain type kwargs={"prompt": legal prompt}, return source documents=True Test the chain question = "What are the grounds for wrongful termination in California?" response = qa chain.invoke {"query": question} print f"Answer: {response 'result' }" print f"Sources: { doc.metadata 'source' for doc in response 'source documents' }" The temperature=0.1 setting keeps GPT-4 focused and factual - appropriate for legal work where hallucination is costly. The stuff chain type concatenates all retrieved documents and sends them to the LLM at once fine for 4 chunks; for larger contexts, use map reduce or refine mode . The return source documents=True flag ensures users can verify where answers came from. Legal document search often needs filtering by jurisdiction, case type, or date. Extend the retriever to support metadata-aware searches: python from langchain.retrievers.self query.base import SelfQueryRetriever from langchain.chains.query constructor.base import AttributeInfo Define filterable metadata fields metadata field info = AttributeInfo name="source", description="The file name or document title e.g., 'California Labor Code.pdf' ", type="string", , AttributeInfo name="page", description="Page number in the original document", type="integer", , AttributeInfo name="jurisdiction", description="State or country e.g., 'California', 'Federal' ", type="string", , Create a self-querying retriever that can filter retriever = SelfQueryRetriever.from llm llm=llm, vectorstore=vectorstore, document contents="Legal documents: statutes, case law, contracts", metadata field info=metadata field info, verbose=True Query with implicit filtering question = "What does California labor law say about at-will employment?" response = qa chain.invoke {"query": question} The self-querying retriever uses GPT-4 to parse the user's question and automatically extract filters. So "California labor law" is interpreted as a filter on jurisdiction='California' plus a semantic search for "at-will employment." This saves law firms from writing complex filter syntax. Wrap the QA chain in a FastAPI server so law firm teams can query via web or integration: python from fastapi import FastAPI from pydantic import BaseModel import uvicorn app = FastAPI title="Legal Document Search" class QueryRequest BaseModel : question: str top k: int = 4 class QueryResponse BaseModel : answer: str sources: list str @app.post "/search" async def search request: QueryRequest : """Search legal documents and return answer with sources.""" response = qa chain.invoke { "query": request.question, "k": request.top k } return QueryResponse answer=response "result" , sources= doc.metadata "source" for doc in response "source documents" @app.get "/health" async def health : return {"status": "ok"} if name == " main ": uvicorn.run app, host="0.0.0.0", port=8000 Run it locally with uvicorn app:app --reload , then test with curl: curl -X POST http://localhost:8000/search \ -H "Content-Type: application/json" \ -d '{"question": "What are non-compete clause limits in California?", "top k": 5}' Deploy to a cloud platform AWS Lambda, Google Cloud Run, or Heroku for production. Add authentication API keys or OAuth if multiple users will access it. Building semantic search for legal documents introduces several real failure modes. Here's how to handle each: Rate limits and embedding costs. OpenAI's API has rate limits 3,500 requests per minute on the free tier, higher on paid accounts . If you're embedding thousands of documents at once, you'll hit these limits. Fix: batch your embedding jobs across multiple hours, or use exponential backoff with a sleep counter between API calls. Also monitor your embedding costs - at ~$0.02 per 1M tokens, 100,000 legal documents 50 pages average can cost $100-$500 to embed. Budget for this upfront. Stale or irrelevant retrieval. If your top-k retrieved chunks don't actually answer the question, GPT-4 will admit it or make an inference that sounds plausible but is wrong. This happens when your chunk size is too large or too small, or when your documents have inconsistent formatting. Fix: test your retriever on 10-20 real legal questions before going live. If top results are off-topic, lower chunk size to 500 tokens or adjust your prompt to be more strict "Answer only from the provided context" . Pinecone costs scaling unexpectedly. Pinecone charges per query for serverless mode. If you have 1,000 law firm users running 10 searches a day, that's 10,000 queries per day. Check Pinecone's current pricing page to estimate monthly cost. Fix: cache frequently asked questions e.g., "What is California wrongful termination law?" so you don't re-query for identical questions. Use a Redis cache layer in front of your API. Token limits in context. GPT-4 has an 8,192-token context window or 128,000 for GPT-4 Turbo . If you retrieve 10 chunks and each is 800 tokens, you're at 8,000 tokens before the user's question even arrives. Fix: limit k to 3-5 chunks, or use a reranker a smaller, faster model that re-scores retrieved chunks for relevance before sending to GPT-4 . Missing or inconsistent metadata. If PDFs don't have consistent metadata page numbers, dates, author , your citations will be vague. Fix: add a preprocessing step that extracts and validates metadata from PDF headers, or manually tag documents before uploading. Cold-start latency. The first query after deployment can take 5-10 seconds because Pinecone needs to warm up. Users expect <1 second. Fix: warm up Pinecone with a dummy query on startup, or accept the cold-start and communicate it to users. If you're building semantic search for legal documents, you have multiple vector database options. Here's how they compare: Pinecone serverless, managed is easiest for production because it's hosted and scales automatically. You pay per query and storage, with no infrastructure overhead. Pinecone is ideal if you want to ship fast and don't mind a monthly bill $0.04-$0.10 per 100 queries, roughly . Chroma open-source, self-hosted is free and runs in-memory or on disk. It's great for prototypes and small deployments <10,000 documents , but you manage scaling and backups. No monthly cost, but you run the server yourself. Weaviate open-source + managed is a middle ground: free self-hosted version, or a managed cloud plan. It supports filtering and multi-modal search text + images . If you need production-grade features without Pinecone's cost, Weaviate is worth evaluating. FAISS open-source, Facebook is a bare-metal vector library, not a database. It's extremely fast for similarity search over millions of vectors, but it doesn't handle persistence, filtering, or distributed queries. Use FAISS if you're building a custom search engine with very large-scale data and you have an engineering team to maintain it. For a law firm, Pinecone is the most practical choice because it handles auth, audit logging, and uptime guarantees - all non-negotiable in regulated industries. If cost is the primary concern and you have in-house DevOps, Weaviate is competitive. Let's walk through a concrete example: embedding a corpus of California labor law statutes and case summaries, then querying them. Assume you have three PDFs: California Labor Code.pdf 100 pages Wrongful Termination Cases 2023.pdf 50 pages Non Compete Agreement Guide.pdf 30 pages After chunking, you'll have roughly 600-1,000 chunks at 800 tokens per chunk . Uploading to Pinecone costs ~$1 in OpenAI embeddings 1M tokens $0.02 / 1M . Then, every search query costs ~$0.05 embedding the question + a small LLM inference . At 100 queries per month, that's ~$5 in query costs. A user asks: "Can a California company enforce a non-compete clause?" The LangChain chain will: The end-to-end cost is ~$0.02 per query. Accuracy depends on your chunk quality and metadata tagging. For a deeper technical reference, see OpenAI's docs https://platform.openai.com/docs . Set temperature=0.1 in your LLM config to keep outputs factual and deterministic. Use a prompt that explicitly says "Do not infer beyond the provided documents." Add a human approval step before any answer is sent to a client - no AI system is 100% accurate in law, and GPT-4's mistakes could be costly. Yes. Open-source models like Mistral-7B or Llama-2-70B can work, especially if you fine-tune them on legal documents. However, they typically require more prompting to stay on-topic, and they're less familiar with U.S. law than GPT-4. If you self-host them via Ollama or vLLM , you save on API costs but add infrastructure complexity. For law firms, GPT-4's accuracy usually justifies the cost. Keyword search looks for exact word matches "non-compete" returns only documents with that word . Semantic search embeds meaning, so "non-compete clause" and "restriction on competition" return similar results even without exact phrase overlap. For legal documents, semantic search is more powerful because statutes use varied language, but it requires the retriever to be well-trained - hence why you need to test your chunks and prompts carefully. Don't embed the entire document as one chunk. Split into 500-800 token chunks with 100-token overlap, as shown in Step 1. If documents have clear sections e.g., "Article III", "Section 2" , use those as natural split points instead of raw character counts. This preserves semantic boundaries and makes retrieval more precise. Yes. Use Weaviate check the provider's current pricing and self-hosting options for a free self-hosted option, or Chroma for rapid prototyping. Both work with LangChain and GPT-4. The tradeoff is infrastructure: Weaviate and Chroma require you to run and scale the vector database, whereas Pinecone is fully managed. For a law firm MVP, start with Chroma locally; migrate to a managed solution Pinecone or Weaviate Cloud once you have real users and want production guarantees. Log every query, result, and source document. Store logs in a database with timestamps and user IDs. Ensure your API is authenticated use OAuth or API keys . If your documents are confidential attorney-client privilege , run the system on-premise or in a VPC, and encrypt data in transit and at rest. Consult your firm's legal and compliance team on data handling before going live. You now have a working blueprint for semantic search on legal documents. The next stage is tuning for your specific use case: test retrieval accuracy on 20-30 real questions, adjust chunk size and top-k based on results, and add role-based access control if multiple teams will use it. If you're building AI automation systems like this for clients, explore the AI automations you can sell https://getaab.com/ai-automations-to-sell guide to understand which automation workflows are most in-demand. And grab the free guide https://getaab.com/free for a deeper dive into RAG pipelines, cost optimization, and avoiding common pitfalls.