I Built a Simple RAG App with LangChain, OpenAI, and Pinecone

A developer built a simple Retrieval-Augmented Generation (RAG) application using LangChain, OpenAI embeddings, and Pinecone. The app ingests text documents by splitting them into chunks, converting them to embeddings, and storing them in Pinecone. When a user asks a question, the app retrieves relevant chunks and feeds them to an AI model to generate grounded answers.

Large language models know a lot, but they do not automatically know the contents of our private files, company documents, notes, or recently written articles. This is where Retrieval-Augmented Generation, commonly called RAG, becomes useful. In this project, I built a small RAG application that: Reads a text document. Breaks the document into smaller sections. Converts those sections into numerical representations called embeddings. Stores the embeddings in Pinecone. Finds the most relevant sections when a user asks a question. Gives those sections to an AI model so it can produce a grounded answer. A grounded answer is an answer based on the retrieved source material. RAG reduces unsupported answers, but it does not guarantee that every response will be correct I used: Python for the application LangChain to connect the different steps OpenAI embeddings to represent text as numbers Pinecone to store and search those embeddings uv to manage the Python project and dependencies In this article, I will explain how the application works, how the different components fit together, and what I learned while debugging it. What Is RAG? Press enter or click to view image in full size RAG Pipeline Imagine that you are taking an open-book examination. Without a book, you answer only from memory. You may know the answer, but you may also forget something or provide incorrect information. Press enter or click to view image in full size Debugging using LangSmith With an open book, you can: Read the question. Search for the relevant page. Use that information to write the answer. A RAG application works in a similar way. The language model is the person answering the question. The vector database acts like a searchable book. Before answering, the application finds the most relevant information and gives it to the model. The model does not need to memorize the entire document. It only receives the sections that are likely to contain the answer. The Two Main Parts of a RAG Application A basic RAG application has two phases. Phase 1: Ingestion Ingestion means preparing the documents so that they can be searched later. Text document ↓ Load the document ↓ Split it into smaller chunks ↓ Convert each chunk into an embedding ↓ Store the embeddings in Pinecone This process usually runs when documents are added or updated. Phase 2: Retrieval and Answer Generation This phase runs whenever a user asks a question. User question ↓ Convert the question into an embedding ↓ Search Pinecone for similar document chunks ↓ Add those chunks to the prompt ↓ Ask the language model to answer Keeping these phases separate makes the application easier to understand. Ingestion prepares the knowledge. Retrieval uses that knowledge. What Is an Embedding? Press enter or click to view image in full size Computers cannot directly compare the meaning of sentences in the same way people do. An embedding model converts text into a list of numbers called a vector. For example: "How do I reset my password?" might become something that looks like: 0.12, -0.48, 0.77, 0.03, ... The actual vector contains hundreds or thousands of numbers. Sentences with similar meanings usually receive vectors that are close to each other. This allows the application to search by meaning rather than only by exact keywords. Press enter or click to view image in full size How embeddings work in real. LangChain provides a standard interface for embedding models, while OpenAI’s text-embedding-3 models allow developers to control the number of dimensions returned. What Is Pinecone? Press enter or click to view image in full size Pinecone is a vector database. A normal database is excellent at questions such as: Find the user whose ID is 123. A vector database is useful for questions such as: Find the document sections whose meaning is closest to this question. Pinecone stores the document vectors and searches for the vectors that are closest to the user’s question. The Pinecone index and the embedding model must use the same vector dimension. Otherwise, Pinecone cannot store or compare the vectors. Project Structure Now that the basic idea is clear, let us look at how the application is structured and how each part of the pipeline works. My project contains two main Python files: rag-project/ ├── rag-gist/ │ ├── ingest.py │ ├── main.py │ └── mediumblog.txt ├── .env ├── .gitignore └── pyproject.toml Here is what each file does: mediumblog.txt contains the source information. ingest.py loads and stores the document in Pinecone. main.py accepts a question and generates an answer. .env stores API keys. .gitignore prevents secrets from being committed. pyproject.toml contains the project dependencies. The project contains separate scripts for document ingestion and question answering. Step 1: Install the Dependencies I used uv to manage the project. The required packages can be installed with: uv add langchain \ langchain-community \ langchain-openai \ langchain-pinecone \ langchain-text-splitters \ python-dotenv The main packages have different responsibilities: langchain-openai connects LangChain to OpenAI models. langchain-pinecone connects LangChain to Pinecone. langchain-community provides document loaders. langchain-text-splitters provides tools for dividing documents into chunks. python-dotenv loads environment variables from a .env file. LLM libraries change quickly, so it is better to keep the generated lock file in the repository instead of relying only on version numbers written in an article. Step 2: Add the API Keys Create a .env file in the project root: OPENAI API KEY=your openai api key PINECONE API KEY=your pinecone api key Never upload this file to GitHub. Add it to .gitignore: .env .venv/ pycache / You can also create a safe example file called .env.example: OPENAI API KEY= PINECONE API KEY= This tells other developers which variables are required without exposing the real values. Step 3: Create the Pinecone Index Before running the ingestion script, create a dense Pinecone index. Pinecone does not generate the answer. Its job is to find the document chunks that are most relevant to the user’s question. For this project, I used: Index name: medium-blogs-embedding-index Dimension: 512 Metric: cosine The dimension is important. The embedding model and the Pinecone index must use the same number of dimensions. Otherwise, Pinecone cannot store or compare the vectors. OpenAI embeddings can be used for semantic search by comparing the similarity between document and query vectors. Press enter or click to view image in full size The Pinecone index uses 512 dimensions, matching the embedding configuration in the Python code Step 4: Load and Store the Document The ingestion script reads the text file, divides it into chunks, converts each chunk into an embedding, and stores the result in Pinecone. Here is the ingestion script: from pathlib import Path from dotenv import load dotenv from langchain community.document loaders import TextLoader from langchain openai import OpenAIEmbeddings from langchain pinecone import PineconeVectorStore from langchain text splitters import CharacterTextSplitter load dotenv INDEX NAME = "medium-blogs-embedding-index" EMBEDDING MODEL = "text-embedding-3-small" EMBEDDING DIMENSION = 512 def main - None: print "Ingesting..." file path = Path file .with name "mediumblog.txt" if not file path.exists : raise FileNotFoundError f"Knowledge file was not found: {file path}" print "Loading document..." loader = TextLoader str file path , encoding="utf-8" documents = loader.load text splitter = CharacterTextSplitter chunk size=1000, chunk overlap=0, chunks = text splitter.split documents documents print f"Created {len chunks } text chunks" embeddings = OpenAIEmbeddings model=EMBEDDING MODEL, dimensions=EMBEDDING DIMENSION, PineconeVectorStore.from documents documents=chunks, embedding=embeddings, index name=INDEX NAME, print "Ingestion complete." if name == " main ": main Run the script with: uv run python rag-gist/ingest.py The output should look similar to this: Ingesting... Loading document... Created 31 text chunks Ingestion complete. Press enter or click to view image in full size The ingestion script loaded the document, created 31 chunks, and stored them in Pinecone. Understanding the Ingestion Code Let us look at what the script is doing. Loading the Document loader = TextLoader str file path , encoding="utf-8" documents = loader.load The loader turns the text file into LangChain Document objects. A document usually contains: page content: the actual text metadata: information such as the source filename Splitting the Document text splitter = CharacterTextSplitter chunk size=1000, chunk overlap=0, Instead of embedding the entire document as one large block, we divide it into smaller chunks. Press enter or click to view image in full size Chunking is essential for context window : and token optimization. Instead of sending the entire document to the language model, the application sends only the sections that are most relevant to the question. This is important because a user’s question may only be related to one small part of the document. A chunk size of 1000 means that each section contains approximately 1,000 characters. For this first version, I used no overlap. In a stronger version, I would add some overlap so that information near the end of one chunk is also available at the beginning of the next chunk. Creating Embeddings embeddings = OpenAIEmbeddings model="text-embedding-3-small", dimensions=512, This code tells OpenAI to return embeddings with 512 dimensions. The dimensions option is supported by OpenAI’s text-embedding-3 family of models. Storing the Chunks PineconeVectorStore.from documents documents=chunks, embedding=embeddings, index name=INDEX NAME, This method: Reads the text from each document chunk. Sends the text to the embedding model. Receives a vector for each chunk. Stores the vector and document metadata in Pinecone. Step 5: Retrieve Information and Generate an Answer After ingestion is complete, we can ask questions. Become a Medium member Here is the retrieval script: from dotenv import load dotenv from langchain core.prompts import ChatPromptTemplate from langchain openai import ChatOpenAI, OpenAIEmbeddings from langchain pinecone import PineconeVectorStore load dotenv INDEX NAME = "medium-blogs-embedding-index" EMBEDDING MODEL = "text-embedding-3-small" EMBEDDING DIMENSION = 512 embeddings = OpenAIEmbeddings model=EMBEDDING MODEL, dimensions=EMBEDDING DIMENSION, llm = ChatOpenAI model="gpt-4o-mini", temperature=0, vectorstore = PineconeVectorStore index name=INDEX NAME, embedding=embeddings, retriever = vectorstore.as retriever search kwargs={"k": 5} prompt template = ChatPromptTemplate.from template """ You are a helpful assistant. Answer the question using only the context provided below. If the answer is not present in the context, say: "I could not find that information in the provided document." Context: {context} Question: {question} Answer: """ def format documents documents - str: return "\n\n".join document.page content for document in documents def answer question question: str - str: documents = retriever.invoke question context = format documents documents messages = prompt template.format messages context=context, question=question, response = llm.invoke messages return response.content if name == " main ": user question = input "Ask a question: " .strip if not user question: raise ValueError "The question cannot be empty." answer = answer question user question print "\nAnswer:\n" print answer Run it with: uv run python rag-gist/main.py The program asks for a question: Ask a question: What is the main idea of the document? It then searches Pinecone and sends the most relevant sections to the language model. Press enter or click to view image in full size Query for simple RAG System Caption: The application retrieved relevant document sections and used them to generate an answer. Why Use a Retriever? The vector store handles storage and similarity search. A retriever provides a simpler interface for requesting relevant documents: retriever = vectorstore.as retriever search kwargs={"k": 5} The value k=5 means: Return the five document chunks that are most closely related to the question. When this line runs: documents = retriever.invoke question the application: Converts the question into an embedding, Searches Pinecone, Finds the closest vectors. Returns their original text. How the Prompt Reduces Unsupported Answers The prompt contains an important instruction: Answer the question using only the context provided below. It also tells the model what to do when the answer is missing: If the answer is not present in the context, say: "I could not find that information in the provided document." This does not guarantee perfect accuracy, but it gives the model a clear boundary. A RAG system should not only retrieve information. It should also tell the model how that information must be used. What Is a LangChain Runnable? LangChain uses a common interface called a Runnable. A runnable is something that accepts input and produces output. For example: documents = retriever.invoke question and: response = llm.invoke messages Both the retriever and the language model support .invoke . That common interface makes it easier to connect prompts, retrievers, models, and custom functions into larger chains. Problems I Encountered The most useful part of building this project was understanding the errors. Index 'mediumblog' not found. Did you mean: medium-blogs-embedding-index The name in the Python script did not match the name in Pinecone. I fixed the constant: INDEX NAME = "medium-blogs-embedding-index" A useful improvement is to keep shared settings in one configuration file so that ingestion and retrieval cannot accidentally use different index names. Vector dimension 1536 does not match the dimension of the index 512 The Pinecone index expected vectors containing 512 values, while the application was sending vectors of a different size. I fixed it by explicitly setting: dimensions=512 The same embedding model and dimension must be used during both ingestion and retrieval. PineconeVectorStore index name=INDEX NAME, embeddings=embeddings, However, the constructor expected embedding, singular: PineconeVectorStore index name=INDEX NAME, embedding=embeddings, Current LangChain-Pinecone examples also initialize the vector store with an embedding object. This was a small naming difference, but it prevented the application from starting. {question} But the code passed: query=query LangChain raised: KeyError: 'question' The variable name passed to the template must exactly match the placeholder: messages = prompt template.format messages context=context, question=query, This error also reminded me to keep naming consistent throughout the application. What I Would Improve Next This project is a good learning version, but it is not yet a production-ready RAG system. My next improvements would be: Use a Better Text Splitter I would replace CharacterTextSplitter with RecursiveCharacterTextSplitter. It tries to keep paragraphs and sentences together instead of cutting the text only by character count. Add Chunk Overlap For example: chunk size=1000 chunk overlap=150 The overlap helps preserve information that appears near chunk boundaries. Prevent Duplicate Ingestion Running the ingestion script repeatedly can store the same content more than once. A production version should create stable document IDs and update existing records instead of always inserting new ones. Return Source Information The answer should show which chunks or source files were used. For example: Source: mediumblog.txt, chunk 8 This would make the application easier to trust and debug. Add a Relevance Threshold The application currently returns the top five chunks even when none of them is strongly related to the question. A similarity threshold could help reject unrelated results. Add Logging and Tracing Tracing tools can show: which question was asked which chunks were retrieved how long each step took how many tokens were used what answer was generated Create an Evaluation Set I would prepare a small collection of: test questions expected sources expected answer points This would make it easier to compare changes to chunk size, retrieval count, prompts, and embedding models. What I Learned Before building this project, RAG looked like one complicated AI feature. After implementing it manually, I understood that it is simply a pipeline: Documents become vectors. Questions become vectors. Vector search finds related text. The language model turns that text into an answer. The difficult part is not calling one magical RAG function. The important decisions are: how the document is divided, which embedding model is used whether vector dimensions match how many chunks are retrieved how irrelevant results are handled what instructions are given to the language model how sources are shown to the user One of the most useful lessons from this project was to inspect the retrieved chunks before changing the prompt. If the correct information is not retrieved, the language model cannot use it, no matter how well the prompt is written. Building each step separately made the architecture much easier to understand. Final Thoughts A small RAG application is a useful project for learning how modern AI applications work with external knowledge. The best way to learn RAG is to begin with one document, one question, and one successful retrieval. It teaches more than prompt writing. It introduces document processing, embeddings, vector search, retrieval, prompt design, configuration management, and debugging. The first version does not need to be perfect. Start with one document, one ingestion script, one retrieval script, and one question. Once the basic pipeline works reliably, improve one part at a time and observe how each change affects retrieval quality.