Building RAG Applications with Spring AI: Connect LLMs to Your Own Data

A developer demonstrates building a Retrieval-Augmented Generation (RAG) application using Spring AI, connecting large language models to private data sources. The post outlines the RAG pipeline, including document loading, chunking, embeddings, vector databases, and similarity search, with code examples in Java.

Large Language Models are powerful. But there is one fundamental limitation: An LLM doesn't automatically know your application's private data. Your company policies, product documentation, internal knowledge base, customer records, PDFs, technical documentation, or database content are not necessarily part of the model's training data. This is where RAG — Retrieval-Augmented Generation comes in. Instead of asking an LLM to answer directly, we first retrieve relevant information from our own data and provide that information as context to the model. In this article, we'll build the foundation of a RAG application using Spring AI . RAG stands for: Retrieval-Augmented Generation The idea is simple: User Question ↓ Retrieve Relevant Information ↓ Add Retrieved Context to Prompt ↓ LLM ↓ Generated Answer For example, imagine we have a company's internal documentation. A user asks: What is our refund policy for annual subscriptions? Instead of expecting the LLM to magically know the answer, our application: This is the core idea behind RAG. A normal LLM application looks like this: User ↓ Application ↓ LLM ↓ Answer The problem? The LLM only has access to the information available to it. With RAG, the architecture becomes: ┌──────────────┐ │ Documents │ └──────┬───────┘ ↓ Chunking ↓ Embeddings ↓ Vector Database ↑ │ User → Query → Similarity Search ↓ Relevant Context ↓ LLM ↓ Answer Now the model can work with information from our own knowledge base. A production RAG pipeline typically contains these stages: Documents ↓ Document Loading ↓ Chunking ↓ Embeddings ↓ Vector Database ↓ Similarity Search ↓ Relevant Context ↓ Prompt ↓ LLM ↓ Answer Let's understand each step. First, we need to get our data into the application. The source could be: Spring AI provides abstractions for working with documents. A document can be represented using Spring AI's Document abstraction. Conceptually: Document document = new Document "Spring Boot is a framework for building Java applications..." ; We now have content that can be processed by our RAG pipeline. We shouldn't usually store an entire document as a single vector. Imagine a 100-page PDF. A user asks: How do I configure authentication? We don't want to retrieve the entire PDF. Instead, we split the document into smaller pieces called chunks . For example: Document ↓ Chunk 1 Chunk 2 Chunk 3 Chunk 4 ... Chunk 100 Each chunk represents a smaller piece of knowledge. A simple example: Chunk 1: Introduction to Spring Security Chunk 2: Configuring authentication Chunk 3: Creating users Chunk 4: JWT authentication Chunk 5: Role-based authorization Now when the user asks about JWT authentication, we can retrieve the relevant chunk instead of the entire document. This is where things get interesting. A computer doesn't understand the semantic meaning of text in the same way humans do. We need a way to represent text numerically. That's where embeddings come in. An embedding model converts text into a vector. For example: "How do I configure JWT authentication?" might become something conceptually like: 0.12, -0.42, 0.87, 0.31, ... The actual vector contains many dimensions. The important part is: Semantically similar text produces vectors that are relatively close together in vector space. For example: "How can I configure JWT?" ↓ 0.12, 0.81, 0.42, ... "JWT authentication configuration" ↓ 0.15, 0.78, 0.45, ... These vectors should have high similarity. Now we need somewhere to store these embeddings. That's where a vector database comes in. Popular choices include: For a Spring Boot application, PostgreSQL with pgvector is an especially interesting option because you can keep your relational data and vector data within the same ecosystem. Conceptually: Document Chunk ↓ Embedding Model ↓ Vector ↓ Vector Database Our database might contain something conceptually like: ID | Content | Embedding ---|----------------------------|---------------- 1 | JWT configuration... | 0.12,... 2 | OAuth2 configuration... | 0.42,... 3 | Database configuration... | 0.71,... Now suppose the user asks: How do I configure JWT authentication in Spring Boot? We first generate an embedding for the question. User Query ↓ Embedding Model ↓ Query Vector Then we search the vector database for similar vectors. Query Vector ↓ Vector Database ↓ Similarity Search ↓ Top K Relevant Chunks For example: Result 1 → JWT configuration Result 2 → Spring Security authentication Result 3 → SecurityFilterChain configuration These results become our context . Now we combine: For example: Context: Spring Security can be configured using SecurityFilterChain. JWT authentication can be implemented using a custom authentication filter... Question: How do I configure JWT authentication in Spring Boot? The LLM receives this information and generates the answer. This is the Augmented part of Retrieval-Augmented Generation. The final flow looks like: User Question ↓ Embedding ↓ Vector Search ↓ Relevant Documents ↓ Prompt + Context ↓ LLM ↓ Answer The LLM isn't searching the database itself. Our application retrieves the information first and provides it to the model. Now let's look at how Spring AI simplifies this architecture. A typical Spring AI RAG application contains: Spring Boot │ ├── Document Reader │ ├── Text Splitter │ ├── Embedding Model │ ├── Vector Store │ └── Chat Model The exact model and vector store can be swapped without rewriting the entire application. That's one of the strengths of Spring AI's abstraction-based approach. Let's create a Spring Boot project. We'll need Spring AI dependencies for: For example, with Maven: <dependency <groupId org.springframework.ai</groupId <artifactId spring-ai-starter-model-bedrock-converse</artifactId </dependency <dependency <groupId org.springframework.ai</groupId <artifactId spring-ai-starter-vector-store-pgvector</artifactId </dependency You should use versions compatible with the Spring AI version used by your project. If we're using AWS Bedrock, our application needs AWS credentials and model configuration. Conceptually: spring: ai: bedrock: aws: region: us-east-1 The exact configuration depends on the Spring AI version and Bedrock model you're using. For production environments, don't hard-code AWS credentials inside application.yml . Use: Environment Variables ↓ IAM Roles ↓ AWS Credentials Provider Chain Whenever possible, prefer IAM roles over static credentials. For PostgreSQL + pgvector, the architecture looks like: Spring Boot ↓ Spring AI VectorStore ↓ PostgreSQL ↓ pgvector The vector store becomes the bridge between our application and the vector database. Conceptually: VectorStore vectorStore; We can then add documents: vectorStore.add documents ; And later search: List<Document results = vectorStore.similaritySearch SearchRequest.builder .query question .topK 5 .build ; The important abstraction here is: VectorStore Our application doesn't need to manually implement vector similarity calculations. Let's say we have a PDF containing product documentation. Spring AI provides document readers that can load different document formats. Conceptually: js var documents = reader.get ; We then split the documents into chunks. For example: TokenTextSplitter splitter = new TokenTextSplitter ; List<Document chunks = splitter.apply documents ; The exact splitter and configuration should be chosen based on your document structure and model context window. The embedding model converts every chunk into a vector. Conceptually: Document Chunk ↓ Embedding Model ↓ Vector Spring AI handles this through the embedding/vector-store integration. Then we can store the documents: vectorStore.add chunks ; Our ingestion pipeline is now: PDF ↓ Documents ↓ Chunks ↓ Embeddings ↓ Vector Store Now let's handle the user's question. Suppose the user asks: How does authentication work in our application? We search the vector store: List<Document results = vectorStore.similaritySearch SearchRequest.builder .query question .topK 5 .build ; We now have the most relevant pieces of information. For example: Result 1 JWT authentication uses... Result 2 The authentication filter... Result 3 Security configuration... We can now construct a prompt using the retrieved documents. For example: String context = results.stream .map Document::getText .collect Collectors.joining "\n\n" ; String prompt = """ Answer the question using only the provided context. Context: %s Question: %s """.formatted context, question ; Then send it to the chat model. Conceptually: ChatResponse response = chatModel.call new Prompt prompt ; And we return: response.getResult .getOutput .getText ; That's a basic RAG implementation. Putting everything together: INGESTION PIPELINE ┌───────────────┐ │ Documents │ └───────┬───────┘ ↓ Text Splitting ↓ Embeddings ↓ Vector Store │ │ │ ▼ RETRIEVAL ▲ │ User Question ↓ Embedding ↓ Similarity Search ↓ Top K Chunks ↓ Context ↓ Prompt ↓ LLM ↓ Answer This is the basic architecture behind many modern knowledge-based AI applications. One of the most underestimated parts of RAG is chunking. Bad chunking can produce bad retrieval. Imagine this document: Spring Security Authentication allows the application to verify the identity of a user. Authorization determines whether the authenticated user has permission to access a resource. If we split this badly: Chunk 1: Authentication allows... Chunk 2: the identity of a user. Authorization determines... Chunk 3: whether the authenticated user... we may destroy important semantic relationships. A better strategy is to preserve meaningful boundaries where possible. Depending on the data, you may experiment with: Chunk Size Overlap Sentence boundaries Paragraph boundaries Markdown headings Semantic sections There is no universal chunk size that works for every RAG application. When performing similarity search, we usually retrieve the top K results. For example: .topK 5 means: Return the 5 most relevant chunks. But bigger isn't always better. If we retrieve too little: K = 1 we may miss important context. If we retrieve too much: K = 50 we may introduce irrelevant information and increase token usage. A common approach is to start with a small value and evaluate retrieval quality. For example: K = 3 K = 5 K = 10 Then measure which configuration works best for your dataset. A common question is: Why not fine-tune the model instead? RAG and fine-tuning solve different problems. Best when: Useful when you want to change: A useful mental model is: RAG → Give the model the right information. Fine-tuning → Change how the model behaves. In many production systems, they can also be used together. A basic RAG pipeline is only the beginning. Production RAG systems often introduce additional stages: Query ↓ Query Transformation ↓ Hybrid Retrieval ↓ Metadata Filtering ↓ Vector Search ↓ Reranking ↓ Context Compression ↓ Prompt Construction ↓ LLM You might eventually introduce techniques such as: This is where RAG becomes an engineering discipline rather than simply "put documents into a vector database." If you're building RAG for production, don't stop at: PDF → Vector DB → LLM You also need to think about: How frequently are documents updated? New document ↓ Process ↓ Chunk ↓ Embed ↓ Update Vector Store Store useful metadata alongside chunks: document id source page number tenant id created at updated at document type This becomes extremely useful for filtering. For example: tenant id = "company-123" can ensure that users only retrieve documents belonging to their tenant. This is critical. A RAG system must not retrieve documents that the current user isn't authorized to access. Your retrieval layer should respect application permissions. User ↓ Authentication ↓ Authorization ↓ Metadata Filters ↓ Retrieval ↓ LLM Never assume that because the LLM can't "see" a document directly, the document is secure. Track things such as: Retrieval latency Embedding latency LLM latency Token usage Retrieved chunks Similarity scores Failure rate Answer quality Without observability, debugging RAG becomes extremely difficult. If you're new to RAG, remember this: Embedding Model = "Convert meaning into numbers" Vector Database = "Store and search those meanings" Retriever = "Find relevant information" LLM = "Use that information to generate an answer" Together: Retrieve → Augment → Generate That's RAG. Spring AI gives Java developers abstractions around many of these building blocks. Instead of manually wiring every AI provider and vector database integration, we can work with abstractions such as: ChatModel EmbeddingModel VectorStore Document DocumentReader This allows us to focus more on the application architecture rather than provider-specific implementation details. And that's particularly useful when building enterprise Java applications where we may want to change: AWS Bedrock ↓ Another Model Provider without completely rewriting our application. A production-oriented Spring AI RAG application can eventually look like this: ┌───────────────────┐ │ User / App │ └─────────┬─────────┘ ↓ Spring Boot API ↓ Query Processing ↓ Retrieval Layer ↓ ┌───────────────────┐ │ Vector Store │ │ PostgreSQL │ │ + pgvector │ └─────────┬─────────┘ ↓ Relevant Context ↓ Prompt Assembly ↓ ┌───────────────────┐ │ Spring AI │ └─────────┬─────────┘ ↓ ┌───────────────────┐ │ AWS Bedrock │ └─────────┬─────────┘ ↓ Answer And separately: Documents ↓ Document Reader ↓ Chunking ↓ Embedding Model ↓ PostgreSQL + pgvector We started this series by exploring how Spring AI can connect Java applications with modern AI models. With AWS Bedrock, we can access powerful foundation models without managing the underlying model infrastructure. Now, with RAG, we can take another major step: We can connect those models to our own data. The journey now looks like: LLM ↓ Spring AI ↓ AWS Bedrock ↓ RAG ↓ Vector Database ↓ Our Own Data But there is another important capability missing. What if we don't just want the model to answer questions ? What if we want the model to take actions ? For example: User ↓ AI Agent ↓ Decide what to do ↓ Call a Tool ↓ Execute Action ↓ Return Result That's where tool calling and AI agents come in. Next up: Building AI Agents with Spring AI — Tool Calling, Memory, and Autonomous Workflows. If you're building AI applications with Java and Spring Boot, RAG is one of the most important patterns to understand.