Large Language Models are powerful.
But there is one fundamental limitation:
An LLM doesn't automatically know your application's private data.
Your company policies, product documentation, internal knowledge base, customer records, PDFs, technical documentation, or database content are not necessarily part of the model's training data.
This is where RAG — Retrieval-Augmented Generation comes in.
Instead of asking an LLM to answer directly, we first retrieve relevant information from our own data and provide that information as context to the model.
In this article, we'll build the foundation of a RAG application using Spring AI.
RAG stands for:
Retrieval-Augmented Generation
The idea is simple:
User Question
↓
Retrieve Relevant Information
↓
Add Retrieved Context to Prompt
↓
LLM
↓
Generated Answer
For example, imagine we have a company's internal documentation.
A user asks:
What is our refund policy for annual subscriptions?
Instead of expecting the LLM to magically know the answer, our application:
This is the core idea behind RAG.
A normal LLM application looks like this:
User
↓
Application
↓
LLM
↓
Answer
The problem?
The LLM only has access to the information available to it.
With RAG, the architecture becomes:
┌──────────────┐
│ Documents │
└──────┬───────┘
↓
Chunking
↓
Embeddings
↓
Vector Database
↑
│
User → Query → Similarity Search
↓
Relevant Context
↓
LLM
↓
Answer
Now the model can work with information from our own knowledge base.
A production RAG pipeline typically contains these stages:
Documents
↓
Document
↓
Chunking
↓
Embeddings
↓
Vector Database
↓
Similarity Search
↓
Relevant Context
↓
Prompt
↓
LLM
↓
Answer
Let's understand each step.
First, we need to get our data into the application.
The source could be:
Spring AI provides abstractions for working with documents.
A document can be represented using Spring AI's Document
abstraction.
Conceptually:
Document document = new Document(
"Spring Boot is a framework for building Java applications..."
);
We now have content that can be processed by our RAG pipeline.
We shouldn't usually store an entire document as a single vector.
Imagine a 100-page PDF.
A user asks:
How do I configure authentication?
We don't want to retrieve the entire PDF.
Instead, we split the document into smaller pieces called chunks.
For example:
Document
↓
Chunk 1
Chunk 2
Chunk 3
Chunk 4
...
Chunk 100
Each chunk represents a smaller piece of knowledge.
A simple example:
Chunk 1:
Introduction to Spring Security
Chunk 2:
Configuring authentication
Chunk 3:
Creating users
Chunk 4:
JWT authentication
Chunk 5:
Role-based authorization
Now when the user asks about JWT authentication, we can retrieve the relevant chunk instead of the entire document.
This is where things get interesting.
A computer doesn't understand the semantic meaning of text in the same way humans do.
We need a way to represent text numerically.
That's where embeddings come in.
An embedding model converts text into a vector.
For example:
"How do I configure JWT authentication?"
might become something conceptually like:
[0.12, -0.42, 0.87, 0.31, ...]
The actual vector contains many dimensions.
The important part is:
Semantically similar text produces vectors that are relatively close together in vector space.
For example:
"How can I configure JWT?"
↓
[0.12, 0.81, 0.42, ...]
"JWT authentication configuration"
↓
[0.15, 0.78, 0.45, ...]
These vectors should have high similarity.
Now we need somewhere to store these embeddings.
That's where a vector database comes in.
Popular choices include:
For a Spring Boot application, PostgreSQL with pgvector
is an especially interesting option because you can keep your relational data and vector data within the same ecosystem.
Conceptually:
Document Chunk
↓
Embedding Model
↓
Vector
↓
Vector Database
Our database might contain something conceptually like:
ID | Content | Embedding
---|----------------------------|----------------
1 | JWT configuration... | [0.12,...]
2 | OAuth2 configuration... | [0.42,...]
3 | Database configuration... | [0.71,...]
Now suppose the user asks:
How do I configure JWT authentication in Spring Boot?
We first generate an embedding for the question.
User Query
↓
Embedding Model
↓
Query Vector
Then we search the vector database for similar vectors.
Query Vector
↓
Vector Database
↓
Similarity Search
↓
Top K Relevant Chunks
For example:
Result 1 → JWT configuration
Result 2 → Spring Security authentication
Result 3 → SecurityFilterChain configuration
These results become our context.
Now we combine:
For example:
Context:
Spring Security can be configured using SecurityFilterChain.
JWT authentication can be implemented using a custom
authentication filter...
Question:
How do I configure JWT authentication in Spring Boot?
The LLM receives this information and generates the answer.
This is the Augmented part of Retrieval-Augmented Generation.
The final flow looks like:
User Question
↓
Embedding
↓
Vector Search
↓
Relevant Documents
↓
Prompt + Context
↓
LLM
↓
Answer
The LLM isn't searching the database itself.
Our application retrieves the information first and provides it to the model.
Now let's look at how Spring AI simplifies this architecture.
A typical Spring AI RAG application contains:
Spring Boot
│
├── Document Reader
│
├── Text Splitter
│
├── Embedding Model
│
├── Vector Store
│
└── Chat Model
The exact model and vector store can be swapped without rewriting the entire application.
That's one of the strengths of Spring AI's abstraction-based approach.
Let's create a Spring Boot project.
We'll need Spring AI dependencies for:
For example, with Maven:
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-starter-model-bedrock-converse</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-starter-vector-store-pgvector</artifactId>
</dependency>
You should use versions compatible with the Spring AI version used by your project.
If we're using AWS Bedrock, our application needs AWS credentials and model configuration.
Conceptually:
spring:
ai:
bedrock:
aws:
region: us-east-1
The exact configuration depends on the Spring AI version and Bedrock model you're using.
For production environments, don't hard-code AWS credentials inside application.yml
.
Use:
Environment Variables
↓
IAM Roles
↓
AWS Credentials Provider Chain
Whenever possible, prefer IAM roles over static credentials.
For PostgreSQL + pgvector, the architecture looks like:
Spring Boot
↓
Spring AI VectorStore
↓
PostgreSQL
↓
pgvector
The vector store becomes the bridge between our application and the vector database.
Conceptually:
VectorStore vectorStore;
We can then add documents:
vectorStore.add(documents);
And later search:
List<Document> results =
vectorStore.similaritySearch(
SearchRequest.builder()
.query(question)
.topK(5)
.build()
);
The important abstraction here is:
VectorStore
Our application doesn't need to manually implement vector similarity calculations.
Let's say we have a PDF containing product documentation.
Spring AI provides document readers that can load different document formats.
Conceptually:
var documents = reader.get();
We then split the documents into chunks.
For example:
TokenTextSplitter splitter = new TokenTextSplitter();
List<Document> chunks =
splitter.apply(documents);
The exact splitter and configuration should be chosen based on your document structure and model context window.
The embedding model converts every chunk into a vector.
Conceptually:
Document Chunk
↓
Embedding Model
↓
Vector
Spring AI handles this through the embedding/vector-store integration.
Then we can store the documents:
vectorStore.add(chunks);
Our ingestion pipeline is now:
PDF
↓
Documents
↓
Chunks
↓
Embeddings
↓
Vector Store
Now let's handle the user's question.
Suppose the user asks:
How does authentication work in our application?
We search the vector store:
List<Document> results =
vectorStore.similaritySearch(
SearchRequest.builder()
.query(question)
.topK(5)
.build()
);
We now have the most relevant pieces of information.
For example:
Result 1
JWT authentication uses...
Result 2
The authentication filter...
Result 3
Security configuration...
We can now construct a prompt using the retrieved documents.
For example:
String context = results.stream()
.map(Document::getText)
.collect(Collectors.joining("\n\n"));
String prompt = """
Answer the question using only the provided context.
Context:
%s
Question:
%s
""".formatted(context, question);
Then send it to the chat model.
Conceptually:
ChatResponse response =
chatModel.call(
new Prompt(prompt)
);
And we return:
response.getResult()
.getOutput()
.getText();
That's a basic RAG implementation.
Putting everything together:
INGESTION PIPELINE
┌───────────────┐
│ Documents │
└───────┬───────┘
↓
Text Splitting
↓
Embeddings
↓
Vector Store
│
│
│
▼
RETRIEVAL
▲
│
User Question
↓
Embedding
↓
Similarity Search
↓
Top K Chunks
↓
Context
↓
Prompt
↓
LLM
↓
Answer
This is the basic architecture behind many modern knowledge-based AI applications.
One of the most underestimated parts of RAG is chunking.
Bad chunking can produce bad retrieval.
Imagine this document:
Spring Security
Authentication allows the application to verify
the identity of a user.
Authorization determines whether the authenticated
user has permission to access a resource.
If we split this badly:
Chunk 1:
Authentication allows...
Chunk 2:
the identity of a user. Authorization determines...
Chunk 3:
whether the authenticated user...
we may destroy important semantic relationships.
A better strategy is to preserve meaningful boundaries where possible.
Depending on the data, you may experiment with:
Chunk Size
Overlap
Sentence boundaries
Paragraph boundaries
Markdown headings
Semantic sections
There is no universal chunk size that works for every RAG application.
When performing similarity search, we usually retrieve the top K results.
For example:
.topK(5)
means:
Return the 5 most relevant chunks.
But bigger isn't always better.
If we retrieve too little:
K = 1
we may miss important context.
If we retrieve too much:
K = 50
we may introduce irrelevant information and increase token usage.
A common approach is to start with a small value and evaluate retrieval quality.
For example:
K = 3
K = 5
K = 10
Then measure which configuration works best for your dataset.
A common question is:
Why not fine-tune the model instead?
RAG and fine-tuning solve different problems.
Best when:
Useful when you want to change:
A useful mental model is:
RAG
→ Give the model the right information.
Fine-tuning
→ Change how the model behaves.
In many production systems, they can also be used together.
A basic RAG pipeline is only the beginning.
Production RAG systems often introduce additional stages:
Query
↓
Query Transformation
↓
Hybrid Retrieval
↓
Metadata Filtering
↓
Vector Search
↓
Reranking
↓
Context Compression
↓
Prompt Construction
↓
LLM
You might eventually introduce techniques such as:
This is where RAG becomes an engineering discipline rather than simply "put documents into a vector database."
If you're building RAG for production, don't stop at:
PDF → Vector DB → LLM
You also need to think about:
How frequently are documents updated?
New document
↓
Process
↓
Chunk
↓
Embed
↓
Update Vector Store
Store useful metadata alongside chunks:
document_id
source
page_number
tenant_id
created_at
updated_at
document_type
This becomes extremely useful for filtering.
For example:
tenant_id = "company-123"
can ensure that users only retrieve documents belonging to their tenant.
This is critical.
A RAG system must not retrieve documents that the current user isn't authorized to access.
Your retrieval layer should respect application permissions.
User
↓
Authentication
↓
Authorization
↓
Metadata Filters
↓
Retrieval
↓
LLM
Never assume that because the LLM can't "see" a document directly, the document is secure.
Track things such as:
Retrieval latency
Embedding latency
LLM latency
Token usage
Retrieved chunks
Similarity scores
Failure rate
Answer quality
Without observability, debugging RAG becomes extremely difficult.
If you're new to RAG, remember this:
"Convert meaning into numbers"
"Store and search those meanings"
"Find relevant information"
"Use that information to generate an answer"
Together:
Retrieve → Augment → Generate
That's RAG.
Spring AI gives Java developers abstractions around many of these building blocks.
Instead of manually wiring every AI provider and vector database integration, we can work with abstractions such as:
ChatModel
EmbeddingModel
VectorStore
Document
DocumentReader
This allows us to focus more on the application architecture rather than provider-specific implementation details.
And that's particularly useful when building enterprise Java applications where we may want to change:
AWS Bedrock
↓
Another Model Provider
without completely rewriting our application.
A production-oriented Spring AI RAG application can eventually look like this:
┌───────────────────┐
│ User / App │
└─────────┬─────────┘
↓
Spring Boot API
↓
Query Processing
↓
Retrieval Layer
↓
┌───────────────────┐
│ Vector Store │
│ PostgreSQL │
│ + pgvector │
└─────────┬─────────┘
↓
Relevant Context
↓
Prompt Assembly
↓
┌───────────────────┐
│ Spring AI │
└─────────┬─────────┘
↓
┌───────────────────┐
│ AWS Bedrock │
└─────────┬─────────┘
↓
Answer
And separately:
Documents
↓
Document Reader
↓
Chunking
↓
Embedding Model
↓
PostgreSQL + pgvector
We started this series by exploring how Spring AI can connect Java applications with modern AI models.
With AWS Bedrock, we can access powerful foundation models without managing the underlying model infrastructure.
Now, with RAG, we can take another major step:
We can connect those models to our own data.
The journey now looks like:
LLM
↓
Spring AI
↓
AWS Bedrock
↓
RAG
↓
Vector Database
↓
Our Own Data
But there is another important capability missing.
What if we don't just want the model to answer questions?
What if we want the model to take actions?
For example:
User
↓
AI Agent
↓
Decide what to do
↓
Call a Tool
↓
Execute Action
↓
Return Result
That's where tool calling and AI agents come in.
Next up: Building AI Agents with Spring AI — Tool Calling, Memory, and Autonomous Workflows.
If you're building AI applications with Java and Spring Boot, RAG is one of the most important patterns to understand.