Large Language Models are excellent at generating text.
But generation alone isn't enough to build truly useful AI applications.
Imagine asking an AI assistant:
What's the status of my order?
A normal LLM can explain how order tracking works.
But it cannot magically access your order database.
Or suppose you ask:
Cancel my order #ORD-10291.
The model can tell you how to cancel an order.
But it cannot actually cancel anything unless your application gives it the ability to perform that action.
This is where tool calling and AI agents come in.
Instead of simply generating an answer, an AI application can:
Understand the request
↓
Decide what action is required
↓
Select a tool
↓
Execute the tool
↓
Observe the result
↓
Continue reasoning
↓
Generate the final response
In this article, we'll explore how to build this architecture using Spring AI.
An AI agent is an application where an LLM can decide what actions need to be performed and use available tools to accomplish a goal.
A traditional LLM application looks like:
User
↓
Prompt
↓
LLM
↓
Response
An agent-based application looks more like:
User
↓
LLM
↓
Decision
↓
Tool
↓
Result
↓
LLM
↓
Decision
↓
Another Tool
↓
Result
↓
Final Answer
The important difference is:
The LLM is no longer limited to generating text.
It can interact with the application through controlled capabilities.
Tool calling allows an LLM to request the execution of a function exposed by your application.
For example, imagine our application provides:
getOrderStatus()
cancelOrder()
getCustomer()
createSupportTicket()
The user asks:
Where is my order?
The model might determine that it needs:
getOrderStatus()
The application executes the function and returns:
Order #10291
Status: Shipped
Expected delivery: September 10
The LLM can then generate:
Your order has been shipped and is expected to arrive on September 10.
The LLM didn't directly access the database.
Instead:
LLM
↓
Tool Request
↓
Application
↓
Database
↓
Tool Result
↓
LLM
↓
Answer
This distinction is extremely important for enterprise applications.
Without tools:
LLM
↓
Text
With tools:
LLM
↓
Tools
├── Database
├── REST APIs
├── Search
├── Payment systems
├── CRM
├── Internal services
└── Business workflows
This turns the LLM from a text-generation component into an interface for interacting with your application.
For example, an AI sales assistant could have:
getCustomer()
getCustomerOrders()
createLead()
updateLead()
sendEmail()
scheduleMeeting()
A support agent could have:
searchKnowledgeBase()
getCustomerAccount()
getOrder()
createTicket()
updateTicket()
An internal developer assistant could have:
searchDocumentation()
searchGitRepository()
getBuildStatus()
createIssue()
The possibilities are much broader than simple question answering.
At this point, it is useful to distinguish RAG from tool calling.
RAG is primarily about retrieving information.
Tool calling is about performing actions or retrieving live data through application capabilities.
For example:
RAG
↓
Retrieve company documentation
↓
Answer question
Tool calling:
LLM
↓
Call order API
↓
Get live order status
↓
Answer
They can also be combined.
User
↓
AI Agent
├── RAG → Search company policies
│
├── Tool → Get customer account
│
└── Tool → Check order status
↓
LLM
↓
Answer
This combination is extremely powerful.
Spring AI provides abstractions that make it easier to expose application capabilities to language models.
A simplified architecture looks like:
Spring Boot
│
├── ChatModel
│
├── Tools
│
├── Advisors
│
├── Chat Memory
│
└── Vector Store
The application controls which tools are available.
The model decides whether a tool is needed.
This separation is important.
The model should not have unrestricted access to your application.
Instead, the application exposes specific capabilities.
Imagine we have an order service.
@Service
public class OrderService {
public OrderStatus getOrderStatus(String orderId) {
// Fetch order from database
return orderRepository.findStatus(orderId);
}
}
We can expose a controlled method as an AI tool.
Conceptually:
@Tool(
description = "Get the current status of an order"
)
public OrderStatus getOrderStatus(String orderId) {
return orderService.getOrderStatus(orderId);
}
The description is important.
The model uses the tool description to understand:
What does this tool do?
When should I use it?
What parameters does it require?
Tool:
getOrderStatus
Description:
Returns the current shipping and delivery status
for a customer order.
Input:
orderId
The model can then determine whether this tool is appropriate.
A tool can be thought of as:
Tool Name
+
Description
+
Input Schema
+
Execution Logic
getOrderStatus(
orderId: String
)
The model might produce a tool request conceptually like:
{
"name": "getOrderStatus",
"arguments": {
"orderId": "ORD-10291"
}
}
The application receives the request and executes the corresponding Java method.
A typical interaction looks like this:
User
│
│ "What's the status of ORD-10291?"
↓
LLM
│
│ Tool Request
↓
getOrderStatus("ORD-10291")
│
↓
Order Service
│
↓
Database
│
↓
Tool Result
│
↓
LLM
│
↓
Final Answer
Notice something important.
The LLM doesn't execute Java code itself.
The application remains responsible for execution.
The model only requests the action.
Spring AI's ChatClient provides a convenient API for interacting with chat models.
ChatClient chatClient;
String response = chatClient.prompt()
.user("What's the status of order ORD-10291?")
.tools(orderTools)
.call()
.content();
The exact APIs may vary depending on the Spring AI version you're using, but the architecture remains the same:
ChatClient
↓
ChatModel
↓
Tool Selection
↓
Tool Execution
↓
Tool Result
↓
Final Response
A real agent usually has more than one tool.
CustomerAgentTools
├── getCustomer()
├── getCustomerOrders()
├── getOrderStatus()
├── createSupportTicket()
└── updateCustomer()
Now consider this question:
My order is late. Please check the status
and create a support ticket if necessary.
The model might determine:
1. getOrderStatus()
2. Analyze result
3. createSupportTicket()
4. Return final response
The application executes each requested operation.
This is where the concept of an agent starts becoming much more interesting.
A simple agent loop can be represented as:
┌───────────────┐
│ User │
└───────┬───────┘
↓
┌───────────┐
│ LLM │
└─────┬─────┘
↓
Need a Tool?
/ \
No Yes
↓ ↓
Final Answer Tool Call
↓
Tool Execution
↓
Tool Result
↓
LLM
↓
Need Another Tool?
The model can repeatedly interact with tools until it has enough information to produce the final response.
This distinction is important.
People often hear:
AI Agent
and immediately think:
Give AI access to everything
↓
Let AI do whatever it wants
That is not how production systems should be designed.
A production agent should operate inside clear boundaries.
Allowed Tools
↓
Authorization
↓
Validation
↓
Execution
↓
Audit
The application remains in control.
The model should not be trusted with unrestricted capabilities.
Imagine we expose:
@Tool
public void deleteCustomer(String customerId) {
...
}
This is potentially dangerous.
An LLM should not automatically receive unrestricted permission to perform destructive operations.
Instead, sensitive tools should have additional controls.
User
↓
Authentication
↓
Authorization
↓
Agent
↓
Tool Request
↓
Permission Check
↓
Confirmation
↓
Execution
For destructive operations, you may require explicit user confirmation.
Example:
AI:
I found customer account C-19291.
Deleting this account is irreversible.
Do you want me to continue?
User:
Yes.
AI:
Executing deletion...
The AI should assist with the decision process, not bypass your security model.
A useful production architecture is to classify tools.
READ
getCustomer()
getOrder()
searchDocuments()
getInvoice()
Then:
WRITE
createTicket()
updateCustomer()
createLead()
And:
DESTRUCTIVE
deleteCustomer()
cancelSubscription()
refundPayment()
Different permission levels can then be applied.
READ
→ Automatically allowed
WRITE
→ Role-based authorization
DESTRUCTIVE
→ Authorization + confirmation
This makes agent behavior much safer.
Tool calling solves one problem.
But another problem appears quickly:
What does the agent remember?
Consider this conversation:
User:
My order is late.
AI:
What's your order number?
User:
ORD-10291.
AI:
Let me check it.
Now the next message is:
Can you create a support ticket for it?
The AI needs to understand that:
"it"
refers to:
ORD-10291
This requires conversational context.
That's where chat memory becomes important.
A simple conversation can be represented as:
User:
My order is late.
Assistant:
What's your order number?
User:
ORD-10291.
Assistant:
Let me check that order.
The application maintains the conversation history.
Conversation ID
↓
Chat Memory
↓
Previous Messages
↓
Current Prompt
↓
LLM
Spring AI provides abstractions for managing chat memory.
It is useful to distinguish two concepts.
Conversation context.
User:
My order is late.
User:
It's order 10291.
User:
Can you check it?
The system remembers the current conversation.
Persistent information about the user.
Customer:
Ayush
Preferences:
Preferred language = English
Preferred notification = Email
Long-term memory usually requires persistence in a database or another storage system.
A production architecture might look like:
Conversation
↓
Chat Memory Store
↓
PostgreSQL / Redis
The exact storage mechanism depends on the application.
Now we can combine:
User
↓
Agent
↓
Memory
↓
LLM
↓
Tools
↓
Tool Results
↓
Memory
↓
LLM
↓
Answer
This enables more natural multi-turn interactions.
Another important Spring AI concept is the Advisor.
Advisors can intercept and influence the interaction between the application and the model.
They can be used for concerns such as:
Conversation memory
RAG
Logging
Security
Prompt modification
Context injection
Observability
User
↓
ChatClient
↓
Advisor
↓
ChatModel
↓
Advisor
↓
Response
This allows cross-cutting AI behavior to be separated from business logic.
Now things become much more powerful.
Imagine an enterprise support agent.
It has:
RAG
↓
Company documentation
Tools:
getCustomer()
getOrder()
createTicket()
Memory:
Conversation history
The architecture becomes:
User
↓
AI Agent
↓
┌───────┼────────┐
↓ ↓ ↓
RAG Tools Memory
↓ ↓ ↓
Knowledge APIs Conversation
│ │ │
└───────┼────────┘
↓
LLM
↓
Response
This is much closer to a production AI application.
Consider the request:
My payment failed for order ORD-19291.
Can you check what happened and tell me
what I should do?
The agent could perform:
1. getOrder("ORD-19291")
2. getPaymentStatus("ORD-19291")
3. searchKnowledgeBase("payment failure")
4. Generate explanation
The final answer could be:
Your payment attempt failed because the transaction
was declined by the payment provider.
According to the payment policy, you can retry the
payment using another payment method.
Would you like me to create a support ticket?
The model combined:
Live application data
+
Knowledge base
+
Conversation context
This is significantly more useful than a standalone chatbot.
Agents can also perform multi-step workflows.
User:
Find my overdue invoices and send reminders.
The agent could reason through:
getCustomer()
↓
getInvoices()
↓
Filter overdue invoices
↓
sendReminder()
↓
Return summary
The workflow becomes:
Goal
↓
Plan
↓
Tool
↓
Observe
↓
Next Decision
↓
Tool
↓
Observe
↓
Final Result
This pattern is often called an agent loop.
A simplified conceptual implementation looks like:
while (!completed) {
AgentDecision decision =
llm.decide(context);
if (decision.requiresTool()) {
ToolResult result =
toolExecutor.execute(
decision.toolCall()
);
context.add(result);
} else {
return decision.finalAnswer();
}
}
In real applications, frameworks handle much of this interaction.
But understanding the underlying loop is important.
An important engineering lesson:
Not every AI feature needs an agent.
If your workflow is deterministic:
Validate request
↓
Call API
↓
Save result
↓
Return response
you probably don't need an autonomous agent.
A normal service workflow may be better.
Agents become more useful when:
The next step depends on the current result.
Check order
↓
If delayed
↓
Check refund policy
↓
If eligible
↓
Ask for confirmation
↓
Create refund request
The dynamic decision-making is where agents become valuable.
A → B → C → D
Everything is predetermined.
A
↓
LLM decides
├── B
├── C
└── D
↓
Observe result
↓
Decide again
Agents provide flexibility.
Traditional workflows provide predictability.
Production systems often use both.
A practical Spring Boot architecture might look like:
┌───────────────┐
│ Frontend │
└───────┬───────┘
↓
┌───────────────┐
│ Spring Boot │
│ API │
└───────┬───────┘
↓
┌────────────┐
│ ChatClient │
└─────┬──────┘
↓
┌──────────────┐
│ Agent │
└──────┬───────┘
↓
┌────────────┼────────────┐
↓ ↓ ↓
Memory RAG Tools
↓ ↓ ↓
PostgreSQL pgvector APIs
↓
Microservices
This architecture fits naturally into existing Spring Boot applications.
Agent systems can become difficult to debug.
Imagine an agent performs:
Tool 1
Tool 2
Tool 3
Tool 4
and the final response is incorrect.
You need to know:
What did the model decide?
Which tools were selected?
What arguments were sent?
How long did each tool take?
What did each tool return?
How many model calls happened?
How many tokens were consumed?
Therefore, observability is critical.
Track:
LLM latency
Tool latency
Retrieval latency
Token usage
Tool calls
Tool failures
Model responses
Agent iterations
Errors
An agent can potentially continue calling tools indefinitely.
LLM
↓
Tool
↓
LLM
↓
Tool
↓
LLM
↓
Tool
↓
...
Production systems should enforce limits.
Maximum iterations = 10
Maximum tool calls = 20
Maximum execution time = 30 seconds
You should also define clear failure behavior.
Agent limit reached
↓
Stop execution
↓
Return safe response
↓
Log failure
Never blindly trust model-generated tool arguments.
Suppose the model requests:
{
"orderId": "ORD-999999999"
}
Your application should still validate:
Does the order exist?
Does the user own the order?
Is the user authorized?
Is the order accessible to this tenant?
The architecture should be:
LLM
↓
Tool Request
↓
Schema Validation
↓
Authorization
↓
Business Validation
↓
Tool Execution
The LLM is not your security boundary.
Your application is.
This becomes especially important in SaaS applications.
Imagine:
Tenant A
├── Customers
├── Orders
└── Documents
Tenant B
├── Customers
├── Orders
└── Documents
An AI agent must never retrieve Tenant B's information while processing a Tenant A request.
Every tool and retrieval operation should carry tenant context.
tenant_id
user_id
roles
permissions
User
↓
Authentication
↓
Tenant Context
↓
Agent
↓
Tool
↓
Authorization
↓
Tenant-scoped Data
The same principle applies to RAG.
Vector Search
+
tenant_id filter
should ensure that retrieved documents belong to the correct tenant.
Production agents should have explicit guardrails.
Examples:
Input validation
Output validation
Tool authorization
Rate limiting
Token limits
Iteration limits
PII protection
Audit logging
Human approval
For high-risk actions:
Agent
↓
Tool Request
↓
Risk Evaluation
↓
Human Approval
↓
Execution
This creates a human-in-the-loop architecture.
Not every decision should be fully automated.
Refund amount < $50
↓
Automatic
Refund amount > $50
↓
Human approval
Or:
Create support ticket
↓
Automatic
Delete account
↓
Confirmation required
This gives us a practical balance:
AI Automation
+
Business Rules
+
Human Oversight
At this point, we can combine everything we've discussed.
User
↓
Spring Boot
↓
ChatClient
↓
AI Agent
↓
┌───────────┼───────────┐
↓ ↓ ↓
Memory RAG Tools
↓ ↓ ↓
PostgreSQL pgvector REST APIs
↓
Business Services
↓
Database
This is a strong foundation for enterprise AI applications.
Imagine a sales assistant.
Show me the latest opportunities for Acme
and tell me which ones are likely to close this month.
The agent could:
1. getCustomer("Acme")
2. getOpportunities("Acme")
3. retrieve sales documentation
4. analyze opportunity information
5. generate summary
Now the user says:
Create a follow-up task for the highest priority opportunity.
The agent can:
1. Identify opportunity
2. createFollowUpTask()
3. Return task details
This is where AI starts becoming an application interface rather than simply a chatbot.
Think about the responsibilities this way:
Reasoning + Language
Knowledge Retrieval
Actions + Live Data
Conversation Context
Application + Security + Business Logic
Together:
LLM
+
RAG
+
Tools
+
Memory
+
AI Application
Spring AI provides abstractions that allow Java developers to work with AI capabilities using familiar Spring patterns.
Important building blocks include:
ChatClient
ChatModel
EmbeddingModel
VectorStore
Document
Advisors
Chat Memory
Tools
This means an enterprise Java team can integrate AI into an existing Spring Boot architecture instead of creating an entirely separate AI stack.
Existing Spring Boot Application
↓
Spring AI Layer
↓
Model + RAG + Tools
↓
Existing Microservices
This makes AI integration much more practical for Java teams.
A more complete production system might eventually look like:
┌───────────────┐
│ User │
└───────┬───────┘
↓
API Gateway
↓
Authentication
↓
Spring Boot API
↓
AI Agent
↓
┌──────────────────────┼──────────────────────┐
↓ ↓ ↓
Memory RAG Tools
↓ ↓ ↓
PostgreSQL pgvector Microservices
↓
Business Database
↓
LLM Provider
↓
Response
And around the entire system:
Security
Observability
Rate Limiting
Audit Logging
Guardrails
Evaluation
These are not optional concerns in serious enterprise deployments.
It is useful to understand the difference.
User
↓
LLM
↓
Answer
User
↓
Retrieve Knowledge
↓
LLM
↓
Answer
User
↓
LLM
↓
Tool
↓
Result
↓
Answer
User
↓
Agent
↓
Reason
↓
Tool
↓
Observe
↓
Reason
↓
Tool
↓
Observe
↓
Final Answer
The complexity increases at every stage.
We can now think about the evolution of an AI application:
Level 1
LLM
↓
Text Generation
Level 2
LLM + RAG
↓
Knowledge Retrieval
Level 3
LLM + Tools
↓
Actions
Level 4
LLM + Tools + Memory
↓
Contextual Assistant
Level 5
LLM + RAG + Tools + Memory
↓
Agent
Level 6
Multiple Agents + Workflows
↓
Agentic System
This progression is useful when deciding how much complexity your application actually needs.
The evolution from traditional AI applications to agentic applications can be summarized as:
LLM
↓
Generate Text
RAG
↓
Retrieve Knowledge
Tool Calling
↓
Take Actions
Memory
↓
Remember Context
Agents
↓
Make Decisions
Workflows
↓
Coordinate Multiple Steps
Spring AI provides Java developers with abstractions for building many of these capabilities inside the Spring ecosystem.
The most important engineering principle is:
Let the model decide, but let your application control.
The LLM can decide which tool may be useful.
Your application should decide whether that tool is actually allowed to execute.
That separation gives us a much safer architecture for enterprise AI.
We've now covered three major capabilities:
LLM
↓
Generate
RAG
↓
Retrieve
Tools
↓
Act
But there is another challenge.
What happens when a system has:
Multiple agents
↓
Multiple tools
↓
Multiple services
↓
Multiple AI models
How do these agents communicate?
How do we standardize tool discovery?
How can an AI agent securely interact with external tools and services?
This leads us toward another important concept in modern AI engineering:
Model Context Protocol — MCP.
In the next article, we'll explore:
Building MCP Clients and Tool-Based AI Applications with Spring AI.
The future of enterprise AI isn't just:
LLM → Answer
It's increasingly:
LLM
↓
Reason
↓
Retrieve
↓
Call Tools
↓
Observe
↓
Act
↓
Remember
↓
Complete the Goal
And that's where AI agents with Spring AI become truly interesting.