Run Local LLMs with Ollama and Spring AI

A developer demonstrates how to run local large language models with Ollama and integrate them into a Spring Boot application using Spring AI, building an AI customer support assistant that keeps data on-premises. The tutorial covers installing Ollama, pulling models like llama3.2, and connecting Spring AI's ChatClient to the local Ollama API, while noting that local deployment does not automatically ensure security.

In the previous parts, we connected Spring AI with cloud-based AI models. But there is one important question: What if you don't want to send your data to an external AI provider? What if you want to: This is where Ollama becomes very useful. In this article, we will learn how to run a local LLM using Ollama and connect it with Spring AI . We will build a simple real-world AI Customer Support Assistant using Java, Spring Boot, Spring AI, and Ollama. Our application will look like this: User | | HTTP Request v +---------------------+ | Spring Boot API | +---------------------+ | v +-------------+ | Spring AI | | ChatClient | +-------------+ | v +--------+ | Ollama | +--------+ | v Local LLM Llama/Qwen | v AI Response | v User The important part is that the LLM is running locally . There is no need to send every prompt to OpenAI, Anthropic, or another cloud provider. Ollama https://ollama.com/ makes it easy to run open-source LLMs locally. Instead of calling a remote API like: Spring Boot | v OpenAI API | v Cloud LLM we can run: Spring Boot | v Spring AI | v Ollama | v Local LLM Ollama can run models such as: The exact models available change over time, so always check the Ollama model library before choosing one. Imagine you are building an internal HR application. Employees may send questions such as: What is our maternity leave policy? or: What is the process for requesting annual leave? You may not want internal company information leaving your infrastructure. A local LLM can help: Employee | v Spring Boot | v RAG / Business Logic | v Ollama | v Local LLM This can provide a useful privacy boundary. However, remember: Running an LLM locally does not automatically make your application secure. You still need proper authentication, authorization, logging, data protection, network security, and prompt/data controls. First, install Ollama on your operating system. After installation, verify it: ollama --version If the command works, Ollama is installed. Now we need an LLM. For example: ollama pull llama3.2 Then run it: ollama run llama3.2 You can now talk to the model directly from your terminal. For example: Explain Java interfaces in simple English. The model will generate a response locally. At a high level, the architecture is: Your Application | v Ollama API | v Model Runtime | v Local Model | v Response Ollama exposes an API that applications can communicate with. Spring AI can communicate with this API for us. That means we don't have to manually build HTTP requests to the Ollama API. Let's create a Spring Boot application. You can use Spring Initializr or your preferred IDE. Basic project: local-ai-demo │ ├── src │ └── main │ ├── java │ │ └── com.example.localai │ │ └── LocalAiApplication.java │ │ │ └── resources │ └── application.yml │ └── pom.xml We need: Add the Spring AI Ollama starter to your pom.xml . <dependency <groupId org.springframework.ai</groupId <artifactId spring-ai-starter-model-ollama</artifactId </dependency You should use the Spring AI version compatible with your Spring Boot version. For production projects, avoid randomly mixing Spring Boot and Spring AI versions. Now configure the Ollama model. spring: ai: ollama: base-url: http://localhost:11434 chat: options: model: llama3.2 The important part is: localhost:11434 This is the default Ollama API endpoint. Your architecture now becomes: Spring Boot | | HTTP v localhost:11434 | v Ollama | v llama3.2 Spring AI provides ChatClient , which gives us a clean API for interacting with chat models. Create a configuration class: python package com.example.localai.config; import org.springframework.ai.chat.client.ChatClient; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; @Configuration public class AiConfig { @Bean ChatClient chatClient ChatClient.Builder builder { return builder.build ; } } That's it. Spring AI will use the configured Ollama chat model. Now let's create a simple controller. python package com.example.localai.controller; import org.springframework.ai.chat.client.ChatClient; import org.springframework.web.bind.annotation. ; @RestController @RequestMapping "/api/ai" public class AiController { private final ChatClient chatClient; public AiController ChatClient chatClient { this.chatClient = chatClient; } @GetMapping "/ask" public String ask @RequestParam String question { return chatClient .prompt .user question .call .content ; } } Start your Spring Boot application. Then call: GET /api/ai/ask?question=Explain Spring Boot in simple English The request flows like this: HTTP Request | v AiController | v ChatClient | v Spring AI | v Ollama | v Local LLM | v AI Response And the response comes back to the client. Let's make our application more useful. Imagine we are building an AI Customer Support Assistant . A customer sends: My payment was deducted but my subscription is still inactive. What should I do? Instead of simply passing the question to the model, we can provide a system instruction. @GetMapping "/support" public String support @RequestParam String question { return chatClient .prompt .system """ You are a helpful customer support assistant. Answer in simple English. Do not invent company policies. If you do not know something, clearly say that you do not have enough information. """ .user question .call .content ; } Now the model has some context about its role. Spring AI allows us to separate instructions from user input. For example: return chatClient .prompt .system """ You are an AI customer support assistant. Keep answers short and easy to understand. Never make up information. """ .user question .call .content ; Think about it like this: System Prompt | | "Who are you?" | "How should you behave?" | v LLM ^ | | "What does the user want?" | User Prompt This separation becomes extremely useful when building production AI applications. Putting everything inside the controller is not a good architecture. Instead: Controller | v Service | v ChatClient | v Ollama Create: python package com.example.localai.service; import org.springframework.ai.chat.client.ChatClient; import org.springframework.stereotype.Service; @Service public class AiService { private final ChatClient chatClient; public AiService ChatClient chatClient { this.chatClient = chatClient; } public String ask String question { return chatClient .prompt .user question .call .content ; } } Then the controller becomes: python package com.example.localai.controller; import com.example.localai.service.AiService; import org.springframework.web.bind.annotation. ; @RestController @RequestMapping "/api/ai" public class AiController { private final AiService aiService; public AiController AiService aiService { this.aiService = aiService; } @GetMapping "/ask" public String ask @RequestParam String question { return aiService.ask question ; } } This structure is much easier to extend later. Let's make the prompt more useful. public String support String question { return chatClient .prompt .system """ You are a professional customer support assistant. Rules: 1. Use simple English. 2. Be polite and helpful. 3. Give step-by-step instructions when possible. 4. Never invent policies. 5. If information is missing, ask for clarification. """ .user question .call .content ; } Now a question like: I cannot reset my password. could produce something like: I'm sorry you're having trouble resetting your password. Please try these steps: 1. Open the login page. 2. Click "Forgot Password". 3. Enter your registered email. 4. Check your email for the reset link. If you still cannot reset your password, contact support. This is already a useful AI feature. A basic AI call is stateless. For example: User: My order is late. AI: Please provide your order number. User: It is 12345. AI: What order are you referring to? The model doesn't automatically know the previous conversation unless we provide that context. In a real application, we need conversation memory. The architecture becomes: php User | v Spring Boot | +------ Conversation Store | | | v | Previous Messages | v Spring AI | v Ollama | v LLM Depending on your application, the conversation history can be stored in databases such as PostgreSQL or Redis. For example: conversation id | v +----------------------+ | User message | | AI response | | User message | | AI response | +----------------------+ Then the relevant history can be included when making the next model call. This is an important step from a simple AI demo toward a production AI application. This is where local models become especially interesting. Suppose your company has: Employee Handbook Product Documentation Customer FAQs Internal Policies Technical Documentation We can build a RAG system: Documents | v Text Parser | v Chunking | v Embeddings | v Vector Database | | User Question --- Retrieval | v Relevant Data | v Spring AI | v Ollama | v Local LLM | v Answer Now the LLM doesn't need to know your company information beforehand. We retrieve the relevant information and provide it as context. Imagine an employee asks: How many days of annual leave can I take? The system can search the company's HR documents. Suppose the vector database retrieves: Employees receive 24 days of annual leave per calendar year. Spring AI can then construct a prompt: Answer the question using only the following context. Context: Employees receive 24 days of annual leave per calendar year. Question: How many days of annual leave can I take? The local LLM generates: According to the company policy, employees receive 24 days of annual leave per calendar year. The complete architecture becomes: Employee | v Spring Boot API | v Spring AI RAG / \ / \ v v Vector Database Ollama | | v v Relevant Data Local LLM \ / \ / v v Answer This is much closer to a real enterprise AI architecture. Imagine an organization has sensitive documents. With a cloud-only architecture: Application | v Cloud AI API | v External Model With a local architecture: php Application | v Internal Infrastructure | +------ Vector DB | +------ Ollama | v Local LLM This can be attractive for: But again, local inference is not a complete security strategy by itself. For chat applications, waiting for the entire response can feel slow. A better experience is: AI is typing... Hello Hello, how Hello, how can Hello, how can I Hello, how can I help? Spring AI supports streaming responses through Flux . For example: @GetMapping value = "/stream", produces = "text/event-stream" public Flux<String stream @RequestParam String question { return chatClient .prompt .user question .stream .content ; } The client can receive pieces of the response as they are generated. This is useful for: Not every local model is good for every task. For example: Small model | +-- Faster +-- Less memory +-- Lower hardware requirements Large model | +-- Better reasoning potential +-- More memory +-- Higher latency Your choice depends on: For a simple local experiment, start with a relatively small model. Then benchmark larger models if your hardware allows it. A simple comparison: | Feature | Local Ollama | Cloud LLM | |---|---|---| | Internet required | Usually no | Yes | | API cost | No per-token cloud fee | Usually usage-based | | Data leaves machine | Can stay local | Sent to provider | | Hardware required | Yes | Mostly no | | Scaling | Your responsibility | Provider handles infrastructure | | Model choice | Open/local models | Provider-specific models | | Setup | More infrastructure | Usually easier | | Latency | Depends on hardware | Depends on network/provider | There is no universal winner. A practical architecture may even use both. For example: AI Gateway | +--------+--------+ | | v v Ollama Cloud LLM | | v v Local Model External Model You could use: Sensitive requests | v Ollama and: Complex reasoning | v Cloud Model The routing decision can be implemented inside your application. This gives you more flexibility. Running Ollama on your laptop is great for development. Production is different. You need to think about: Load Balancer | v Spring Boot Instances | v AI Service | v GPU-enabled inference servers Track: Protect: Do not expose an unauthenticated Ollama service directly to the public internet. AI services can fail. Your application should not assume every model call succeeds. For example: public String ask String question { try { return chatClient .prompt .user question .call .content ; } catch Exception ex { throw new RuntimeException "AI service is currently unavailable", ex ; } } In a production application, use a proper exception hierarchy and global exception handling rather than exposing raw exceptions. You may also add: Timeout Retry Circuit Breaker Fallback Rate Limiting Observability A more realistic architecture could look like this: Client | v API Gateway | v Spring Boot API | +-----------+-----------+ | | v v Conversation RAG Service Service | | v | Vector Database | | +-----------+-----------+ | v AI Service | +--------+--------+ | | v v Ollama Cloud LLM | | v v Local Model External Model This architecture allows you to evolve from a simple local experiment into a production-grade AI platform. Here is the complete service: @Service public class AiService { private final ChatClient chatClient; public AiService ChatClient chatClient { this.chatClient = chatClient; } public String ask String question { return chatClient .prompt .system """ You are a helpful AI assistant. Answer in simple English. Do not invent facts. """ .user question .call .content ; } } Controller: @RestController @RequestMapping "/api/ai" public class AiController { private final AiService aiService; public AiController AiService aiService { this.aiService = aiService; } @GetMapping "/ask" public String ask @RequestParam String question { return aiService.ask question ; } } Configuration: spring: ai: ollama: base-url: http://localhost:11434 chat: options: model: llama3.2 Architecture: HTTP Client | v AiController | v AiService | v ChatClient | v Spring AI | v Ollama | v Local LLM That's enough to build your first local AI backend. Start Ollama: ollama run llama3.2 Then start Spring Boot: ./mvnw spring-boot:run On Windows: mvnw.cmd spring-boot:run Then call: GET http://localhost:8080/api/ai/ask?question=Explain dependency injection in Spring Boot You should receive an answer generated by your local model. In this article, we built a local AI backend using: Java + Spring Boot + Spring AI + Ollama + Local LLM We learned: ChatClient Running an LLM locally changes the way we think about AI applications. You don't always need to start with an expensive cloud API. You can start on your own laptop: Ollama | Local LLM | Spring AI | Spring Boot | REST API Then gradually evolve it: Local LLM ↓ RAG ↓ Vector Database ↓ Conversation Memory ↓ Tool Calling ↓ AI Agents ↓ Observability ↓ Production AI Platform And this is where Spring AI becomes interesting for Java developers . You can use the Spring ecosystem you already know and add modern AI capabilities without completely changing how you build backend applications. In the next part, we can go one step further and build a RAG application with Spring AI, embeddings, PostgreSQL + pgvector, and a local Ollama model . That is where our simple chatbot starts becoming a real AI application.