{"slug": "run-local-llms-with-ollama-and-spring-ai", "title": "Run Local LLMs with Ollama and Spring AI", "summary": "A developer demonstrates how to run local large language models with Ollama and integrate them into a Spring Boot application using Spring AI, building an AI customer support assistant that keeps data on-premises. The tutorial covers installing Ollama, pulling models like llama3.2, and connecting Spring AI's ChatClient to the local Ollama API, while noting that local deployment does not automatically ensure security.", "body_md": "In the previous parts, we connected Spring AI with cloud-based AI models.\n\nBut there is one important question:\n\n**What if you don't want to send your data to an external AI provider?**\n\nWhat if you want to:\n\nThis is where **Ollama** becomes very useful.\n\nIn this article, we will learn how to run a local LLM using Ollama and connect it with **Spring AI**.\n\nWe will build a simple real-world **AI Customer Support Assistant** using Java, Spring Boot, Spring AI, and Ollama.\n\nOur application will look like this:\n\n```\n                  User\n                    |\n                    | HTTP Request\n                    v\n          +---------------------+\n          |   Spring Boot API   |\n          +---------------------+\n                    |\n                    v\n             +-------------+\n             |  Spring AI  |\n             |  ChatClient |\n             +-------------+\n                    |\n                    v\n               +--------+\n               | Ollama |\n               +--------+\n                    |\n                    v\n              Local LLM\n             (Llama/Qwen)\n                    |\n                    v\n              AI Response\n                    |\n                    v\n                  User\n```\n\nThe important part is that the LLM is running **locally**.\n\nThere is no need to send every prompt to OpenAI, Anthropic, or another cloud provider.\n\n[Ollama](https://ollama.com/) makes it easy to run open-source LLMs locally.\n\nInstead of calling a remote API like:\n\n```\nSpring Boot\n     |\n     v\nOpenAI API\n     |\n     v\nCloud LLM\n```\n\nwe can run:\n\n```\nSpring Boot\n     |\n     v\nSpring AI\n     |\n     v\nOllama\n     |\n     v\nLocal LLM\n```\n\nOllama can run models such as:\n\nThe exact models available change over time, so always check the Ollama model library before choosing one.\n\nImagine you are building an internal HR application.\n\nEmployees may send questions such as:\n\n```\nWhat is our maternity leave policy?\n```\n\nor:\n\n```\nWhat is the process for requesting annual leave?\n```\n\nYou may not want internal company information leaving your infrastructure.\n\nA local LLM can help:\n\n```\nEmployee\n   |\n   v\nSpring Boot\n   |\n   v\nRAG / Business Logic\n   |\n   v\nOllama\n   |\n   v\nLocal LLM\n```\n\nThis can provide a useful privacy boundary.\n\nHowever, remember:\n\n**Running an LLM locally does not automatically make your application secure.**\n\nYou still need proper authentication, authorization, logging, data protection, network security, and prompt/data controls.\n\nFirst, install Ollama on your operating system.\n\nAfter installation, verify it:\n\n```\nollama --version\n```\n\nIf the command works, Ollama is installed.\n\nNow we need an LLM.\n\nFor example:\n\n```\nollama pull llama3.2\n```\n\nThen run it:\n\n```\nollama run llama3.2\n```\n\nYou can now talk to the model directly from your terminal.\n\nFor example:\n\n```\n>>> Explain Java interfaces in simple English.\n```\n\nThe model will generate a response locally.\n\nAt a high level, the architecture is:\n\n```\n             Your Application\n                    |\n                    v\n              Ollama API\n                    |\n                    v\n             Model Runtime\n                    |\n                    v\n              Local Model\n                    |\n                    v\n                Response\n```\n\nOllama exposes an API that applications can communicate with.\n\nSpring AI can communicate with this API for us.\n\nThat means we don't have to manually build HTTP requests to the Ollama API.\n\nLet's create a Spring Boot application.\n\nYou can use Spring Initializr or your preferred IDE.\n\nBasic project:\n\n```\nlocal-ai-demo\n│\n├── src\n│   └── main\n│       ├── java\n│       │   └── com.example.localai\n│       │       └── LocalAiApplication.java\n│       │\n│       └── resources\n│           └── application.yml\n│\n└── pom.xml\n```\n\nWe need:\n\nAdd the Spring AI Ollama starter to your `pom.xml`\n\n.\n\n```\n<dependency>\n    <groupId>org.springframework.ai</groupId>\n    <artifactId>spring-ai-starter-model-ollama</artifactId>\n</dependency>\n```\n\nYou should use the Spring AI version compatible with your Spring Boot version.\n\nFor production projects, avoid randomly mixing Spring Boot and Spring AI versions.\n\nNow configure the Ollama model.\n\n```\nspring:\n  ai:\n    ollama:\n      base-url: http://localhost:11434\n      chat:\n        options:\n          model: llama3.2\n```\n\nThe important part is:\n\n```\nlocalhost:11434\n```\n\nThis is the default Ollama API endpoint.\n\nYour architecture now becomes:\n\n```\nSpring Boot\n     |\n     | HTTP\n     v\nlocalhost:11434\n     |\n     v\nOllama\n     |\n     v\nllama3.2\n```\n\nSpring AI provides `ChatClient`\n\n, which gives us a clean API for interacting with chat models.\n\nCreate a configuration class:\n\n``` python\npackage com.example.localai.config;\n\nimport org.springframework.ai.chat.client.ChatClient;\nimport org.springframework.context.annotation.Bean;\nimport org.springframework.context.annotation.Configuration;\n\n@Configuration\npublic class AiConfig {\n\n    @Bean\n    ChatClient chatClient(ChatClient.Builder builder) {\n        return builder.build();\n    }\n}\n```\n\nThat's it.\n\nSpring AI will use the configured Ollama chat model.\n\nNow let's create a simple controller.\n\n``` python\npackage com.example.localai.controller;\n\nimport org.springframework.ai.chat.client.ChatClient;\nimport org.springframework.web.bind.annotation.*;\n\n@RestController\n@RequestMapping(\"/api/ai\")\npublic class AiController {\n\n    private final ChatClient chatClient;\n\n    public AiController(ChatClient chatClient) {\n        this.chatClient = chatClient;\n    }\n\n    @GetMapping(\"/ask\")\n    public String ask(@RequestParam String question) {\n\n        return chatClient\n                .prompt()\n                .user(question)\n                .call()\n                .content();\n    }\n}\n```\n\nStart your Spring Boot application.\n\nThen call:\n\n```\nGET /api/ai/ask?question=Explain Spring Boot in simple English\n```\n\nThe request flows like this:\n\n```\nHTTP Request\n     |\n     v\nAiController\n     |\n     v\nChatClient\n     |\n     v\nSpring AI\n     |\n     v\nOllama\n     |\n     v\nLocal LLM\n     |\n     v\nAI Response\n```\n\nAnd the response comes back to the client.\n\nLet's make our application more useful.\n\nImagine we are building an **AI Customer Support Assistant**.\n\nA customer sends:\n\n```\nMy payment was deducted but my subscription is still inactive.\nWhat should I do?\n```\n\nInstead of simply passing the question to the model, we can provide a system instruction.\n\n```\n@GetMapping(\"/support\")\npublic String support(@RequestParam String question) {\n\n    return chatClient\n            .prompt()\n            .system(\"\"\"\n                    You are a helpful customer support assistant.\n\n                    Answer in simple English.\n                    Do not invent company policies.\n                    If you do not know something, clearly say that\n                    you do not have enough information.\n                    \"\"\")\n            .user(question)\n            .call()\n            .content();\n}\n```\n\nNow the model has some context about its role.\n\nSpring AI allows us to separate instructions from user input.\n\nFor example:\n\n```\nreturn chatClient\n        .prompt()\n        .system(\"\"\"\n                You are an AI customer support assistant.\n                Keep answers short and easy to understand.\n                Never make up information.\n                \"\"\")\n        .user(question)\n        .call()\n        .content();\n```\n\nThink about it like this:\n\n```\nSystem Prompt\n     |\n     | \"Who are you?\"\n     | \"How should you behave?\"\n     |\n     v\n   LLM\n     ^\n     |\n     | \"What does the user want?\"\n     |\nUser Prompt\n```\n\nThis separation becomes extremely useful when building production AI applications.\n\nPutting everything inside the controller is not a good architecture.\n\nInstead:\n\n```\nController\n    |\n    v\nService\n    |\n    v\nChatClient\n    |\n    v\nOllama\n```\n\nCreate:\n\n``` python\npackage com.example.localai.service;\n\nimport org.springframework.ai.chat.client.ChatClient;\nimport org.springframework.stereotype.Service;\n\n@Service\npublic class AiService {\n\n    private final ChatClient chatClient;\n\n    public AiService(ChatClient chatClient) {\n        this.chatClient = chatClient;\n    }\n\n    public String ask(String question) {\n\n        return chatClient\n                .prompt()\n                .user(question)\n                .call()\n                .content();\n    }\n}\n```\n\nThen the controller becomes:\n\n``` python\npackage com.example.localai.controller;\n\nimport com.example.localai.service.AiService;\nimport org.springframework.web.bind.annotation.*;\n\n@RestController\n@RequestMapping(\"/api/ai\")\npublic class AiController {\n\n    private final AiService aiService;\n\n    public AiController(AiService aiService) {\n        this.aiService = aiService;\n    }\n\n    @GetMapping(\"/ask\")\n    public String ask(@RequestParam String question) {\n        return aiService.ask(question);\n    }\n}\n```\n\nThis structure is much easier to extend later.\n\nLet's make the prompt more useful.\n\n```\npublic String support(String question) {\n\n    return chatClient\n            .prompt()\n            .system(\"\"\"\n                    You are a professional customer support assistant.\n\n                    Rules:\n                    1. Use simple English.\n                    2. Be polite and helpful.\n                    3. Give step-by-step instructions when possible.\n                    4. Never invent policies.\n                    5. If information is missing, ask for clarification.\n                    \"\"\")\n            .user(question)\n            .call()\n            .content();\n}\n```\n\nNow a question like:\n\n```\nI cannot reset my password.\n```\n\ncould produce something like:\n\n```\nI'm sorry you're having trouble resetting your password.\n\nPlease try these steps:\n\n1. Open the login page.\n2. Click \"Forgot Password\".\n3. Enter your registered email.\n4. Check your email for the reset link.\n\nIf you still cannot reset your password, contact support.\n```\n\nThis is already a useful AI feature.\n\nA basic AI call is stateless.\n\nFor example:\n\n```\nUser:\nMy order is late.\n\nAI:\nPlease provide your order number.\n\nUser:\nIt is 12345.\n\nAI:\nWhat order are you referring to?\n```\n\nThe model doesn't automatically know the previous conversation unless we provide that context.\n\nIn a real application, we need conversation memory.\n\nThe architecture becomes:\n\n``` php\nUser\n |\n v\nSpring Boot\n |\n +------> Conversation Store\n |              |\n |              v\n |          Previous Messages\n |\n v\nSpring AI\n |\n v\nOllama\n |\n v\nLLM\n```\n\nDepending on your application, the conversation history can be stored in databases such as PostgreSQL or Redis.\n\nFor example:\n\n```\nconversation_id\n        |\n        v\n+----------------------+\n| User message         |\n| AI response          |\n| User message         |\n| AI response          |\n+----------------------+\n```\n\nThen the relevant history can be included when making the next model call.\n\nThis is an important step from a simple AI demo toward a production AI application.\n\nThis is where local models become especially interesting.\n\nSuppose your company has:\n\n```\nEmployee Handbook\nProduct Documentation\nCustomer FAQs\nInternal Policies\nTechnical Documentation\n```\n\nWe can build a RAG system:\n\n```\n              Documents\n                  |\n                  v\n             Text Parser\n                  |\n                  v\n               Chunking\n                  |\n                  v\n              Embeddings\n                  |\n                  v\n            Vector Database\n                  |\n                  |\nUser Question ---> Retrieval\n                  |\n                  v\n             Relevant Data\n                  |\n                  v\n             Spring AI\n                  |\n                  v\n               Ollama\n                  |\n                  v\n             Local LLM\n                  |\n                  v\n               Answer\n```\n\nNow the LLM doesn't need to know your company information beforehand.\n\nWe retrieve the relevant information and provide it as context.\n\nImagine an employee asks:\n\n```\nHow many days of annual leave can I take?\n```\n\nThe system can search the company's HR documents.\n\nSuppose the vector database retrieves:\n\n```\nEmployees receive 24 days of annual leave per calendar year.\n```\n\nSpring AI can then construct a prompt:\n\n```\nAnswer the question using only the following context.\n\nContext:\nEmployees receive 24 days of annual leave per calendar year.\n\nQuestion:\nHow many days of annual leave can I take?\n```\n\nThe local LLM generates:\n\n```\nAccording to the company policy, employees receive\n24 days of annual leave per calendar year.\n```\n\nThe complete architecture becomes:\n\n```\n                    Employee\n                       |\n                       v\n                Spring Boot API\n                       |\n                       v\n                Spring AI RAG\n                  /        \\\n                 /          \\\n                v            v\n        Vector Database    Ollama\n                |             |\n                v             v\n          Relevant Data     Local LLM\n                 \\            /\n                  \\          /\n                   v        v\n                    Answer\n```\n\nThis is much closer to a real enterprise AI architecture.\n\nImagine an organization has sensitive documents.\n\nWith a cloud-only architecture:\n\n```\nApplication\n     |\n     v\nCloud AI API\n     |\n     v\nExternal Model\n```\n\nWith a local architecture:\n\n``` php\nApplication\n     |\n     v\nInternal Infrastructure\n     |\n     +------> Vector DB\n     |\n     +------> Ollama\n                |\n                v\n             Local LLM\n```\n\nThis can be attractive for:\n\nBut again, local inference is not a complete security strategy by itself.\n\nFor chat applications, waiting for the entire response can feel slow.\n\nA better experience is:\n\n```\nAI is typing...\n\nHello\nHello, how\nHello, how can\nHello, how can I\nHello, how can I help?\n```\n\nSpring AI supports streaming responses through `Flux`\n\n.\n\nFor example:\n\n```\n@GetMapping(value = \"/stream\", produces = \"text/event-stream\")\npublic Flux<String> stream(@RequestParam String question) {\n\n    return chatClient\n            .prompt()\n            .user(question)\n            .stream()\n            .content();\n}\n```\n\nThe client can receive pieces of the response as they are generated.\n\nThis is useful for:\n\nNot every local model is good for every task.\n\nFor example:\n\n```\nSmall model\n    |\n    +-- Faster\n    +-- Less memory\n    +-- Lower hardware requirements\n\nLarge model\n    |\n    +-- Better reasoning potential\n    +-- More memory\n    +-- Higher latency\n```\n\nYour choice depends on:\n\nFor a simple local experiment, start with a relatively small model.\n\nThen benchmark larger models if your hardware allows it.\n\nA simple comparison:\n\n| Feature | Local Ollama | Cloud LLM |\n|---|---|---|\n| Internet required | Usually no | Yes |\n| API cost | No per-token cloud fee | Usually usage-based |\n| Data leaves machine | Can stay local | Sent to provider |\n| Hardware required | Yes | Mostly no |\n| Scaling | Your responsibility | Provider handles infrastructure |\n| Model choice | Open/local models | Provider-specific models |\n| Setup | More infrastructure | Usually easier |\n| Latency | Depends on hardware | Depends on network/provider |\n\nThere is no universal winner.\n\nA practical architecture may even use both.\n\nFor example:\n\n```\n                 AI Gateway\n                     |\n            +--------+--------+\n            |                 |\n            v                 v\n        Ollama            Cloud LLM\n            |                 |\n            v                 v\n       Local Model       External Model\n```\n\nYou could use:\n\n```\nSensitive requests\n        |\n        v\n      Ollama\n```\n\nand:\n\n```\nComplex reasoning\n        |\n        v\n    Cloud Model\n```\n\nThe routing decision can be implemented inside your application.\n\nThis gives you more flexibility.\n\nRunning Ollama on your laptop is great for development.\n\nProduction is different.\n\nYou need to think about:\n\n```\nLoad Balancer\n      |\n      v\nSpring Boot Instances\n      |\n      v\nAI Service\n      |\n      v\nGPU-enabled inference servers\n```\n\nTrack:\n\nProtect:\n\nDo not expose an unauthenticated Ollama service directly to the public internet.\n\nAI services can fail.\n\nYour application should not assume every model call succeeds.\n\nFor example:\n\n```\npublic String ask(String question) {\n\n    try {\n        return chatClient\n                .prompt()\n                .user(question)\n                .call()\n                .content();\n\n    } catch (Exception ex) {\n        throw new RuntimeException(\n                \"AI service is currently unavailable\", ex);\n    }\n}\n```\n\nIn a production application, use a proper exception hierarchy and global exception handling rather than exposing raw exceptions.\n\nYou may also add:\n\n```\nTimeout\nRetry\nCircuit Breaker\nFallback\nRate Limiting\nObservability\n```\n\nA more realistic architecture could look like this:\n\n```\n                    Client\n                      |\n                      v\n               API Gateway\n                      |\n                      v\n              Spring Boot API\n                      |\n          +-----------+-----------+\n          |                       |\n          v                       v\n     Conversation             RAG Service\n       Service                    |\n          |                       v\n          |                 Vector Database\n          |                       |\n          +-----------+-----------+\n                      |\n                      v\n                 AI Service\n                      |\n             +--------+--------+\n             |                 |\n             v                 v\n          Ollama           Cloud LLM\n             |                 |\n             v                 v\n        Local Model      External Model\n```\n\nThis architecture allows you to evolve from a simple local experiment into a production-grade AI platform.\n\nHere is the complete service:\n\n```\n@Service\npublic class AiService {\n\n    private final ChatClient chatClient;\n\n    public AiService(ChatClient chatClient) {\n        this.chatClient = chatClient;\n    }\n\n    public String ask(String question) {\n\n        return chatClient\n                .prompt()\n                .system(\"\"\"\n                        You are a helpful AI assistant.\n                        Answer in simple English.\n                        Do not invent facts.\n                        \"\"\")\n                .user(question)\n                .call()\n                .content();\n    }\n}\n```\n\nController:\n\n```\n@RestController\n@RequestMapping(\"/api/ai\")\npublic class AiController {\n\n    private final AiService aiService;\n\n    public AiController(AiService aiService) {\n        this.aiService = aiService;\n    }\n\n    @GetMapping(\"/ask\")\n    public String ask(@RequestParam String question) {\n        return aiService.ask(question);\n    }\n}\n```\n\nConfiguration:\n\n```\nspring:\n  ai:\n    ollama:\n      base-url: http://localhost:11434\n      chat:\n        options:\n          model: llama3.2\n```\n\nArchitecture:\n\n```\nHTTP Client\n    |\n    v\nAiController\n    |\n    v\nAiService\n    |\n    v\nChatClient\n    |\n    v\nSpring AI\n    |\n    v\nOllama\n    |\n    v\nLocal LLM\n```\n\nThat's enough to build your first local AI backend.\n\nStart Ollama:\n\n```\nollama run llama3.2\n```\n\nThen start Spring Boot:\n\n```\n./mvnw spring-boot:run\n```\n\nOn Windows:\n\n```\nmvnw.cmd spring-boot:run\n```\n\nThen call:\n\n```\nGET http://localhost:8080/api/ai/ask?question=Explain dependency injection in Spring Boot\n```\n\nYou should receive an answer generated by your local model.\n\nIn this article, we built a local AI backend using:\n\n```\nJava\n   +\nSpring Boot\n   +\nSpring AI\n   +\nOllama\n   +\nLocal LLM\n```\n\nWe learned:\n\n`ChatClient`\n\nRunning an LLM locally changes the way we think about AI applications.\n\nYou don't always need to start with an expensive cloud API.\n\nYou can start on your own laptop:\n\n```\nOllama\n   |\nLocal LLM\n   |\nSpring AI\n   |\nSpring Boot\n   |\nREST API\n```\n\nThen gradually evolve it:\n\n```\nLocal LLM\n    ↓\nRAG\n    ↓\nVector Database\n    ↓\nConversation Memory\n    ↓\nTool Calling\n    ↓\nAI Agents\n    ↓\nObservability\n    ↓\nProduction AI Platform\n```\n\nAnd this is where **Spring AI becomes interesting for Java developers**.\n\nYou can use the Spring ecosystem you already know and add modern AI capabilities without completely changing how you build backend applications.\n\nIn the next part, we can go one step further and build a **RAG application with Spring AI, embeddings, PostgreSQL + pgvector, and a local Ollama model**.\n\nThat is where our simple chatbot starts becoming a real AI application.", "url": "https://wpnews.pro/news/run-local-llms-with-ollama-and-spring-ai", "canonical_source": "https://dev.to/ayshriv/run-local-llms-with-ollama-and-spring-ai-36l3", "published_at": "2026-08-20 06:14:52+00:00", "updated_at": "2026-08-20 06:43:18.233734+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-products"], "entities": ["Ollama", "Spring AI", "Spring Boot", "Llama", "Qwen", "OpenAI", "Anthropic"], "alternates": {"html": "https://wpnews.pro/news/run-local-llms-with-ollama-and-spring-ai", "markdown": "https://wpnews.pro/news/run-local-llms-with-ollama-and-spring-ai.md", "text": "https://wpnews.pro/news/run-local-llms-with-ollama-and-spring-ai.txt", "jsonld": "https://wpnews.pro/news/run-local-llms-with-ollama-and-spring-ai.jsonld"}}