{"slug": "run-ai-models-locally-with-docker-model-runner-and-spring-ai", "title": "Run AI Models Locally with Docker Model Runner and Spring AI", "summary": "Docker Model Runner enables Java developers to run AI models locally and integrate them with Spring AI applications, eliminating the need for cloud-hosted model APIs during development. The tool exposes an OpenAI-compatible API, allowing Spring AI's abstractions like ChatModel and ChatClient to communicate with local models, reducing API costs and keeping data on-premises.", "body_md": "Generative AI development doesn't always require a cloud-hosted model.\n\nIf you're building Java applications with Spring AI, you can run AI models locally and connect them to your Spring Boot application without depending on external model APIs.\n\nOne interesting option is **Docker Model Runner**.\n\nIn this article, we'll explore how Docker Model Runner works, why it is useful for Java developers, and how to connect it with Spring AI.\n\nDocker Model Runner allows you to run AI models locally using Docker.\n\nInstead of sending every prompt to a cloud provider such as OpenAI or AWS Bedrock, you can run supported models on your own machine.\n\nThe architecture looks like this:\n\n```\nSpring Boot Application\n        |\n        v\n     Spring AI\n        |\n        v\n   OpenAI-compatible API\n        |\n        v\n Docker Model Runner\n        |\n        v\n     Local LLM\n```\n\nYour Java application interacts with the model through an API, while Docker handles running the model locally.\n\nThis gives developers a convenient way to experiment with LLM applications without immediately provisioning cloud infrastructure.\n\nThere are several reasons you may want to run an LLM locally.\n\nDuring development, you may send hundreds or thousands of prompts.\n\nRunning a model locally can eliminate API charges during experimentation.\n\nYour prompts and application data can remain on your machine instead of being sent to an external AI provider.\n\nThis can be particularly useful when experimenting with sensitive or proprietary data.\n\nOnce the model is available locally, you don't need an internet connection for every inference request.\n\nDevelopers can experiment with prompts, tool calling, RAG pipelines, and application logic without repeatedly configuring cloud credentials.\n\nOne of the biggest advantages is that your application can continue using Spring AI abstractions.\n\nYour business logic doesn't need to be tightly coupled to a specific model provider.\n\nSpring AI provides abstractions such as:\n\n```\nChatModel\nChatClient\n```\n\nThis means your application can interact with an LLM without having to directly implement provider-specific HTTP calls.\n\nThe important idea is:\n\n```\nApplication\n     |\n     v\n ChatClient\n     |\n     v\n ChatModel\n     |\n     v\n Model API\n     |\n     v\nLocal Model\n```\n\nIf your local model exposes an OpenAI-compatible API, Spring AI can communicate with it using the appropriate OpenAI configuration.\n\nFirst, make sure Docker Desktop is installed and running on your machine.\n\nDocker Model Runner is available through Docker's model functionality, depending on your Docker Desktop version and configuration.\n\nYou can verify that Docker is available with:\n\n```\ndocker --version\n```\n\nThen make sure Docker Desktop is running.\n\nDocker Desktop provides the model-running infrastructure required to run supported AI models locally.\n\nOnce enabled, you can work with models directly through Docker.\n\nThe exact commands and model availability can change as Docker's model ecosystem evolves, so check the current Docker documentation for the model you want to use.\n\nThe important concept for our Spring AI application is that Docker Model Runner exposes an API endpoint that our application can communicate with.\n\nCreate a Spring Boot application with Spring AI.\n\nFor Maven, add the Spring AI OpenAI starter:\n\n```\n<dependency>\n    <groupId>org.springframework.ai</groupId>\n    <artifactId>spring-ai-starter-model-openai</artifactId>\n</dependency>\n```\n\nThe reason we're using the OpenAI starter is not because we're calling OpenAI's cloud service.\n\nWe're using the OpenAI-compatible API supported by the local model runtime.\n\nThis is an important concept:\n\nAn OpenAI-compatible API does not necessarily mean you're using OpenAI's infrastructure.\n\nIt simply means the API follows a compatible request/response format.\n\nIn `application.properties`\n\n, configure the OpenAI base URL to point to your local Docker Model Runner endpoint.\n\nFor example:\n\n```\nspring.ai.openai.base-url=http://localhost:<model-runner-port>\nspring.ai.openai.api-key=dummy\nspring.ai.openai.chat.options.model=<your-local-model>\n```\n\nThe exact endpoint and model name depend on your Docker Model Runner setup.\n\nThe API key may not actually be required by the local runtime, but the Spring AI OpenAI client expects the configuration property, so a placeholder value can be used when appropriate.\n\nNow we can create a `ChatClient`\n\n.\n\n```\n@RestController\n@RequestMapping(\"/ai\")\npublic class AIController {\n\n    private final ChatClient chatClient;\n\n    public AIController(ChatClient.Builder builder) {\n        this.chatClient = builder.build();\n    }\n\n    @GetMapping(\"/chat\")\n    public String chat(@RequestParam String message) {\n\n        return chatClient\n                .prompt()\n                .user(message)\n                .call()\n                .content();\n    }\n}\n```\n\nNow your Spring Boot application has a simple endpoint:\n\n```\nGET /ai/chat?message=Explain dependency injection in Spring\n```\n\nThe flow becomes:\n\n```\nHTTP Request\n     |\n     v\nSpring Boot\n     |\n     v\nChatClient\n     |\n     v\nSpring AI\n     |\n     v\nDocker Model Runner\n     |\n     v\nLocal LLM\n     |\n     v\nGenerated Response\n```\n\nThis is where Spring AI becomes useful.\n\nYour controller doesn't need to know whether the model is:\n\nThe application interacts with the Spring AI abstraction.\n\nFor example:\n\n```\nchatClient\n        .prompt()\n        .user(\"Explain Spring Boot dependency injection\")\n        .call()\n        .content();\n```\n\nThe application focuses on **what it wants from the model**, rather than implementing the underlying model communication itself.\n\nRunning a model locally doesn't mean local models are always better.\n\nThere are trade-offs.\n\n| Local Models | Cloud Models |\n|---|---|\n| Data stays locally | Data sent to provider |\n| No per-request API cost | Usually usage-based pricing |\n| Requires local compute | Provider handles infrastructure |\n| Potentially slower | Often faster |\n| Limited by local hardware | Access to larger models |\n| Useful for development | Useful for production workloads |\n\nFor example, a developer laptop may be perfectly capable of running a smaller model.\n\nBut running a large frontier model locally may require significantly more memory and compute.\n\nSo the right question isn't:\n\n**\"Local or cloud?\"**\n\nIt is:\n\n**\"Which model deployment strategy fits this workload?\"**\n\nDocker Model Runner is particularly interesting for developers who already use Docker as part of their development workflow.\n\nYou can think of it as bringing model execution closer to the rest of your local development environment.\n\nInstead of:\n\n``` php\nSpring Boot\n     |\n     +----> OpenAI\n     |\n     +----> AWS\n```\n\nyou can have:\n\n```\nDocker Development Environment\n        |\n        +---- Spring Boot\n        |\n        +---- Database\n        |\n        +---- Redis\n        |\n        +---- AI Model\n```\n\nThis can make local AI application development much easier to reproduce.\n\nOnce you move beyond a simple chatbot, the architecture becomes more interesting.\n\nFor example:\n\n```\n             Spring Boot\n                  |\n             Spring AI\n                  |\n        +---------+---------+\n        |                   |\n    ChatClient          Embeddings\n        |                   |\n        v                   v\nDocker Model Runner     Vector Store\n        |\n        v\n     Local LLM\n```\n\nThis architecture can support applications involving:\n\nAnd because the application uses Spring AI abstractions, you can change parts of the architecture later.\n\nOne of the biggest benefits of local models is actually **developer experimentation**.\n\nImagine you're building a RAG application.\n\nYou need to test:\n\n```\nDocuments\n    ↓\nChunking\n    ↓\nEmbeddings\n    ↓\nVector Store\n    ↓\nRetrieval\n    ↓\nPrompt\n    ↓\nLLM\n```\n\nDuring development, you may repeatedly modify prompts and retrieval strategies.\n\nHaving a local model can make that experimentation cheaper and easier.\n\nLater, when you're ready for production, you can evaluate whether a managed model provider makes more sense.\n\nThis is where architecture decisions become important.\n\nA local model running on a developer laptop is obviously different from a production AI infrastructure setup.\n\nFor production, you need to think about:\n\nFor some workloads, a managed cloud model will be the better option.\n\nFor others, self-hosted inference can make sense because of privacy, compliance, cost, or latency requirements.\n\nThe important thing is that your application architecture should avoid unnecessary coupling to one model provider.\n\nThis is one of the reasons I like the Spring AI abstraction.\n\nYour business code can work at a higher level:\n\n```\nChatClient\n    .prompt()\n    .user(prompt)\n    .call()\n    .content();\n```\n\nInstead of manually implementing:\n\n```\nHTTP request\n    ↓\nAuthentication\n    ↓\nJSON serialization\n    ↓\nProvider API\n    ↓\nResponse parsing\n```\n\nSpring AI handles much of that integration layer.\n\nThat allows Java developers to focus on building the actual AI application.\n\nDocker Model Runner gives Java developers another option for experimenting with generative AI locally.\n\nCombined with Spring AI, the architecture becomes relatively straightforward:\n\n```\nSpring Boot\n     ↓\nSpring AI\n     ↓\nChatClient\n     ↓\nOpenAI-Compatible API\n     ↓\nDocker Model Runner\n     ↓\nLocal LLM\n```\n\nThe bigger lesson isn't simply how to run one particular model.\n\nIt's understanding **model portability**.\n\nYour application shouldn't necessarily care whether the underlying model is running locally or in the cloud.\n\nSpring AI gives you abstractions that help separate your application logic from the model provider.\n\nAnd that's an important foundation for building production-ready AI applications.\n\nIn Part 6, we'll move from local inference to the cloud and explore **AWS Bedrock with Spring AI**.\n\nWe'll look at how Spring Boot applications can interact with foundation models available through AWS and how the architecture changes when AI inference moves from your local machine to a managed cloud platform.", "url": "https://wpnews.pro/news/run-ai-models-locally-with-docker-model-runner-and-spring-ai", "canonical_source": "https://dev.to/ayshriv/run-ai-models-locally-with-docker-model-runner-and-spring-ai-2god", "published_at": "2026-08-23 08:22:51+00:00", "updated_at": "2026-08-23 08:43:19.284290+00:00", "lang": "en", "topics": ["developer-tools", "generative-ai", "artificial-intelligence"], "entities": ["Docker", "Spring AI", "OpenAI", "AWS Bedrock"], "alternates": {"html": "https://wpnews.pro/news/run-ai-models-locally-with-docker-model-runner-and-spring-ai", "markdown": "https://wpnews.pro/news/run-ai-models-locally-with-docker-model-runner-and-spring-ai.md", "text": "https://wpnews.pro/news/run-ai-models-locally-with-docker-model-runner-and-spring-ai.txt", "jsonld": "https://wpnews.pro/news/run-ai-models-locally-with-docker-model-runner-and-spring-ai.jsonld"}}