Run AI Models Locally with Docker Model Runner and Spring AI

Docker Model Runner enables Java developers to run AI models locally and integrate them with Spring AI applications, eliminating the need for cloud-hosted model APIs during development. The tool exposes an OpenAI-compatible API, allowing Spring AI's abstractions like ChatModel and ChatClient to communicate with local models, reducing API costs and keeping data on-premises.

Generative AI development doesn't always require a cloud-hosted model. If you're building Java applications with Spring AI, you can run AI models locally and connect them to your Spring Boot application without depending on external model APIs. One interesting option is Docker Model Runner . In this article, we'll explore how Docker Model Runner works, why it is useful for Java developers, and how to connect it with Spring AI. Docker Model Runner allows you to run AI models locally using Docker. Instead of sending every prompt to a cloud provider such as OpenAI or AWS Bedrock, you can run supported models on your own machine. The architecture looks like this: Spring Boot Application | v Spring AI | v OpenAI-compatible API | v Docker Model Runner | v Local LLM Your Java application interacts with the model through an API, while Docker handles running the model locally. This gives developers a convenient way to experiment with LLM applications without immediately provisioning cloud infrastructure. There are several reasons you may want to run an LLM locally. During development, you may send hundreds or thousands of prompts. Running a model locally can eliminate API charges during experimentation. Your prompts and application data can remain on your machine instead of being sent to an external AI provider. This can be particularly useful when experimenting with sensitive or proprietary data. Once the model is available locally, you don't need an internet connection for every inference request. Developers can experiment with prompts, tool calling, RAG pipelines, and application logic without repeatedly configuring cloud credentials. One of the biggest advantages is that your application can continue using Spring AI abstractions. Your business logic doesn't need to be tightly coupled to a specific model provider. Spring AI provides abstractions such as: ChatModel ChatClient This means your application can interact with an LLM without having to directly implement provider-specific HTTP calls. The important idea is: Application | v ChatClient | v ChatModel | v Model API | v Local Model If your local model exposes an OpenAI-compatible API, Spring AI can communicate with it using the appropriate OpenAI configuration. First, make sure Docker Desktop is installed and running on your machine. Docker Model Runner is available through Docker's model functionality, depending on your Docker Desktop version and configuration. You can verify that Docker is available with: docker --version Then make sure Docker Desktop is running. Docker Desktop provides the model-running infrastructure required to run supported AI models locally. Once enabled, you can work with models directly through Docker. The exact commands and model availability can change as Docker's model ecosystem evolves, so check the current Docker documentation for the model you want to use. The important concept for our Spring AI application is that Docker Model Runner exposes an API endpoint that our application can communicate with. Create a Spring Boot application with Spring AI. For Maven, add the Spring AI OpenAI starter: <dependency <groupId org.springframework.ai</groupId <artifactId spring-ai-starter-model-openai</artifactId </dependency The reason we're using the OpenAI starter is not because we're calling OpenAI's cloud service. We're using the OpenAI-compatible API supported by the local model runtime. This is an important concept: An OpenAI-compatible API does not necessarily mean you're using OpenAI's infrastructure. It simply means the API follows a compatible request/response format. In application.properties , configure the OpenAI base URL to point to your local Docker Model Runner endpoint. For example: spring.ai.openai.base-url=http://localhost:<model-runner-port spring.ai.openai.api-key=dummy spring.ai.openai.chat.options.model=<your-local-model The exact endpoint and model name depend on your Docker Model Runner setup. The API key may not actually be required by the local runtime, but the Spring AI OpenAI client expects the configuration property, so a placeholder value can be used when appropriate. Now we can create a ChatClient . @RestController @RequestMapping "/ai" public class AIController { private final ChatClient chatClient; public AIController ChatClient.Builder builder { this.chatClient = builder.build ; } @GetMapping "/chat" public String chat @RequestParam String message { return chatClient .prompt .user message .call .content ; } } Now your Spring Boot application has a simple endpoint: GET /ai/chat?message=Explain dependency injection in Spring The flow becomes: HTTP Request | v Spring Boot | v ChatClient | v Spring AI | v Docker Model Runner | v Local LLM | v Generated Response This is where Spring AI becomes useful. Your controller doesn't need to know whether the model is: The application interacts with the Spring AI abstraction. For example: chatClient .prompt .user "Explain Spring Boot dependency injection" .call .content ; The application focuses on what it wants from the model , rather than implementing the underlying model communication itself. Running a model locally doesn't mean local models are always better. There are trade-offs. | Local Models | Cloud Models | |---|---| | Data stays locally | Data sent to provider | | No per-request API cost | Usually usage-based pricing | | Requires local compute | Provider handles infrastructure | | Potentially slower | Often faster | | Limited by local hardware | Access to larger models | | Useful for development | Useful for production workloads | For example, a developer laptop may be perfectly capable of running a smaller model. But running a large frontier model locally may require significantly more memory and compute. So the right question isn't: "Local or cloud?" It is: "Which model deployment strategy fits this workload?" Docker Model Runner is particularly interesting for developers who already use Docker as part of their development workflow. You can think of it as bringing model execution closer to the rest of your local development environment. Instead of: php Spring Boot | +---- OpenAI | +---- AWS you can have: Docker Development Environment | +---- Spring Boot | +---- Database | +---- Redis | +---- AI Model This can make local AI application development much easier to reproduce. Once you move beyond a simple chatbot, the architecture becomes more interesting. For example: Spring Boot | Spring AI | +---------+---------+ | | ChatClient Embeddings | | v v Docker Model Runner Vector Store | v Local LLM This architecture can support applications involving: And because the application uses Spring AI abstractions, you can change parts of the architecture later. One of the biggest benefits of local models is actually developer experimentation . Imagine you're building a RAG application. You need to test: Documents ↓ Chunking ↓ Embeddings ↓ Vector Store ↓ Retrieval ↓ Prompt ↓ LLM During development, you may repeatedly modify prompts and retrieval strategies. Having a local model can make that experimentation cheaper and easier. Later, when you're ready for production, you can evaluate whether a managed model provider makes more sense. This is where architecture decisions become important. A local model running on a developer laptop is obviously different from a production AI infrastructure setup. For production, you need to think about: For some workloads, a managed cloud model will be the better option. For others, self-hosted inference can make sense because of privacy, compliance, cost, or latency requirements. The important thing is that your application architecture should avoid unnecessary coupling to one model provider. This is one of the reasons I like the Spring AI abstraction. Your business code can work at a higher level: ChatClient .prompt .user prompt .call .content ; Instead of manually implementing: HTTP request ↓ Authentication ↓ JSON serialization ↓ Provider API ↓ Response parsing Spring AI handles much of that integration layer. That allows Java developers to focus on building the actual AI application. Docker Model Runner gives Java developers another option for experimenting with generative AI locally. Combined with Spring AI, the architecture becomes relatively straightforward: Spring Boot ↓ Spring AI ↓ ChatClient ↓ OpenAI-Compatible API ↓ Docker Model Runner ↓ Local LLM The bigger lesson isn't simply how to run one particular model. It's understanding model portability . Your application shouldn't necessarily care whether the underlying model is running locally or in the cloud. Spring AI gives you abstractions that help separate your application logic from the model provider. And that's an important foundation for building production-ready AI applications. In Part 6, we'll move from local inference to the cloud and explore AWS Bedrock with Spring AI . We'll look at how Spring Boot applications can interact with foundation models available through AWS and how the architecture changes when AI inference moves from your local machine to a managed cloud platform.