LangChain4j and Spring AI: The Plumbing to make your Java Apps talk to LLMs

LangChain4j and Spring AI bring LLM integration to Java, providing reusable components for managing conversation history, document splitting, embeddings, vector search, and function calling. Spring AI 2.0, released in June, auto-configures with Spring Boot, allowing developers to call an LLM in about six lines of code without a Python sidecar. The libraries enable structured data extraction, grounding answers in internal documentation, and triggering real code from model outputs.

If you've heard about LangChain and assumed it was a Python thing, that's fair. It mostly was. LangChain became popular because building with an LLM turns out to involve a lot of repetitive plumbing. You need to manage conversation history, split documents into chunks, generate embeddings, search a vector store, wire up functions the model can call, and parse whatever comes back. None of that is hard, but writing it from scratch for every project gets old fast. LangChain packaged those pieces into reusable components, and the pattern caught on. The Java ecosystem has that now too, in two flavors. LangChain4j is a Java library built around the same idea, though it was written for Java from the ground up rather than ported over. Spring AI does the same job the Spring way, with auto-configuration and dependency injection, and it hit 2.0 this June. Both are production ready. Your existing Spring Boot service can call an LLM in about six lines, and you don't need a Python sidecar or a separate service to do it. The six lines aren't the interesting part, though. What matters is the distance between a chat endpoint that echoes text back and something you'd actually ship: getting typed objects instead of strings, grounding answers in your own documentation, and letting a model trigger real code in your app. That's what this post covers. We'll build up from hello-world to a service that answers questions about your internal docs and can call your APIs, one step at a time. I'll use Spring AI for the walkthrough since most of us are already in a Boot service, then show what the same thing looks like in LangChain4j so you can pick. I'm assuming you know Java and Spring Boot, and nothing about AI. No math, no theory, just the parts you need to build something. One idea first, because it makes everything else fall into place. An LLM is a stateless function.Text in, text out. It doesn't remember your last call, can't reach the internet, and knows nothing about your systems. Everything below is a workaround for that: The model never does anything. Your code does everything. The model produces text, and sometimes that text is a decision about what your code should do next. That one idea demystifies most of this space. Everything from here is plumbing around a stateless function, and plumbing is something we're already good at. If you want more background on how LLMs, RAG, and agents fit together before diving in, I wrote about that here: Spring AI 2.0 needs Spring Boot 4.0+ and Java 17+. <dependency <groupId org.springframework.ai</groupId <artifactId spring-ai-starter-model-openai</artifactId </dependency spring.ai.openai.api-key=${OPENAI API KEY} spring.ai.openai.chat.model=gpt-4o-mini @RestController class ChatController { private final ChatClient chatClient; ChatController ChatClient.Builder builder { this.chatClient = builder.build ; } @GetMapping "/chat" String chat @RequestParam String message { return chatClient.prompt .user message .call .content ; } } That's a working endpoint. Spring Boot auto-configured the ChatClient.Builder the same way it gives you a JdbcTemplate , so there's nothing new to learn about the Spring part. This is where most first attempts fall apart. You ask for structured data and get back a paragraph. Or JSON wrapped in markdown fences. Or JSON with a chatty preamble in front of it. So you write a parser, then a fallback parser, then a regex. It gets miserable quickly. Ask for a type instead: record ActionItem String task, String owner, String dueDate {} record MeetingNotes String summary, List<ActionItem actionItems {} MeetingNotes notes = chatClient.prompt .user "Extract the summary and action items:\n" + transcript .call .entity MeetingNotes.class ; .entity derives a JSON schema from your record, tells the model to conform, and deserializes the response. You get an object. You can pass it around, test it, persist it. This is the highest-leverage feature in the framework. It's what turns an LLM demo into a component you can put in a real system. One thing that surprises people: name your fields clearly. dueDate gets better results than d2 , because the schema you generate becomes part of the prompt the model sees. Ask "what's our rollback procedure?" and the model will confidently invent one. The fix is unglamorous. You find the relevant docs and paste them into the prompt. That's RAG Retrieval-Augmented Generation . The name sounds architectural, but it's really a paste operation with a good search in front of it. The only interesting part is the search. Keyword matching is too brittle here, because the user asks about "rollback" while your runbook says "reverting a bad deploy." So instead you use embeddings . Each chunk of text gets converted into a vector that represents its meaning, and similar meanings end up near each other. Now those two phrases match even though they share no words. Load your docs once: js var reader = new TextReader runbook ; var splitter = new TokenTextSplitter ; vectorStore.add splitter.apply reader.get ; Wire retrieval in: @Bean ChatClient ragChatClient ChatClient.Builder builder, VectorStore vectorStore { return builder .defaultAdvisors new QuestionAnswerAdvisor vectorStore .build ; } Ask: String answer = ragChatClient.prompt .user "What's our deployment rollback procedure?" .call .content ; Notice the calling code is identical to Step 1. Nothing about it says "RAG." The QuestionAnswerAdvisor sits in the middle and does the work: it intercepts the request, embeds the question, searches the store, injects the matches into the prompt, then passes it along. That's the Advisor pattern, and it's the core idea in Spring AI. If you've ever written a servlet Filter or a HandlerInterceptor , you already know the shape. Memory, retries, and tool calling all work the same way. RAG can only surface documents you loaded ahead of time. It can't tell you whether an order shipped, because that's a live lookup rather than a document. For that you need tool calling . The name is a little misleading, because the model does not call your tool . Here's what actually happens: getOrderStatus with orderId=A1234 ."The model decides, your code does. In practice Spring AI hides steps 2 through 4 from you. You just write the method: @Component class OrderTools { @Tool description = "Look up the current status of a customer order by its ID" String getOrderStatus @ToolParam description = "The order ID, e.g. A1234" String orderId { return orders.findById orderId .map Order::status .orElse "Not found" ; } } String answer = chatClient.prompt .user "Has order A1234 shipped yet?" .tools orderTools .call .content ; The description is the API.It isn't documentation for humans. It's the only thing the model uses to decide whether your method is the right one to call, so a vague description gets you wrong calls. Worth knowing if you're reading older tutorials: Spring AI 2.0 moved the tool-calling loop out of the individual chat models and into the advisor chain. In 1.x you could call tools, but you couldn't build on top of the loop itself. Now you can intercept and compose around it, which matters a lot if you're building agents. You'll hear a lot about MCP Model Context Protocol right now, and it gets conflated with tool calling constantly. The distinction is simple: tool calling is the capability, MCP is a delivery mechanism. Everything in Step 4 works without MCP. MCP answers a different question: what if that tool should be available to other apps too? Instead of every team hardcoding their own version of the same integration, the tool lives in a standalone server that any MCP-compatible client can connect to. It's a standard interface, closer to USB than to a new kind of electricity. If you're building one app with a handful of your own tools, skip it and use Step 4. If you want to reuse tools across services, or plug into the growing ecosystem of pre-built servers, that's when MCP earns its place. Everything above works in LangChain4j too. The difference is philosophical: Spring AI is opinionated about composition, so everything flows through the advisor chain and Spring wires it up for you. LangChain4j hands you independent building blocks and lets you assemble them yourself. Its best feature is AI Services . You declare an interface and it generates the implementation: SupportAssistant assistant = AiServices.builder SupportAssistant.class .chatModel model .contentRetriever retriever // RAG .tools new OrderTools orders // tool calling .chatMemory MessageWindowChatMemory.withMaxMessages 10 .build ; That's all four steps in a single builder. If you've used Spring Data repositories or Feign clients, the pattern needs no explanation: you describe what you want in a typed interface, and the library handles the rest. | Spring AI | LangChain4j | | |---|---|---| Best when | You're already on Spring Boot | Quarkus, Micronaut, or plain Java | Style | Auto-configured, opinionated | Assemble it yourself | Core idea | Advisor chain | AI Services | Observability | Micrometer built in | Bring your own | Use LangChain4j if you're not on Spring, or if you'd rather own the composition yourself. Use Spring AI if you're already in a Boot service and want auto-configuration, Micrometer observability, and the advisor chain to build on. Both are good. This isn't a decision worth agonizing over, so pick the one that matches the stack you're already in. Token usage matters more than framework overhead. Network latency to the model dwarfs any abstraction cost, so don't pick a framework on performance. What does add up is tokens. Both frameworks quietly append things to your requests: memory, RAG chunks, tool definitions. Log what's actually going out before you scale. You probably want RAG, not fine-tuning. Use RAG when the model needs to know something, like your docs or current data. Use fine-tuning when it needs to behave a certain way, like matching a tone or format. Most people reaching for fine-tuning actually want RAG, which is cheaper and updates by writing to a database. Get a ChatClient returning a string. That's an afternoon. Then make it return a record instead, which is the moment it stops being a toy. Then add one tool. Then add RAG. Each step is genuinely small, and the libraries are good enough now that the plumbing mostly disappears. What's left is the interesting part: deciding what your system should actually do. ⚠️ Heads-up:Spring AI 2.0 was a real redesign with breaking changes from 1.x, and a lot of tutorials you'll find online are still written against 1.x. Check which version you're reading before you copy anything.