{"slug": "langchain4j-and-spring-ai-the-plumbing-to-make-your-java-apps-talk-to-llms", "title": "LangChain4j and Spring AI: The Plumbing to make your Java Apps talk to LLMs", "summary": "LangChain4j and Spring AI bring LLM integration to Java, providing reusable components for managing conversation history, document splitting, embeddings, vector search, and function calling. Spring AI 2.0, released in June, auto-configures with Spring Boot, allowing developers to call an LLM in about six lines of code without a Python sidecar. The libraries enable structured data extraction, grounding answers in internal documentation, and triggering real code from model outputs.", "body_md": "If you've heard about LangChain and assumed it was a Python thing, that's fair. It mostly was.\n\nLangChain became popular because building with an LLM turns out to involve a lot of repetitive plumbing. You need to manage conversation history, split documents into chunks, generate embeddings, search a vector store, wire up functions the model can call, and parse whatever comes back. None of that is hard, but writing it from scratch for every project gets old fast. LangChain packaged those pieces into reusable components, and the pattern caught on.\n\nThe Java ecosystem has that now too, in two flavors. **LangChain4j** is a Java library built around the same idea, though it was written for Java from the ground up rather than ported over. **Spring AI** does the same job the Spring way, with auto-configuration and dependency injection, and it hit 2.0 this June.\n\nBoth are production ready. Your existing Spring Boot service can call an LLM in about six lines, and you don't need a Python sidecar or a separate service to do it.\n\nThe six lines aren't the interesting part, though. What matters is the distance between a chat endpoint that echoes text back and something you'd actually ship: getting typed objects instead of strings, grounding answers in your own documentation, and letting a model trigger real code in your app.\n\n**That's what this post covers.** We'll build up from hello-world to a service that answers questions about your internal docs and can call your APIs, one step at a time. I'll use Spring AI for the walkthrough since most of us are already in a Boot service, then show what the same thing looks like in LangChain4j so you can pick.\n\nI'm assuming you know Java and Spring Boot, and nothing about AI. No math, no theory, just the parts you need to build something.\n\nOne idea first, because it makes everything else fall into place.\n\nAn LLM is a stateless function.Text in, text out. It doesn't remember your last call, can't reach the internet, and knows nothing about your systems.\n\nEverything below is a workaround for that:\n\nThe model never *does* anything. Your code does everything. The model produces text, and sometimes that text is a decision about what your code should do next.\n\nThat one idea demystifies most of this space. Everything from here is plumbing around a stateless function, and plumbing is something we're already good at.\n\nIf you want more background on how LLMs, RAG, and agents fit together before diving in, I wrote about that here:\n\nSpring AI 2.0 needs Spring Boot 4.0+ and Java 17+.\n\n```\n<dependency>\n  <groupId>org.springframework.ai</groupId>\n  <artifactId>spring-ai-starter-model-openai</artifactId>\n</dependency>\nspring.ai.openai.api-key=${OPENAI_API_KEY}\nspring.ai.openai.chat.model=gpt-4o-mini\n@RestController\nclass ChatController {\n\n    private final ChatClient chatClient;\n\n    ChatController(ChatClient.Builder builder) {\n        this.chatClient = builder.build();\n    }\n\n    @GetMapping(\"/chat\")\n    String chat(@RequestParam String message) {\n        return chatClient.prompt()\n            .user(message)\n            .call()\n            .content();\n    }\n}\n```\n\nThat's a working endpoint. Spring Boot auto-configured the `ChatClient.Builder`\n\nthe same way it gives you a `JdbcTemplate`\n\n, so there's nothing new to learn about the Spring part.\n\nThis is where most first attempts fall apart. You ask for structured data and get back a paragraph. Or JSON wrapped in markdown fences. Or JSON with a chatty preamble in front of it. So you write a parser, then a fallback parser, then a regex. It gets miserable quickly.\n\nAsk for a type instead:\n\n```\nrecord ActionItem(String task, String owner, String dueDate) {}\nrecord MeetingNotes(String summary, List<ActionItem> actionItems) {}\nMeetingNotes notes = chatClient.prompt()\n    .user(\"Extract the summary and action items:\\n\" + transcript)\n    .call()\n    .entity(MeetingNotes.class);\n```\n\n`.entity()`\n\nderives a JSON schema from your record, tells the model to conform, and deserializes the response. You get an object. You can pass it around, test it, persist it.\n\nThis is the highest-leverage feature in the framework. It's what turns an LLM demo into a component you can put in a real system.\n\nOne thing that surprises people: name your fields clearly. `dueDate`\n\ngets better results than `d2`\n\n, because the schema you generate becomes part of the prompt the model sees.\n\nAsk \"what's our rollback procedure?\" and the model will confidently invent one. The fix is unglamorous. You find the relevant docs and paste them into the prompt.\n\nThat's RAG (Retrieval-Augmented Generation). The name sounds architectural, but it's really a paste operation with a good search in front of it.\n\nThe only interesting part is the search. Keyword matching is too brittle here, because the user asks about \"rollback\" while your runbook says \"reverting a bad deploy.\" So instead you use **embeddings**. Each chunk of text gets converted into a vector that represents its meaning, and similar meanings end up near each other. Now those two phrases match even though they share no words.\n\nLoad your docs once:\n\n``` js\nvar reader = new TextReader(runbook);\nvar splitter = new TokenTextSplitter();\nvectorStore.add(splitter.apply(reader.get()));\n```\n\nWire retrieval in:\n\n```\n@Bean\nChatClient ragChatClient(ChatClient.Builder builder, VectorStore vectorStore) {\n    return builder\n        .defaultAdvisors(new QuestionAnswerAdvisor(vectorStore))\n        .build();\n}\n```\n\nAsk:\n\n```\nString answer = ragChatClient.prompt()\n    .user(\"What's our deployment rollback procedure?\")\n    .call()\n    .content();\n```\n\n**Notice the calling code is identical to Step 1.** Nothing about it says \"RAG.\" The `QuestionAnswerAdvisor`\n\nsits in the middle and does the work: it intercepts the request, embeds the question, searches the store, injects the matches into the prompt, then passes it along.\n\nThat's the **Advisor** pattern, and it's the core idea in Spring AI. If you've ever written a servlet `Filter`\n\nor a `HandlerInterceptor`\n\n, you already know the shape. Memory, retries, and tool calling all work the same way.\n\nRAG can only surface documents you loaded ahead of time. It can't tell you whether an order shipped, because that's a live lookup rather than a document. For that you need **tool calling**.\n\nThe name is a little misleading, because **the model does not call your tool**. Here's what actually happens:\n\n`getOrderStatus`\n\nwith `orderId=A1234`\n\n.\"The model decides, your code does. In practice Spring AI hides steps 2 through 4 from you. You just write the method:\n\n```\n@Component\nclass OrderTools {\n\n    @Tool(description = \"Look up the current status of a customer order by its ID\")\n    String getOrderStatus(@ToolParam(description = \"The order ID, e.g. A1234\") String orderId) {\n        return orders.findById(orderId).map(Order::status).orElse(\"Not found\");\n    }\n}\nString answer = chatClient.prompt()\n    .user(\"Has order A1234 shipped yet?\")\n    .tools(orderTools)\n    .call()\n    .content();\n```\n\nThe description is the API.It isn't documentation for humans. It's the only thing the model uses to decide whether your method is the right one to call, so a vague description gets you wrong calls.\n\nWorth knowing if you're reading older tutorials: Spring AI 2.0 moved the tool-calling loop out of the individual chat models and into the advisor chain. In 1.x you could call tools, but you couldn't build on top of the loop itself. Now you can intercept and compose around it, which matters a lot if you're building agents.\n\nYou'll hear a lot about MCP (Model Context Protocol) right now, and it gets conflated with tool calling constantly. The distinction is simple: **tool calling is the capability, MCP is a delivery mechanism.** Everything in Step 4 works without MCP.\n\nMCP answers a different question: what if that tool should be available to other apps too? Instead of every team hardcoding their own version of the same integration, the tool lives in a standalone server that any MCP-compatible client can connect to. It's a standard interface, closer to USB than to a new kind of electricity.\n\nIf you're building one app with a handful of your own tools, skip it and use Step 4. If you want to reuse tools across services, or plug into the growing ecosystem of pre-built servers, that's when MCP earns its place.\n\nEverything above works in LangChain4j too. The difference is philosophical: Spring AI is opinionated about composition, so everything flows through the advisor chain and Spring wires it up for you. LangChain4j hands you independent building blocks and lets you assemble them yourself.\n\nIts best feature is **AI Services**. You declare an interface and it generates the implementation:\n\n```\nSupportAssistant assistant = AiServices.builder(SupportAssistant.class)\n    .chatModel(model)\n    .contentRetriever(retriever)     // RAG\n    .tools(new OrderTools(orders))   // tool calling\n    .chatMemory(MessageWindowChatMemory.withMaxMessages(10))\n    .build();\n```\n\nThat's all four steps in a single builder. If you've used Spring Data repositories or Feign clients, the pattern needs no explanation: you describe what you want in a typed interface, and the library handles the rest.\n\n| Spring AI | LangChain4j | |\n|---|---|---|\nBest when |\nYou're already on Spring Boot | Quarkus, Micronaut, or plain Java |\nStyle |\nAuto-configured, opinionated | Assemble it yourself |\nCore idea |\nAdvisor chain | AI Services |\nObservability |\nMicrometer built in | Bring your own |\n\n**Use LangChain4j** if you're not on Spring, or if you'd rather own the composition yourself. **Use Spring AI** if you're already in a Boot service and want auto-configuration, Micrometer observability, and the advisor chain to build on.\n\nBoth are good. This isn't a decision worth agonizing over, so pick the one that matches the stack you're already in.\n\n**Token usage matters more than framework overhead.** Network latency to the model dwarfs any abstraction cost, so don't pick a framework on performance. What does add up is tokens. Both frameworks quietly append things to your requests: memory, RAG chunks, tool definitions. Log what's actually going out before you scale.\n\n**You probably want RAG, not fine-tuning.** Use RAG when the model needs to *know* something, like your docs or current data. Use fine-tuning when it needs to *behave* a certain way, like matching a tone or format. Most people reaching for fine-tuning actually want RAG, which is cheaper and updates by writing to a database.\n\nGet a `ChatClient`\n\nreturning a string. That's an afternoon. Then make it return a record instead, which is the moment it stops being a toy. Then add one tool. Then add RAG.\n\nEach step is genuinely small, and the libraries are good enough now that the plumbing mostly disappears. What's left is the interesting part: deciding what your system should actually do.\n\n⚠️\n\nHeads-up:Spring AI 2.0 was a real redesign with breaking changes from 1.x, and a lot of tutorials you'll find online are still written against 1.x. Check which version you're reading before you copy anything.", "url": "https://wpnews.pro/news/langchain4j-and-spring-ai-the-plumbing-to-make-your-java-apps-talk-to-llms", "canonical_source": "https://dev.to/shayesta/langchain4j-and-spring-ai-the-plumbing-to-make-your-java-apps-talk-to-llms-1ip8", "published_at": "2026-07-17 00:58:31+00:00", "updated_at": "2026-07-17 01:32:42.968041+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-tools"], "entities": ["LangChain4j", "Spring AI", "Spring Boot", "OpenAI", "GPT-4o-mini"], "alternates": {"html": "https://wpnews.pro/news/langchain4j-and-spring-ai-the-plumbing-to-make-your-java-apps-talk-to-llms", "markdown": "https://wpnews.pro/news/langchain4j-and-spring-ai-the-plumbing-to-make-your-java-apps-talk-to-llms.md", "text": "https://wpnews.pro/news/langchain4j-and-spring-ai-the-plumbing-to-make-your-java-apps-talk-to-llms.txt", "jsonld": "https://wpnews.pro/news/langchain4j-and-spring-ai-the-plumbing-to-make-your-java-apps-talk-to-llms.jsonld"}}