{"slug": "parallelization-workflow-with-spring-ai-fan-out-llm-calls-in-java", "title": "Parallelization Workflow with Spring AI: Fan-Out LLM Calls in Java", "summary": "A developer walkthrough examines Spring AI's parallelization workflow example, which fans out independent LLM calls across a configurable number of worker threads using a ChatClient and returns responses in input order. The pattern, drawn from Anthropic's \"Building Effective Agents\" guidance, supports sectioning (splitting a task into independent subtasks) and voting (repeating one prompt for consensus), while flagging costs around rate limits, per-call billing, and partial failures.", "body_md": "When people first start building with LLMs in Java, the usual shape of the code is simple: build a prompt, call the model, read the answer. That works well for a single question. It starts to hurt when you need to ask the same model the same kind of question a hundred times, or when one request is really five smaller questions that happen to be asked together. Each call waits for the network, the model generates tokens at its own pace, and your thread sits idle for most of that time.\n\nAnthropic's article \"Building Effective Agents\" describes a set of workflow patterns for composing LLM calls. One of them is the **Parallelization Workflow**: instead of running model calls one after another, you run them at the same time and then combine the results. The Spring AI team has published a small, focused example of this pattern in the `spring-ai-examples` repository, under `agentic-patterns/parallelization-workflow`. It is a compact class built on top of Spring AI's `ChatClient`, and it is a good starting point for understanding how to fan out LLM work in a Spring application.\n\nIn this article I walk through the pattern, the two variations it supports (sectioning and voting), the code in the example, and the practical issues you will face when you use it for real workloads: ordering, failures, rate limits, cost, and testing. I will keep the code close to the example so you can follow along with the repository open.\n\nA typical chat completion can take anywhere from a second to tens of seconds, depending on the model, the size of the prompt, and the length of the answer. Those numbers matter when you multiply them.\n\nImagine you need to analyze ten product reviews, one at a time, with a call that takes four seconds. That is forty seconds of wall-clock time, most of it spent waiting. If the calls are independent of one another, there is no reason to wait for review number three before starting review number four. You can send them all at once and collect the answers as they arrive.\n\nParallelization helps in three ways:\n\nIt also has costs. You pay for every call, not for every batch. You can overload a provider's rate limits. And your application has to handle partial failures, which are more complicated when several calls are in flight at once. The rest of this article is about getting the benefits without ignoring those costs.\n\nThe example README names two variations of the pattern. They solve different problems, so it helps to keep them separate in your head.\n\n**Sectioning** breaks a larger task into independent subtasks and processes each one in parallel. The subtasks do not depend on each other, so the order of execution does not matter. Each call gets a different input and, often, a slightly different instruction. A typical case is analyzing several stakeholder groups, several document sections, or several customer segments with the same analytical prompt.\n\n**Voting** runs the same prompt several times, usually against the same input, and then combines the results. The goal is diversity of perspective or a consensus. If you ask a model whether a transaction looks fraudulent five times and four answers say \"review,\" you have a more robust signal than a single answer. The aggregation step can be a majority vote, a rule that requires a minimum agreement, or another model call that summarizes the candidates.\n\nThe distinction is practical. Sectioning is about speed on a set of different problems. Voting is about confidence on one problem. You can combine them, but it is clearer to start with one and then add the other.\n\nThe core of the example is one class: `ParallelizationlWorkflow`, in the package `com.example.agentic`. It takes a `ChatClient` in its constructor and exposes a single method:\n\n```\npublic List<String> parallel(String prompt, List<String> inputs, int nWorkers)\n```\n\nThe method takes a prompt template, a list of inputs, and the number of worker threads. It returns a list of model responses in the same order as the inputs. Here is the heart of the implementation, lightly trimmed:\n\n```\nExecutorService executor = Executors.newFixedThreadPool(nWorkers);\ntry {\n    List<CompletableFuture<String>> futures = inputs.stream()\n            .map(input -> CompletableFuture.supplyAsync(() -> {\n                try {\n                    return chatClient.prompt(prompt + \"\\nInput: \" + input).call().content();\n                } catch (Exception e) {\n                    throw new RuntimeException(\"Failed to process input: \" + input, e);\n                }\n            }, executor))\n            .collect(Collectors.toList());\n\n    CompletableFuture<Void> allFutures = CompletableFuture.allOf(\n            futures.toArray(CompletableFuture[]::new));\n    allFutures.join();\n\n    return futures.stream()\n            .map(CompletableFuture::join)\n            .collect(Collectors.toList());\n} finally {\n    executor.shutdown();\n}\n```\n\nThere are a few design decisions worth noticing.\n\nFirst, the class creates a **fixed thread pool** for each call. The pool size is the `nWorkers` argument, which becomes a hard cap on how many model requests are in flight at once. This is a deliberate choice: the number of concurrent calls is an explicit input, not something you discover by accident.\n\nSecond, each input becomes a `CompletableFuture` submitted to that executor. The futures run concurrently, but they are stored in a list in the same order as the inputs. That is how the method keeps **ordered results**. The fact that a later request finishes first does not change where its answer ends up in the returned list.\n\nThird, the method waits for all futures with `CompletableFuture.allOf(...).join()` and then collects the individual results with `join()` again. Once `allOf` completes, each individual `join()` returns immediately.\n\nFourth, the input is appended to the prompt with a simple `\"\\nInput: \"` separator. This is a minimal way to build a per-item prompt from a shared instruction. In a production system you would probably use a structured template, but the principle is the same.\n\nFinally, the `finally` block shuts the executor down. Because the pool is created inside the method, this prevents thread leaks across calls.\n\nThe example's `Application` class shows the most direct use. It injects a `ChatClient.Builder`, builds a `ChatClient`, and runs the workflow from a `CommandLineRunner`:\n\n```\n@Bean\npublic CommandLineRunner commandLineRunner(ChatClient.Builder chatClientBuilder) {\n    return args -> {\n        List<String> parallelResponse = new ParallelizationlWorkflow(chatClientBuilder.build())\n                .parallel(\"\"\"\n                        Analyze how market changes will impact this stakeholder group.\n                        Provide specific impacts and recommended actions.\n                        Format with clear sections and priorities.\n                        \"\"\",\n                        List.of(\n                                \"Customers: price sensitive, want better tech, environmental concerns\",\n                                \"Employees: job security worries, need new skills, want clear direction\",\n                                \"Investors: expect growth, want cost control, risk concerns\",\n                                \"Suppliers: capacity constraints, price pressures, tech transitions\"),\n                        4);\n\n        System.out.println(parallelResponse);\n    };\n}\n```\n\nIn a real application you would not print to the console. You would expose the workflow as a bean and call it from a service or controller. Here is one way to do that:\n\n```\n@Configuration\nclass WorkflowConfig {\n\n    @Bean\n    ParallelizationlWorkflow parallelizationWorkflow(ChatClient.Builder builder) {\n        return new ParallelizationlWorkflow(builder.build());\n    }\n}\n\n@Service\nclass StakeholderAnalysisService {\n\n    private static final String PROMPT = \"\"\"\n            Analyze how market changes will impact this stakeholder group.\n            Provide specific impacts and recommended actions.\n            \"\"\";\n\n    private final ParallelizationlWorkflow workflow;\n\n    StakeholderAnalysisService(ParallelizationlWorkflow workflow) {\n        this.workflow = workflow;\n    }\n\n    List<String> analyze(List<String> stakeholderDescriptions) {\n        return workflow.parallel(PROMPT, stakeholderDescriptions, 4);\n    }\n}\n```\n\nNote that the example's `application.properties` configures Anthropic and Ollama properties, while the `pom.xml` currently includes the Anthropic model starter. If you run the example, make sure the model provider you actually use is on the classpath and configured. Spring AI's auto-configuration will pick up the `ChatClient.Builder` from whichever model starter is present.\n\nThe stakeholder example is a classic sectioning use case. The four stakeholder groups are independent: the analysis of customers does not depend on the analysis of suppliers. Running them in parallel with four workers means the batch takes roughly as long as the slowest single analysis, not the sum of all four.\n\nThe shape of the input matters. Each string in the list should carry enough context for the model to work on it alone. In the example, each group has a short bullet list of concerns. In a production system, you might pass a full document section, a customer record with its recent activity, or a set of tickets sharing a common category.\n\nSectioning also works well for document processing. Suppose you have a long contract split into clauses. You can ask the model to extract obligations from each clause in parallel:\n\n```\nList<String> clauses = splitIntoClauses(contractText);\n\nList<String> obligations = workflow.parallel(\n        \"\"\"\n        Extract every obligation the parties take on in this clause.\n        Return a short bullet list. If there are none, return \"None\".\n        \"\"\",\n        clauses,\n        6);\n```\n\nThe results come back in clause order, so you can zip them with the original clauses to build a report without any extra bookkeeping.\n\nA common mistake is to treat sectioning as a way to split a task that actually depends on itself. If the second section needs the conclusions of the first, the sections are not independent, and parallelizing them will give you wrong or inconsistent answers. The README makes this point clearly: ensure tasks are truly independent. If you need a chain of dependent steps, that is a different workflow, not a faster version of this one.\n\nVoting uses the same method, but the inputs are the same or nearly the same. The class appends the input to the prompt, so passing identical inputs produces identical prompts. You can use `Collections.nCopies` for that:\n\n```\nString transactionSummary = \"\"\"\n        Amount: 4,900 USD\n        Merchant category: electronics\n        Country: different from the cardholder's usual country\n        Time: 03:12 local time\n        \"\"\";\n\nList<String> votes = workflow.parallel(\n        \"Is this transaction likely fraudulent? Answer with exactly one word: APPROVE or REVIEW.\",\n        Collections.nCopies(5, transactionSummary),\n        5);\n\nlong reviewCount = votes.stream()\n        .filter(v -> v.trim().toUpperCase().startsWith(\"REVIEW\"))\n        .count();\n\nboolean needsReview = reviewCount >= 3;\n```\n\nThere is an important detail here. If the model is deterministic, five identical prompts will produce five identical answers, and voting adds nothing. Diversity comes from sampling. You need a non-zero temperature, or a varied prompt, to get meaningfully different answers. Check the defaults of the model you use and set temperature explicitly through the `ChatClient` options if you depend on variation. Otherwise, you are paying for five calls to get one answer.\n\nYou can also vary the prompt slightly across voters, asking for different perspectives. For example, one voter can focus on the amount and another on the timing. This makes the vote more like a panel of reviewers and less like a repeated sample, but you need to make sure the aggregation logic still makes sense.\n\nParsing the answers deserves care. The example above is strict: it expects a single word and checks the start of the string. In production, you should either ask for structured output and parse it into a type, or define a clear fallback for answers that do not match the expected format. A vote that cannot be counted should not silently count as \"approve.\"\n\nThe method's promise that results come back in input order is one of its most useful properties. It means you can treat the output as a parallel array aligned with the input list. You do not need to attach IDs to each request or sort results afterward.\n\nThis is worth preserving when you extend the code. If you switch to a version that returns results as they complete, you lose the alignment unless you carry an identifier through the pipeline. A simple approach is to return a small record:\n\n```\npublic record ItemResult(int index, String input, String output, Throwable error) {\n    public boolean succeeded() {\n        return error == null;\n    }\n}\n```\n\nEach item keeps its index and its input, and the result can say whether it succeeded. That leads directly to the next topic.\n\nThe example is strict about failures. If any single input fails, the `join()` calls throw, and the whole batch fails. The exception carries a message that names the input that failed, which is helpful for debugging, but it means one bad call can discard the results of the other ones.\n\nWhether that is the right behavior depends on your use case. For a batch that must be complete, such as a compliance check where a missing answer is worse than no answer, failing fast is correct. For a batch where most items matter and a few failures are tolerable, you want partial results.\n\nHere is a sketch of a tolerant variant that keeps the successful answers and records the errors:\n\n```\npublic List<ItemResult> parallelTolerant(String prompt, List<String> inputs, int nWorkers) {\n    ExecutorService executor = Executors.newFixedThreadPool(nWorkers);\n    try {\n        List<CompletableFuture<ItemResult>> futures = new ArrayList<>();\n        for (int i = 0; i < inputs.size(); i++) {\n            int index = i;\n            String input = inputs.get(i);\n            futures.add(CompletableFuture\n                    .supplyAsync(() -> chatClient.prompt(prompt + \"\\nInput: \" + input)\n                            .call()\n                            .content(), executor)\n                    .handle((output, error) -> new ItemResult(index, input, output, error)));\n        }\n        return futures.stream()\n                .map(CompletableFuture::join)\n                .toList();\n    } finally {\n        executor.shutdown();\n    }\n}\n```\n\nThe `handle` method runs whether the future succeeded or failed, so every item produces an `ItemResult`. The caller can then decide what to do with failed items: retry them, mark them for manual review, or skip them with a log entry.\n\nTwo practical notes. First, do not swallow errors silently. A result list full of nulls is much harder to debug than a list with explicit error records. Second, if you need retries, add them around the model call inside the supplier, with a bounded number of attempts and a backoff. Retrying an entire batch because one item failed wastes calls and can make rate limiting worse.\n\nThe `nWorkers` argument is the most important tuning knob in this pattern. It determines how many requests can be in flight at once, and therefore how hard your application pushes the model provider.\n\nA higher number of workers reduces latency for large batches, up to the point where you hit a rate limit. Once the provider starts returning rate-limit errors, you are no longer parallelizing usefully; you are generating retries. A lower number of workers is slower but more predictable.\n\nThere is no universal right value. Start with a small number, measure the throughput and the error rate, and increase gradually. The README's advice to consider API rate limits is the right instinct. Your limits depend on your provider, your account tier, the model, and the token counts per request, so treat any number as a starting point to be verified in your environment.\n\nAlso remember that parallelism does not reduce cost. Five calls cost five times as much as one, whether they run in sequence or in parallel. Parallelization buys you time, not money. If you are running voting or sectioning at scale, the cost model should be part of the design from the start: how many items per day, how many votes per item, how long the prompts are, and which model you use for each step.\n\nOne more consideration is the executor itself. The example creates a new fixed thread pool on every call. For a low-volume application that is fine. For a high-volume service, you might prefer a shared executor bean with a bounded queue, so that concurrent batches from different requests share the same capacity and cannot exhaust your resources together. In that case, you would also need to think about backpressure: what happens when the queue is full, and whether the caller should wait, fail, or degrade.\n\nEvery parallel call should have a timeout. An LLM request that hangs will hold a worker thread indefinitely, and with a fixed pool that means fewer workers available for the rest of the batch. A batch where one slow call blocks the join is not a parallel system; it is a sequential one with extra steps.\n\nYou can bound the wait with `CompletableFuture.orTimeout` or `completeOnTimeout` on each future, or with `get(timeout, unit)` when collecting results. Choose a value that reflects the real latency distribution of your prompts, not an optimistic guess. Log timeouts with the input that caused them, so you can see whether certain items are consistently expensive.\n\nBecause the class depends only on `ChatClient`, it is easy to test with a stub. You do not need a live model to check the important properties: that results come back in order, that inputs are passed through correctly, and that failures are handled as you expect.\n\nHere is one way to structure a test with a fake client. The exact mocking approach depends on your test setup, so treat this as a sketch:\n\n```\n@Test\nvoid resultsKeepInputOrderEvenWhenCallsFinishOutOfOrder() {\n    ChatClient client = mock(ChatClient.class, RETURNS_DEEP_STUBS);\n\n    when(client.prompt(anyString()).call().content()).thenAnswer(invocation -> {\n        String prompt = invocation.getArgument(0);\n        if (prompt.contains(\"Input: slow\")) {\n            Thread.sleep(200);\n        }\n        return prompt.substring(prompt.indexOf(\"Input: \") + 7).toUpperCase();\n    });\n\n    ParallelizationlWorkflow workflow = new ParallelizationlWorkflow(client);\n\n    List<String> results = workflow.parallel(\"Echo in uppercase.\", List.of(\"slow\", \"fast\"), 2);\n\n    assertEquals(List.of(\"SLOW\", \"FAST\"), results);\n}\n```\n\nThe point of this test is the ordering guarantee. The \"slow\" input finishes after the \"fast\" one, but the output list still starts with \"SLOW\". If someone later changes the implementation to return results as they complete, this test will catch it.\n\nYou should also test the empty-input and zero-worker validation, and the failure path. These are cheap tests that protect the contract of the method.\n\nFor integration tests against a real model, keep the number of calls small, use a fixed seed or low temperature where the provider supports it, and assert on structure rather than exact wording. Model output varies, and tests that compare full paragraphs will be brittle.\n\nA few mistakes come up repeatedly with this pattern.\n\n**Treating dependent tasks as independent.** If one call needs the output of another, parallelizing them will produce answers that ignore each other. Use a sequential chain or a different orchestration pattern for those cases.\n\n**Assuming voting adds accuracy without diversity.** Five identical deterministic answers are one answer, five times. Check sampling settings before relying on a vote.\n\n**Ignoring the cost of fan-out.** A batch of five hundred items with five votes each is 2,500 calls. Calculate the cost before running it in production.\n\n**Letting one failure discard a batch.** Decide whether your use case needs all-or-nothing behavior, and implement the tolerant variant if it does not.\n\n**Sizing the pool by instinct.** Measure. The right number of workers depends on the provider, the prompt size, and your rate limits.\n\n**Parsing free text as if it were structured.** When you aggregate votes or extract fields, use a defined output format and handle anything that does not match it.\n\n**Forgetting timeouts.** A single hung request should not stall the batch.\n\nHere is a short checklist I would apply to any production use of this pattern.\n\nParallelization is not always the answer. If the input set is small, the overhead of managing a thread pool and aggregating results may not be worth it. A few sequential calls are easier to reason about and debug.\n\nIf your workflow involves a decision that routes work to different specialists, a routing pattern fits better. If the output of one step should be improved by a critic before moving on, an evaluator-optimizer loop is the right shape. If the subtasks are not known in advance and must be planned by a model, an orchestrator-workers design is more appropriate. The Parallelization Workflow is the right tool when you already know the work splits into independent pieces, or when you want several independent opinions on the same question.\n\nThe example is intentionally small, and there are several natural extensions.\n\nEach of these builds on the same core idea: independent calls, run concurrently, combined at the end.\n\nThe Parallelization Workflow is one of the simplest and most useful patterns for building with LLMs. It turns a series of independent model calls into a concurrent batch, with results returned in a predictable order. The Spring AI example gives you a clean, readable starting point: a fixed thread pool, a `CompletableFuture` per input, and an ordered list of answers.\n\nThe pattern comes in two flavors. Sectioning speeds up work on different subtasks. Voting strengthens a single decision by gathering several perspectives. Both depend on the same discipline: the calls must be independent, the concurrency must be bounded, failures must be visible, and cost must be planned.\n\nIf you take one idea from this article, let it be this: parallelism is a property you design, not a switch you flip. Choose the worker count deliberately, handle the failures you will certainly see, and measure the results. The code in the example is a good foundation. The judgment about how to use it is yours.\n\n`CompletableFuture`, and returns results in input order.", "url": "https://wpnews.pro/news/parallelization-workflow-with-spring-ai-fan-out-llm-calls-in-java", "canonical_source": "https://dev.to/said_olano/parallelization-workflow-with-spring-ai-fan-out-llm-calls-in-java-43fg", "published_at": "2026-11-22 01:05:08+00:00", "updated_at": "2026-11-22 01:12:30.725103+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "developer-tools", "ai-tools"], "entities": ["Spring AI", "Anthropic", "ChatClient", "spring-ai-examples", "ParallelizationlWorkflow"], "also_reported_by": [], "takeaway": "Fan out independent LLM calls across a fixed pool of worker threads and collect results in input order, since ten sequential four-second calls cost forty seconds of wall-clock time while parallel calls overlap the waiting.", "recommendation": {"advice": "Fan out independent LLM calls with a bounded worker pool and preserve input ordering when collecting results, but budget for per-call billing, provider rate limits, and partial failures.", "playbook": "agent-architecture", "playbook_url": "https://wpnews.pro/playbooks/agent-architecture"}, "alternates": {"html": "https://wpnews.pro/news/parallelization-workflow-with-spring-ai-fan-out-llm-calls-in-java", "markdown": "https://wpnews.pro/news/parallelization-workflow-with-spring-ai-fan-out-llm-calls-in-java.md", "text": "https://wpnews.pro/news/parallelization-workflow-with-spring-ai-fan-out-llm-calls-in-java.txt", "jsonld": "https://wpnews.pro/news/parallelization-workflow-with-spring-ai-fan-out-llm-calls-in-java.jsonld"}}