{"slug": "your-spring-ai-tests-are-slow-flaky-and-cost-money-here-s-how-to-make-them", "title": "Your Spring AI tests are slow, flaky, and cost money. Here's how to make them deterministic.", "summary": "Rifat Cakir released spring-ai-test-tools, a VCR-style library that records real Spring AI interactions and replays them deterministically in tests, eliminating slow, flaky, and costly live model calls. The library attaches to ChatClient.Builder via ChatClientBuilderCustomizer, caches exact SHA-256 request hashes, and supports REPLAY_ONLY mode for CI, making cache misses loud failures. Replay runs in about 0.8 ms versus seconds for live calls, and it supports tool calls, streaming, entity mapping, embeddings, and evaluators.", "body_md": "You wire up Spring AI, the `ChatClient`\n\nfluent API feels great, your feature works. Then you sit down to write a test — and every good option is bad.\n\nA test that calls a real model is:\n\nThe usual workarounds all hurt: **Mockito** means hand-building Spring AI's nested `ChatResponse → Generation → AssistantMessage`\n\ngraph and asserting against a response *you* wrote; **WireMock/MockWebServer** means owning each provider's exact wire JSON, SSE frames, and tool-call envelopes, and rewriting it all when you switch providers; **the real model** is the four problems above, accepted rather than solved.\n\nThere's a well-worn answer from the HTTP world — Ruby's VCR, Python's `vcrpy`\n\n: record the real interaction once, replay it deterministically after. The catch is those work at the HTTP layer, and Spring AI's value is the abstraction *above* HTTP. So I built the same idea where Spring AI actually lives.\n\nOne dependency:\n\n```\n<dependency>\n    <groupId>io.github.rifatcakir</groupId>\n    <artifactId>spring-ai-test-tools</artifactId>\n    <version>0.1.0</version>\n    <scope>test</scope>\n</dependency>\n```\n\nOne property (`src/test/resources/application-test.yml`\n\n):\n\n```\nspring:\n  ai:\n    test:\n      vcr:\n        enabled: true\n        mode: RECORD_OR_REPLAY   # REPLAY_ONLY in CI\n```\n\nYour test doesn't change at all — you write it exactly as you would against a real model:\n\n```\n@SpringBootTest\nclass OrderStatusTest {\n\n    @Autowired ChatClient.Builder chatClientBuilder;\n\n    @Test\n    void answersAQuestionAboutTheOrder() {\n        String answer = chatClientBuilder.build().prompt()\n            .user(\"What's the status of order ORD-4471?\")\n            .call().content();\n\n        assertThat(answer).contains(\"shipped\");\n    }\n}\n```\n\nFirst run reaches a real model and writes `src/test/resources/llm-cache/{sha256}.json`\n\n— **you commit that file.** Every run after replays it in under a millisecond, offline.\n\n```\nFIRST RUN          slow · costs tokens · needs network\n  Your test ──▶ ChatClient ──▶ Real LLM  ──writes──▶  cassette.json  (committed)\n\nEVERY RUN AFTER    instant · $0 · fully offline\n  Your test ──▶ ChatClient ◀──reads──  cassette.json                 (~0.8 ms)\n```\n\nThe advisor attaches to every `ChatClient.Builder`\n\nin the context via `ChatClientBuilderCustomizer`\n\n— so **nothing under test, and nothing in production, knows the cache exists.** In CI you seal it with `mode: REPLAY_ONLY`\n\n: now a cache miss is a *loud failure*, not a silent call to a live model. The cache key is an exact SHA-256 over the canonical request; there is no fuzzy matching, ever. (This is why Spring AI's *production* semantic cache doesn't solve the testing problem — it matches on similarity thresholds, which is exactly backwards for a test.)\n\nThe point isn't a benchmark number — it's what disappears:\n\nAnd yes, replay is ~0.8 ms (median over 200 timed iterations in a real Spring context) versus a warm hosted call of ~1–2 s or a local cold call of ~47 s — but treat that as a side effect. The real win is that the network, the cost, and the rate limits are simply gone.\n\nUp front, because senior engineers rightly distrust silver bullets — this sits *above* the HTTP layer, so it cannot test that layer:\n\n`Retry-After`\n\n, connection pooling, a body arriving malformed mid-stream → that's Each of these is verified against a real model, not assumed:\n\n`@Tool`\n\ncall's name and arguments are part of the cache key, and on replay the recorded tool result is injected `Flux<ChatResponse>`\n\nreplays chunk-for-chunk — not a single-chunk fake — tool-call fragments included.`.entity(MyDto.class)`\n\ncall's target schema is part of the cache key, so two output types with the same prompt never collide.`EmbeddingModel`\n\ncalls cache independently of chat; a replayed vector is exactly, not approximately, what was recorded.`RelevancyEvaluator`\n\n/ `FactCheckingEvaluator`\n\nrun deterministically in CI (the judge call itself is recorded), or live on demand for a drift check.Independent, community project (not affiliated with Spring/Broadcom), Apache-2.0, currently `0.1.0`\n\nand early — tested against Java 21 · Spring Boot 4.0.0 · Spring AI 2.0.0. If you try it, issues and feedback are genuinely wanted.", "url": "https://wpnews.pro/news/your-spring-ai-tests-are-slow-flaky-and-cost-money-here-s-how-to-make-them", "canonical_source": "https://dev.to/rifatcakir/your-spring-ai-tests-are-slow-flaky-and-cost-money-heres-how-to-make-them-deterministic-1og7", "published_at": "2026-08-11 10:07:48+00:00", "updated_at": "2026-08-11 10:17:35.088306+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools", "mlops", "artificial-intelligence"], "entities": ["Rifat Cakir", "Spring AI", "ChatClient", "spring-ai-test-tools", "Mockito", "WireMock", "MockWebServer", "VCR"], "alternates": {"html": "https://wpnews.pro/news/your-spring-ai-tests-are-slow-flaky-and-cost-money-here-s-how-to-make-them", "markdown": "https://wpnews.pro/news/your-spring-ai-tests-are-slow-flaky-and-cost-money-here-s-how-to-make-them.md", "text": "https://wpnews.pro/news/your-spring-ai-tests-are-slow-flaky-and-cost-money-here-s-how-to-make-them.txt", "jsonld": "https://wpnews.pro/news/your-spring-ai-tests-are-slow-flaky-and-cost-money-here-s-how-to-make-them.jsonld"}}