I also recorded a short demo showing the complete workflow, from the OpenAPI specification to generating and executing tests, and auto healing:
I originally built this as part of my GSoC 2026 proposal for foss42.
I wasn't selected for GSoC, but while working on the proposal I had already built a working prototype of the system I wanted to contribute: an agentic API testing workflow that could read an OpenAPI specification, generate tests, execute them, and reason about failures.
And while building it, I ran into a problem that had little to do with HTTP itself.
The OpenAPI specification was becoming too large to send to an LLM efficiently.
That led me to build OpenAPI Context Batching -- a deterministic approach for giving an LLM only the relevant part of an OpenAPI specification instead of repeatedly sending the entire API.
In my prototype experiment, this reduced approximate token usage from ~50,000 tokens to ~9,200 tokens across the same five-interaction workflow; an 81.7% reduction🤯.
The original goal of my project was to let an AI agent read an OpenAPI specification, generate test plans, and analyze API failures. But I quickly ran into a huge problem.
Real OpenAPI specifications are gigantic. They contain hundreds of endpoints, nested schemas, and complex validation rules. If you dump that entire JSON file into an LLM prompt, you run into two major problems:
If I just want the AI agent to test a /categories
endpoint, it absolutely does not need to see the schemas for /payments
or /cart
or the endpoints not related to it. I realized I needed a way to give the model a smarter context instead of just a bigger one.
Before I could optimize the tokens, I had to lock down the core architecture. I established one strict rule for the whole system: the AI figures out what to test, and Dart does the actual testing.
Letting an LLM directly execute HTTP calls is a terrible idea because it can easily invent fake responses. To prevent this, the model is constrained to generate a structured JSON test plan.
I built a standalone, headless Dart execution class called ApiTestRunner
. This runner ingests the JSON test plan and performs the real HTTP requests using the networking layer of API Dash. The AI never touches the network directly.
So, how do we actually shrink a massive OpenAPI spec? We do not ask another AI to summarize it. We use deterministic code because OpenAPI is highly structured.
I wrote a custom Dart algorithm called OpenAPI Context Batching. Instead of sending the full specification, the script parses it and breaks it apart deterministically. Here is exactly how it works under the hood:
/auth
or /categories
.$ref
to reuse shared schemas. If you only extract the endpoint path, the AI will not know what the data payload actually looks like. I built a recursive resolver that digs through the endpoint and finds every $ref
. components/schemas
object.Map<String, String>
where each entry is the fully resolved, batched schema for just one domain.When the agent needs to reason about a request, it only receives this highly focused batched schema. Because the algorithm is deterministic, the exact same OpenAPI specification always produces the exact same endpoint partitions every time.
To make the system even smarter, I built a conversational agentic mode with a self healing loop.
If the Dart execution engine runs a test and gets a failing non 2xx status code, it intercepts the error. It then injects the exact status code and response body back into the LLM prompt, asking the agent to fix the parameters automatically. To make sure this does not result in an AI driven infinite loop that drains credits, I hardcoded a strict limit of three maximum retries.
I benchmarked this approach on a custom shop.json
OpenAPI specification during my prototype development. The result was significant: approximately 81.7% fewer input tokens for the tested workflow.
| Approach | Approximate Token Usage |
|---|---|
| Entire OpenAPI specification | ~50,000 tokens |
| Context Batching | ~9,200 tokens |
By only sending the dependency aware context, I achieved an 81.7% reduction in token usage for a five step conversation workflow. Not only did it save a massive amount of tokens, but the focused context meant the AI responded faster and stopped hallucinating.
| Approach | Visualization | Approx tokens |
|---|---|---|
| Entire OpenAPI passed at once | ~50,000 | |
| OpenAPI Context Batching | ~9,200 |
The coolest thing I learned from this project is that we should not use LLMs for everything.
Parsing files, finding dependencies, and executing HTTP requests are tasks best handled by deterministic software. Generating test plans and reasoning about edge cases are where the LLM actually shines.
Building this engine independently of the Flutter UI also allowed me to expose it as an MCP server, meaning external tools like Claude Desktop can trigger real test runs. It was an incredible learning experience, and I am super excited to keep exploring how we can build smarter, more efficient developer tools!