# Beyond Web Access: Building a Reliable Capability-Based Router for AI Agent Tool Routing > Source: > Published: 2026-08-19 12:31:01+00:00 Imagine maintaining an AI agent that uses three different tools to perform user tasks. One tool completes 90% of requests in 500ms at $0.01 per success, but fails 15% of the time. Another handles edge cases with 99.9% reliability at $0.20 per success but takes 2 seconds. Most developers hardcode the first tool, ignoring cost and failures. The pain? Fragile systems that break when the “primary” tool goes down. Our goal: engineer a router that selects tools based on real-time performance, not blind luck. Think of it as a reverse proxy for AI agent capabilities. Early agent frameworks treated “web access” as a single tool. When teams requested a magic endpoint URL, systems made brittle assumptions: “If calling URL A returns 200, cache the result.” Nothing handled rate limits, schema changes, or paid APIs suddenly doubling prices. Agents panicked when providers updated endpoints without warning. Real-world impact: Last month, a recreation booking app using a hardcoded LLM provider failed when the vendor rotated API keys without notice. They refactored to versioned providers emitting OpenAPI specs, but that just shifted the problem down the chain. During a Kubernetes migration, we discovered teams had hard-coded dependencies on a single web-scraping provider. When regional API latency variations appeared, deployment reliability dropped to 78% in Australian nodes. Diagnostics revealed the problem: all scraping tasks routed to US-based URLs only, no logging of API response bodies, and retry strategies used simple backoffs regardless of error type. We recreated this fragility by assigning two developers to work directly with ChatGPT API documentation. One engineered a sophisticated regex parser for response validation while the other simulated rate-limited conditions. The productivity penalty was staggering: 3.2 hours lost per sprint cycle. To route intelligently, specify three things for every tool: input requirements (parameters, formats), success rules (HTTP status, response schema validation), and latency budgets (max milliseconds). Strict parameter validation using Zod or ajv. Example for multimodal tasks: ``` interface LLMTaskParameters { input_type: 'text' | 'code' | 'image'; content: string | File; strict: boolean = false; outlined_requirements?: string[];} ``` We implemented per-tool latency budgets accounting for network roundtrips. Using OpenTelemetry metrics: ``` python from opentelemetry import metricsmeter = metrics.get_meter(__name__)latency_counter = meter.create_counter( 'tool_response_latency', unit='milliseconds')# In request handlerstart = time.perf_counter()response = await client.execute_call()elapsed_ms = (time.perf_counter() - start) * 1000latency_counter.add(elapsed_ms)if elapsed_ms > TOOL_SCORECARD.P95: trigger_provider_switch() ``` At a fintech project, we maintained a versioned corpus of 143 tool API operations. Tool contracts stored as JSON synchronized across services: ``` // Tool contract example{ name: 'Stripe-API-2024', parameters: { amount_min: 1, amount_max: 100000, currency: 'USD', required: ['payment_method'] }, schemaVersion: 2, validationRules: { type: 'json_edit', minProperties: 1 }} ``` When integrating ChatGPT’s v1/v2 API versions, adding version headers became critical. The router needed: ``` GET /v2/completionsHeaders: { "X-API-Version": "20240315"} ``` Without versioning, unexpected schema changes broke 80% of active requests. We learned this when Stripe changed payment ID format, breaking 97% of routing rules, and Telegram’s HTTP tool needed async support added mid-implementation. Measure six metrics monthly from a task corpus: Example dashboard for ‘accessibility-summary-2023’ task set: When Midjourney v6 image generation added new parameters at $0.03/image vs v5’s $0.07, the scorecard naturally recommended switching provider operators. When OpenAI rolled out GPT-4 32k with stricter content policies, our router automatically isolated endpoints using path-checking middleware, created shadow wills to rewrite relevant token counts, and added content warning checks in parameter conversion: ``` js const GPT4Mapping = { modelType: 'gpt-4', maxTokens: 16384, validationRules: { inheritsFrom: 'GPT4-32k', schemaVersion: 3, restrictionOverride: (input: any) => { return input.content.length < 5000 ? omitSensitive(input) : allowSecureInput(input); } }}; ``` Decouple tool chaining from execution contexts through strategy pattern and sidecar containers. ``` class Router { selectTool(registration: Registration, params: Params): ToolOperator { // Compare current tool success rate vs new tool max latency if (registration.llm.latencyP95 > 1500 && newTool.latencyP95 < 800) { return new GeminiOperator(); } return new OpenAIModalOperator(); }} ``` Each tool runs in its own container with health checks and scaling constraints: ``` FROM node:latestWORKDIR /app# Security policiesHEALTHCHECK --interval=5s --timeout=3s CMD ./health-check.sh# Auto-scaling constraintsCMD ["./router-container", "--max-concurrency", "3", "--retry-laps", "2"] ``` Fact: 73% of providers report *claimed* latency in docs. Real-world tests show DALL-E 3 median latency at 3100ms vs advertised 1800ms during peak usage. Instrument every call: ``` python from opentelemetry import tracetracer = trace.get_tracer(__name__)with tracer.start_as_current_span('dalle-call') as span: result = venv.exec() if result.took > 2500: span.set_attribute('validator.failed', True) ``` Not all errors justify switching. Implement priority tiers: **Tier 1: Retryable transient errors** **Tier 2: Recovery errors** **Tier 3: Terminal errors** Implementation showing exponential backoff with provider switching: ``` python def handle_api_error(error: ApiError, current_tool: ToolOperator): if error.is_transient() and current_tool.retry_count < 3: return asyncio.sleep(2 ** current_tool.retry_count) if hasattr(error, 'upstream_provider_id'): return switch_to_provider(error.upstream_provider_id) raise CalledProcessError(ExternalToolError(error)) ``` Their router used a reservation pattern: When our Drupal-based calendar app suffered Google Calendar API outages, we discovered network failures must differentiate from provider policy blocks. Our fallback logic: ``` failover_rules:- provider: stripe-payments conditions: - or: - when: type: APIResponse filters: status: 429 action: retry(max=3, interval_sec: 8) - when: type: SchemaResponse required: 'payment_method' action: forward_to(stripe-cloud-worker)- provider: chart-generator conditions: type: NetworkTimeout action: fallback(render_local_rss, stcp) ``` Run candidate tools on copied task sets without affecting production. Screencome workflow: Shadow deployment middleware in Go: ``` package mainimport "net/http"func main() { http.HandleFunc("/chatgpt/beta", func(w http.ResponseWriter, r *http.Request) { if shouldShadow(r) { r.URL.RawQuery = "format=beta&mock=1" r.URL.Path = "/chatgpt/prod" } proxy.ServeHTTP(w, r) })} ``` Monitoring gets complex when handling 500ms latencies in tool A vs 2.1s in tool B, different error semantics (400 means different things per vendor), and cost-per-1000 requests that defy API-reported usage. Our monitoring indexes combine technical and business metrics through adapter layers: ``` - Tag compound_service = `task-${taskID}-provider-${toolOp.name}`- Trace spans grouped by both capabilityName and schemaVersion- Dialogue success rate: (capability+"success").avg = 91.2% for routable tasks ``` When a marketing team’s content pipeline suddenly doubled in cost, investigation revealed a provider had silently switched from per-token to per-request billing. Our pipeline caught it within 4 hours because we track cost per successful call as a first-class metric, not an afterthought. Pro tips for transitioning to capability-based routing: [Beyond Web Access: Building a Reliable Capability-Based Router for AI Agent Tool Routing](https://pub.towardsai.net/beyond-web-access-building-a-reliable-capability-based-router-for-ai-agent-tool-routing-e58230677685) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.