# How to Evaluate an Agent Platform Without Getting Sold on Hype: The Real Infrastructure Questions

> Source: <https://dev.to/paultwist/how-to-evaluate-an-agent-platform-without-getting-sold-on-hype-the-real-infrastructure-questions-1e2>
> Published: 2026-08-03 16:02:45+00:00

In August 2026, the agent platform market is fractioning into distinct categories and it's getting confusing.

You have:

The market is selling all three as "agent platforms" and teams are getting stuck trying to make pure frameworks handle infrastructure problems, or trying to make managed platforms answer governance questions they weren't designed to solve.

Here's the problem: **You need all three, and they operate at different layers.**

**Layer 1: Agent Logic** (frameworks)

Your agent's reasoning, tool-calling, state management, planning. LangGraph, CrewAI, Pydantic AI all solve this. They are excellent at it.

**Layer 2: Agent Runtime** (managed platforms)

Execution sandbox, session isolation, interrupt/resume, built-in memory. Claude Managed Agents, Bedrock AgentCore, Gemini Enterprise Agent Platform all provide this. They are production-ready for single-runtime deployments.

**Layer 3: Agent Control Plane** (infrastructure)

Multi-runtime coordination, credential centralization, session durability across runtimes, per-agent identity, observability, scheduling, cost attribution, evaluation-driven decision-making. Almost nobody has this built for agents yet. Most teams are building it themselves, month 3 of their agent project.

The silent assumption in agent platform marketing is that managing agents is the same as managing LLMs. It's not.

June 2026 just showed three major team-level agent platforms ship in one week:

This signals that the market is shifting from "single agent, single runtime" to "multiple agents, multiple teams, multiple runtimes."

At that scale, the infrastructure gaps become visible:

Teams solve this by building custom infrastructure. The smart ones realize mid-project that this is not a one-week sprint. It's a 6-8 week architectural commitment.

When you're looking at agent platforms in August 2026, ask these questions. They separate operational infrastructure from hype.

**The question**: Can I run the same agent logic on Claude Managed Agents one day and Bedrock AgentCore the next without rewriting my agent?

**What to look for**: Runtime adapters, agent registration that's independent of runtime choice, session portability with state.

**Red flags**: "Pick one platform and stick with it," "we integrate with AWS," "works best with Claude," "you'll need custom code to support another runtime."

**Why it matters**: You don't know your production constraints at pilot time. You'll need SOC2 compliance (June), so you pick AWS (month 3). Then you need data residency in EU (month 4). Then you need on-prem for sensitive workflows. If your agent infrastructure is locked to one runtime, you're rebuilding every time.

**The question**: Can I manage provider credentials (AWS keys, Anthropic tokens, GitHub OAuth) in one place instead of per-agent per-runtime?

**What to look for**: Vault integration, credential scoping to agents (not global), immutable audit trails of credential usage.

**Red flags**: "Each agent has its own API keys," "credentials live in environment variables on the runtime," "console access is required to rotate credentials."

**Why it matters**: At month 2, you have 3 agents. At month 6, you have 5 agents across 3 runtimes with 10 people who need access. Console access becomes operationally unmaintainable. Credential rotations on departing team members become a scramble. Per-agent credential scoping is the only approach that scales.

**The question**: Can I change an agent's model, tool permissions, rate limits, or cost budget without redeploying the agent itself?

**What to look for**: Configuration stored separately from agent definitions, ability to change policies in a control plane without touching agent code/containers.

**Red flags**: "Config is baked into the agent," "policy changes require rebuilding the container," "you'll need to redeploy to change models."

**Why it matters**: At month 2, your agent works with GPT-4. By month 3, GPT-5 ships and costs less. You want to swap models in a config file, not redeploy. Same for tool permissions: if an agent discovers a new API, you don't want to rebuild—you want to disable/enable it in your control plane.

**The question**: If an agent crashes mid-workflow, can I resume from exactly where it stopped, possibly on a different runtime?

**What to look for**: Postgres-backed durable sessions, session snapshots that include state/memory/execution history, ability to replay sessions for debugging.

**Red flags**: "Sessions are in-memory," "you'll need to implement resumption yourself," "sessions are tied to a specific runtime pod."

**Why it matters**: Long-running agents (24+ hours, 100+ steps) are the value-add for your organization. They're also fragile. Durability is the difference between "lost a day of work" and "lost 4 hours of work." At 5+ agents running simultaneously, session durability becomes the operational baseline.

**The question**: Can I see which agent caused which spend? Can I query all invocations across runtimes? Can I see evaluation signals per agent?

**What to look for**: Per-agent cost dashboards, queryable execution history, evaluation metrics tied to agent identity, cost attribution down to the tool call.

**Red flags**: "Cost is per provider/model," "observability is at the platform level, not per-agent," "you'll need to wire your own logging."

**Why it matters**: At month 3, you'll discover that one agent is calling expensive tools 500x per session while another is nailing it with 3 calls. You need to see that breakdown, not aggregate costs. Same with quality: evaluation metrics are worthless if you can't attribute them to specific agents.

**The question**: If Agent A tries to call a tool it's not authorized for, is that decision made by infrastructure (impossible to bypass) or embedded in the agent's training/prompts (possible to workaround)?

**What to look for**: Invocation-layer tool authorization, immutable audit trails showing *who authorized what*, per-agent tool scoping enforced at the gateway.

**Red flags**: "Authorization is in the system prompt," "guardrails are part of the agent logic," "we rely on the model to follow guidelines."

**Why it matters**: This is the security/compliance line. After July 2026's incidents (OpenClaw, Hugging Face, Langflow), auditors are asking: "Prove to me that Agent A couldn't call the delete endpoint." Prompt-based controls fail under adversarial scenarios. Infrastructure-enforced controls don't.

**The question**: Can I express that "Agent A runs after Agent B succeeds" without external tooling? Can agents pass messages to each other? Can I set budgets across a fleet of agents?

**What to look for**: Native scheduling, agent-to-agent communication, fleet-level resource management, dependency graphs.

**Red flags**: "You'll need n8n/Temporal/Prefect for orchestration," "agents run independently," "you'll need to manage coordination yourself."

**Why it matters**: By month 4, single agents are boring. You want Agent A to find problems, Agent B to analyze them, Agent C to draft solutions. Without native coordination primitives, you're bolting on external orchestration, which adds latency, complexity, and another tool to operate.

Here's what I'm seeing teams adopt when they move from pilots to sustained multi-agent operations:

```
Agent Logic Layer
    ↓ (defined once, runtime-agnostic)
Control Plane
    ├─ Agent Registry (who exists, what can they do)
    ├─ Credential Management (vault, scoped keys)
    ├─ Policy Enforcement (rate limits, budgets, authorization)
    ├─ Session Management (durable state, resumption)
    ├─ Observability (cost, evaluation, audit trails)
    └─ Scheduling (when agents run, dependencies)
    ↓
Data Plane (Fast Routing)
    ├─ Runtime Adapter Layer
    ├─ Provider Translation
    └─ Load Balancing
    ↓
Agent Runtimes
    ├─ Claude Managed Agents
    ├─ Bedrock AgentCore
    ├─ Gemini Enterprise Agent Platform
    └─ Self-Hosted (E2B, Daytona, etc.)
```

The control plane is where all the decisions live. The data plane is where requests move fast. They cannot be combined: governance is stateful (needs Postgres, audit trails, authorization checks), routing is stateless (needs to be <1ms overhead).

After evaluating against the seven questions above, here are the tell-tale signs of infrastructure that will actually scale:

**It separates control plane from data plane.** The platform acknowledges that governance and fast routing are different problems.

**It treats agents as first-class infrastructure.** Agents have durable identity, persistent sessions, queryable history—like microservices, not like chat sessions.

**It's runtime-agnostic by design.** The platform doesn't bet the company on one managed runtime. It works *with* runtimes, not *replacing* them.

**Cost, evaluation, and observability are baked in.** Not bolted on. Not "you can integrate with X." Native first-class primitives.

**It has a Rust or native-compiled data plane.** Python gateways add 7-8ms per request under concurrency. At 100 agent steps per session, that's 700-800ms pure infrastructure tax. For agent workloads, sub-1ms overhead is table-stakes.

**Multi-agent coordination is obvious, not hidden.** If you have to ask "how do I make agents work together," the platform is incomplete.

I'll be direct: I work on LiteLLM infrastructure. Full transparency.

LiteLLM Agent Platform (LAP) is built exactly against the pattern above:

Is it the only platform that answers all seven questions? No. But it's built specifically *for teams that need to scale agents across runtimes* and that's a category that barely existed in June 2026 and is now the market reality.

In August 2026, if you're evaluating agent platforms and you're thinking "our team will run all agents on Claude Managed Agents forever," then you don't need control-plane infrastructure. A framework + managed runtime is sufficient.

But if you're thinking "we'll probably need AWS for compliance, maybe on-prem for sensitive data, and definitely self-hosted for cost control," then you need infrastructure that doesn't lock you to one runtime.

That's when you ask the seven questions above. And you'll find that almost nothing answers all seven yet. The ones that do will become table-stakes by Q4 2026 as teams discover the hard way that "pick one platform" doesn't work at scale.

The teams winning in August 2026 aren't the ones with the smartest agents or the flashiest models. They're the ones that built or adopted boring, reliable infrastructure for agent coordination. Ask the seven questions. Pick the platform that answers "yes" to most of them. Everything else is nice-to-have.

**What's your experience evaluating agent platforms? Are you hitting any of these seven gaps in your own deployments? Drop it in the comments—I'm collecting real operator signal for what's actually mattering in production agent systems right now.**