{"slug": "the-outer-loop-insights-first-an-ambient-quality-agent-that-diagnoses-your-agent", "title": "The Outer Loop, Insights First: An Ambient Quality Agent That Diagnoses Your Production Agent", "summary": "Google released AQuA (Ambient Quality Agent) as open-source composable building blocks in its adk-recipes GitHub repository, a tool that runs unattended alongside a production agent in a Google Cloud project to diagnose recurring failure patterns. AQuA sweeps production trajectories from Cloud Trace, Cloud Logging, or BigQuery on a schedule, after each deployment, or on demand, and runs each sample through a five-stage pipeline that writes structured actual/expected findings labeled NEW, RECURRING, or auto-RESOLVED after 14 days unseen. By default AQuA uses the single-pass session_review judge, which evaluates each conversation against a developer-written goal.md checklist in one model call per session, and never sits in the request path or writes back to the agent.", "body_md": "Your agent returns HTTP 200, stays inside its latency budget, and no tool call raises an error. Yet it still records seat 3A as confirmed without ever asking the seat-selection sub-agent whether 3A is free, or recommends a Florentine steakhouse to a traveler whose profile says vegan.\n\nEvery team we work with that runs an agent in production is trying to answer the same three questions: **How is my agent doing? What are its loss patterns, the ways it fails that keep coming back? And how do I climb:** change something, know it helped, and keep it from sliding back?\n\nGetting an agent to the first 80%, where it handles the test cases you anticipated, can be relatively straightforward and fast today. An evaluation dataset, a coding agent, and a tight inner loop of run, grade, fix, and compare usually get you to task success on known scenarios in days. Then you launch, and the quality curve flattens or degrades.\n\nWhat makes the remaining 20% harder is that the ground moves after launch. Usage shifts as users discover what the agent can actually do, and live traffic stops resembling your initial eval set. At the same time, the system underneath also changes: when you upgrade a model, update your harness, or change a tool or skill, everything still runs and health checks stay green (Figure 2), but conversation quality and task success may shift in ways a standard deploy pipeline never warns you about.\n\nAnd when a session does go wrong, working out *why* means separating several layers that can look nearly identical from the outside. For example:\n\nThose failure modes can have different owners and different fixes. A raw trajectory records what happened next to what, not what caused what. Telling those layers apart requires reading the failing trajectories alongside the exact code revision that produced them. Online evaluation dashboards tell you *that* a score moved, and a coding agent can inspect a trace once you know where to look, but at production volume nobody can read every conversation by hand.\n\nIn [our June post](https://developers.googleblog.com/driving-the-agent-quality-flywheel-from-your-coding-agent/), we walked that pre-launch inner loop end to end on travel-concierge. This post covers the outer loop, [released in the open](https://github.com/google/adk-recipes/tree/main/core/python/ambient-quality-agent) as composable building blocks you can run in your own project, adapt to your stack, and help us shape as we explore how teams run continuous agent quality in production.\n\n**AQuA (Ambient Quality Agent)** runs unattended beside your agent in your Google Cloud project, sweeping production trajectories from Cloud Trace, Cloud Logging, or BigQuery on a schedule, after each deployment, or on demand while your laptop is closed. Raw session transcripts, source snapshots, and BigQuery tables stay inside your project boundary, and AQuA never sits in the request path or writes back to your agent.\n\nEach run reads a sample of recent sessions and puts it through a five-stage pipeline:\n\n`goal.md` in view. When a session fails, it writes a structured `actual / expected` finding.`NEW`, `RECURRING`, or auto-` RESOLVED` after 14 days unseen.\nThink of it as a **junior quality engineer doing the first pass**: reading conversations, filtering out noise, and preparing the case file with evidence sessions for whoever is on rotation.\n\nTo steer Stage 2 toward your domain, you write a plain-English **developer goal** (`goal.md`) that is appended to every review prompt; deterministic Python **custom metrics** (eval_config.yaml) run alongside the judge to trend pass rates. By default, AQuA uses the single-pass `session_review` judge, which evaluates the conversation against that checklist in **one model call per session** (keeping scheduled sweeps economical and producing the `actual / expected` diffs that drive clustering). You can also opt in to the Gemini platform's managed trajectory AutoRaters (`task_success`, `tool_use_quality`, `trajectory_quality`), which run dedicated per-metric evaluators (for example, if you want to score sessions against standardized out-of-the-box rubrics or align metrics with offline evaluations).\n\nWhen an insight is worth investigating, you trigger root-cause analysis from the dashboard **Chat** or `agents-cli aqua run`. AQuA reads the failing trajectories alongside the immutable source snapshot captured at deploy time: when the defect is in your repo, it cites `<path>:<start>-<end>` and proposes an edit anchored to the snapshot lines; when the fault lies outside your code (an upstream dependency, handoff, or retrieved payload), it attributes the failure to that step in the trajectory without proposing a code diff. It never applies an edit or opens a pull request on its own.\n\nIt sits between the loops you already have: offline evals grade a candidate build against known test cases, online evals monitor pass-rate trends in production, and coding agents or specialized optimizers edit prompts and code. AQuA turns raw production traffic into diagnosed, code-anchored insights and archived failure transcripts that seed your inner loop.\n\nLet's walk a run on **`travel-concierge`** ([google/adk-recipes](https://github.com/google/adk-recipes/tree/0521e23/python/agents/travel-concierge)), which routes a traveler across sub-agents (`inspiration_agent`, `place_agent`, `poi_agent, planning_agent`, `flight_search_agent`, `flight_seat_selection_agent`, `booking_agent`, and `pre_trip / in_trip / post_trip`) and stores the working trip in session state via `memorize(key, value)` (`travel_concierge/tools/memory.py`).\n\nAttaching AQuA to the travel-concierge project takes three agents-cli commands:\n\n`agents-cli extension add \"${AQUA_CHECKOUT}\"`\n\n`agents-cli infra single-project --project=\"${GOOGLE_CLOUD_PROJECT}\" --apply`\n\n`agents-cli deploy --project=\"${GOOGLE_CLOUD_PROJECT}\" --region us-east1`\n\nAlongside AQuA's runner, BigQuery dataset, and Cloud Run dashboard (behind Identity-Aware Proxy), `agents-cli deploy` writes an immutable snapshot of `travel-concierge`'s source tree to Cloud Storage keyed by the deployment revision (`Revision 1`).\n\nOn the dashboard's **Configuration** page, we save a **developer goal** (`goal.md`) to steer the review toward high-level product invariants and suppress stylistic noise (while still requiring every finding to cite a concrete turn where an agent or sub-agent violated its instructions or tool state):\n\n```\nGoal: Help travelers move from trip inspiration to a concrete itinerary and confirmed bookings across our sub-agents, with every confirmed flight, hotel, seat, and recommendation grounded in tool results and the traveler's profile.\nFailure modes to make sure we cover:\n- Mid-conversation changes (a weak spot in pre-launch testing): if a user updates destination, dates, or flight/hotel choices after an initial plan, make sure subsequent sub-agent tool calls and state updates reflect the change.\n- Dropped context when handing off or delegating across inspiration_agent, planning_agent, and booking_agent (such as traveler profile preferences or prior selections).\nIgnore: tone, greetings, small talk, or sessions where the user browses options and leaves without booking.\n```\n\nIf you also want a deterministic Python rubric in `eval_config.yaml` to trend pass rates over time, you can publish it alongside the sweep with `agents-cli aqua metrics publish`.\n\nAcross a 32-session multi-agent sweep (32 replays of four scripted traveler journeys, producing 1,583 OpenTelemetry spans across travel_concierge and its sub-agents), 5 sessions pass cleanly and 27 produce 42 structured findings, each scoped by span to the specific sub-agent's isolated instructions and tool declarations. Here is one finding on planning_agent:\n\n**actual:** When the user selected outbound flight UA204 and requested seats 3A and 3B in the same turn, planning_agent saved those seat numbers directly to session state without calling flight_seat_selection_agent to check whether 3A and 3B were available.\n\n**expected:** planning_agent should call flight_seat_selection_agent to verify seat availability and pricing before saving outbound or return seat numbers to session state.\n\nClustering groups those 42 findings into 9 candidate clusters. The verifier (Gemini 3.7 Flash) checks each cluster against up to three full transcripts and each sub-agent's definition, and rejects **3 false-positive clusters**: in two, the traveler had explicitly asked to take the first return flight or jump straight to booking; in the third, clustering had merged unrelated prompt-tool mismatches from two different sub-agents (flight_search_agent and inspiration_agent). That leaves **6 verified issues**, led by:\n\n`tools=[]` ), so it fabricates placeholder `https://example.com/...` URLs.\nOpening the top insight in the dashboard surfaces the full breakdown for the cluster: the number of matching sessions (15 traces), verification summary, and the 15 linked session traces with the exact findings that triggered it:\n\nClicking **Investigate** from the insight card launches the root-cause diagnosis agent (Gemini 3.8 Flash) against Revision 1's 33-file source snapshot. Cross-referencing the 15 failing traces against the repository tree, the agent traces the bug to line 93 of `travel_concierge/sub_agents/planning/prompt.py`: the flight-search instructions only tell planning_agent to call flight_seat_selection_agent when presenting a seat map for the user to choose from, leaving out the case where a user volunteers a seat number directly. It proposes a line-anchored fix in the chat (and similarly traces the vegan profile handoff bug to line 23 of `travel_concierge/sub_agents/inspiration/prompt.py`):\n\nThat same insight payload, including its anchored `edits[]` and `occurrences[].rubrics[].trace`, is available via `agents-cli aqua get-insight`. A coding agent (using the `agents-cli-aqua` and `google-agents-cli-eval` skills) can trigger diagnosis headlessly, apply the edit on a branch, extract the failing session's user inputs into a local replay file, and verify the fix before opening a pull request:\n\n```\n# 1. Pull new insights affecting >= 10 sessions without a root cause\n\nagents-cli aqua list-insights --status NEW --root-cause false \\\n  | jq -r '.insights | sort_by(-.trace_count) | .[] | select(.trace_count >= 10) | \"\\(.insight_id)  \\(.label)\"'\n\n# 2. Trigger root-cause analysis headlessly and fetch the anchored edit + evidence traces\n\nagents-cli aqua run 'Diagnose insight 220d9209e27d4e16a73b4ad4741caa81. What is the root cause, and how would you fix it?'\nagents-cli aqua get-insight 220d9209e27d4e16a73b4ad4741caa81 > insight.json\n\n# 3. Extract the user turns from the attached trace in insight.json for local replay (or add to your eval set)\n\njq '{state: {}, queries: [.occurrences[0].rubrics[0].trace[] | select(.role == \"user\") | .content]}' \\\n  insight.json > ./b4b38471-inputs.json\nadk run --replay ./b4b38471-inputs.json travel_concierge\n```\n\nAfter applying both one-line prompt fixes (`planning/prompt.py:93` and `inspiration/prompt.py:23`) and deploying **Revision 2**, replaying the same 32 sessions against it under the same developer goal shows:\n\nAn agent that diagnoses your agent will sometimes be wrong. That is inherent to using models as judges. When we designed AQuA, the core engineering requirement was making sure an unverified hypothesis never looks the same as a verified finding:\n\n`confidence` field: every occurrence links to its session IDs in Cloud Trace, and every root cause cites `<path>:<start>-<end>` ranges that the server validates against that revision's snapshot. Any citation to a nonexistent file or line is rejected.\nSeveral boundaries in this reference implementation are deliberate trade-offs across practical problems we see in production:\n\n`ORDER BY RAND()`). Adding cheap structural pre-filters (retries, latency spikes, high turn counts, user thumbs-down) is a straightforward next step, though filtering only on anomalies tends to pull a hundred copies of the same timeout. The harder problem is catching a silent regression that breaks 1% of a critical workflow, where every structural signal looks normal, without running a deep judge on all 50,000 daily sessions.`adk run --replay` re-sends recorded user turns against local code, which works when tools are idempotent or mocked, but static replay does **What actually generalizes at the platform level.** Across Google's own first-party agents, low-level primitives (trace and feedback ingestion, core evaluators, dataset management) share cleanly, whereas outer-loop workflows are often bespoke to a product's tools, domain invariants, data pipelines, and orchestration harness. As foundation models and agents improve at reading trajectories and navigating code, individual grading and root-cause reasoning get stronger automatically, which is why we think about AQuA's prompts as modular **recipes** and its insights as an index over raw trajectories and source snapshots, so a stronger model or coding agent can always pull the full example transcripts into context directly. What stronger models and agents alone do *not* give you is the surrounding platform. Here are some of the directions we are thinking about next:\n\nTo explore the dashboard locally on a synthetic month of runs and insights with **no cloud project, no credentials, and no model calls**:\n\n```\ngit clone https://github.com/google/adk-recipes.git\ncd adk-recipes/core/python/ambient-quality-agent\nmake demo         # serves the dashboard locally on synthetic data\n```\n\nTo attach AQuA to your own agent in Google Cloud with turnkey ADK scaffolding (all telemetry, source snapshots, and BigQuery tables stay inside your project under your service account, behind IAP):\n\n```\nagents-cli extension add \"${AQUA_CHECKOUT}\"\nagents-cli infra single-project --project=\"${GOOGLE_CLOUD_PROJECT}\" --apply\nagents-cli deploy --project=\"${GOOGLE_CLOUD_PROJECT}\"\n```\n\nTo drive both loops from your coding agent (Antigravity, Gemini CLI, Claude Code, or Cursor), install the **`agents-cli-aqua`** skill (` skills/agents-cli-aqua/SKILL.md`) alongside the inner-loop evaluation skill (` npx skills add https://github.com/google/agents-cli --skill google-agents-cli-eval`).\n\nWe are sharing AQuA in the open to collaborate with teams running agents in production and shape where this goes together. As you try it on your stack, we'd love to hear what you think:\n\n*Credits (alphabetical):**AQuA built by* Ákos Frohner*,* Aleksandra Grzegorczyk*,* Alessandro Grassi*,* Andrzej Kiewicz*,* Angelica Bilanenko*,* Dima Melnyk*,* Elia Secchi*,* Iwo Naglik*,* Lucas Matuszkowiak*,* Ludwik Trammer*,* Maciej Pawłowski*,* Max Gasztych*,* Pavel Sirotkin*,* Saksham Singhal*,* Xi Liu*,* Yaroslav Polyakov *and the broader Gemini platform team.*\n\n*Learn more:* [Ambient Quality Agent repository](https://github.com/google/adk-recipes/tree/main/core/python/ambient-quality-agent) · [Driving the Agent Quality Flywheel from Your Coding Agent](https://developers.googleblog.com/driving-the-agent-quality-flywheel-from-your-coding-agent/) · [Agent Evaluation docs](https://docs.cloud.google.com/gemini-enterprise-agent-platform/optimize/evaluation/agent-evaluation)", "url": "https://wpnews.pro/news/the-outer-loop-insights-first-an-ambient-quality-agent-that-diagnoses-your-agent", "canonical_source": "https://developers.googleblog.com/the-outer-loop-insights-first-an-ambient-quality-agent-that-diagnoses-your-production-agent/", "published_at": "2026-10-22 17:43:14.547005+00:00", "updated_at": "2026-10-22 17:43:16.604654+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "mlops", "developer-tools", "ai-safety"], "entities": ["Google", "AQuA", "Google Cloud", "Cloud Trace", "Cloud Logging", "BigQuery", "Gemini", "adk-recipes"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-outer-loop-insights-first-an-ambient-quality-agent-that-diagnoses-your-agent", "markdown": "https://wpnews.pro/news/the-outer-loop-insights-first-an-ambient-quality-agent-that-diagnoses-your-agent.md", "text": "https://wpnews.pro/news/the-outer-loop-insights-first-an-ambient-quality-agent-that-diagnoses-your-agent.txt", "jsonld": "https://wpnews.pro/news/the-outer-loop-insights-first-an-ambient-quality-agent-that-diagnoses-your-agent.jsonld"}}