{"slug": "agent-platform-in-action-6-towards-production-grade-ai-agent-ops-building-full", "title": "[Agent Platform in Action (6)] Towards Production-Grade AI Agent Ops: Building Full-Stack Observable LINE Agents with agents-cli", "summary": "A developer wrapped up a six-part series on the LINE AI Card Assistant by adding production AgentOps to the open-source Go project, using agents-cli scaffold enhance to generate Terraform infrastructure-as-code, GitHub Actions CI/CD pipelines, and full-stack Cloud Trace observability. The final phase deploys the ADK 2.0 graph workflow to Vertex AI Agent Runtime via the google_vertex_ai_reasoning_engine resource, with each graph node expanding into a nested span tree for tracing. The project also established an automated eval flywheel covering six golden cases and four metrics, all scoring 1.0000.", "body_md": "Welcome to the sixth (and final) article in the \"**Agent Platform in Action**\" series! Looking back at the past five articles, we have taken **[LINE AI Card Assistant (`kkdai/line-card-assistant`)](https://github.com/kkdai/line-card-assistant)** through the four major stages of evolution from a local prototype to an enterprise-grade AI Agent architecture:\n\n`google.golang.org/adk/v2/workflow` covering Vision OCR, `AddFanOut` / `AddFanIn` parallel background research, and `NewRequestInputEvent` HITL interaction loops.`agents-cli` (`adk_go`) dual-channel architecture, supporting LINE Webhook, ADK WebUI Playground, and A2A protocols.`artifact/gcsartifact` and `session/vertexai` to solve cross-instance HITL recovery and image storage issues in stateless containers.`PostgresStore`) and building a stateless Cloud Run LINE Webhook Gateway (` internal/gateway`).` agents-cli eval` to establish an automated quality flywheel covering 6 golden cases and 4 major evaluation metrics, all scoring `1.0000`.\nIn this final chapter today, we will complete the last piece of the puzzle for **[Roadmap #9](https://github.com/kkdai/line-card-assistant/issues/9)** — **Phase 5 ([Issue #8](https://github.com/kkdai/line-card-assistant/issues/8)): Official Agent Runtime Deployment, Terraform IaC, GitHub Actions CI/CD, and Full-Stack Cloud Trace Observability**!\n\n`gcloud` Commands\" to \"Declarative AgentOps\"\nIn many AI projects, even if the code and evaluation tests are written, production environments often become operational nightmares due to the following four issues:\n\n`go test` can catch syntax and logic errors, but if someone accidentally breaks a Graph Node's `state_delta` or Prompt, how can we automatically intercept it before the PR is merged into `main`?` VisionOCRNode` or in the parallel Fan-Out's `CompanyEnrichNode` (Google Search Grounding)? We need every node of the ADK 2.0 Graph Workflow to expand into a clear nested Span tree in This is exactly what the **Phase 5 Full-Stack Architecture Diagram** at the beginning of the article depicts through four core pillars:\n\n`.github/workflows/pr_checks.yaml`, `staging.yaml`, `deploy-to-prod.yaml`\n`deployment/terraform/single-project/`, `cicd/`, `shared/`\n`agents-cli deploy` + `internal/workflow/telemetry.go`\n`observability.go` $\\rightarrow$ Cloud Trace + Cloud Logging $\\rightarrow$ `telemetry.tf` BigQuery Sink`agents-cli scaffold enhance`\nIn Phase 1, we introduced the Prototype-First philosophy of `agents-cli`: keep the project lightweight first, and then execute `agents-cli scaffold enhance` to layer on production infrastructure when ready for the cloud.\n\nWe execute the following directly in the project root directory:\n\n```\nuvx google-agents-cli@1.10.0 scaffold enhance . \\\n  --cicd-runner github_actions \\\n  --region asia-east1 \\\n  --yes\n```\n\n`agents-cli` automatically detected our `agents-cli-manifest.yaml` (`template: adk_go`, `deployment_target: agent_runtime`) and, without breaking any existing `app/`, `internal/`, `eval/`, or `sessions/` code, added three major modules to the project:\n\n`deployment/terraform/cicd/`` deployment/terraform/shared/telemetry.tf` and `storage.tf``.github/workflows/` and `e2e/`\nNext, we perform customized enhancements for the specific requirements of **LINE AI Card Assistant** (LINE credentials, Cloud SQL PostgreSQL permissions, Taiwan `asia-east1` region)!\n\nLet's open `deployment/terraform/` and see how Terraform declaratively builds the entire Google Cloud Agent Platform:\n\n`service.tf`)\nIn [`deployment/terraform/single-project/service.tf`](https://github.com/kkdai/line-card-assistant/blob/main/deployment/terraform/single-project/service.tf), we use the `google_vertex_ai_reasoning_engine` resource from the `google-beta` provider to package the Go source code and declare it as a managed Agent Runtime instance. We also set `lifecycle { ignore_changes = [spec[0].source_code_spec] }` so that daily code updates are handled by the CI/CD's `agents-cli deploy` for fast rolling deployment, while infrastructure settings (IAM, Buckets, environment variables) are unified under Terraform control:\n\n```\nresource \"google_vertex_ai_reasoning_engine\" \"app\" {\n  provider = google-beta\n  display_name = var.project_name\n  project = var.project_id\n  region = var.region\n\n  labels = {\n    \"agent-framework\" = var.agent_framework\n  }\n\n  spec {\n    identity_type = \"AGENT_IDENTITY\"\n    agent_framework = var.agent_framework\n\n    source_code_spec {\n      inline_source {\n        source_archive = data.archive_file.source.output_path\n      }\n      developer_connect_source {}\n    }\n\n    container_spec {\n      image_uri = \"us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-15:latest\"\n    }\n  }\n\n  lifecycle {\n    ignore_changes = [\n      spec[0].source_code_spec,\n      spec[0].container_spec,\n    ]\n  }\n}\n```\n\n`iam.tf` & `variables.tf`)\nTo prevent `LINE_CHANNEL_SECRET`, `LINE_CHANNEL_ACCESS_TOKEN`, and `DATABASE_URL` from appearing in plain text in configuration files, we declare **Google Cloud Secret Manager** secrets in [`deployment/terraform/single-project/iam.tf`](https://github.com/kkdai/line-card-assistant/blob/main/deployment/terraform/single-project/iam.tf) and [`variables.tf`](https://github.com/kkdai/line-card-assistant/blob/main/deployment/terraform/single-project/variables.tf). We grant `roles/secretmanager.secretAccessor` and `roles/cloudsql.client` to both the **Vertex AI Service Agent (`gcp-sa-aiplatform-re`)** and the **Agent Identity Principal (` principal://${google_vertex_ai_reasoning_engine.app.spec[0].effective_identity}`)**:\n\n```\n# Secret Manager secrets for LINE Messaging API credentials and Cloud SQL DSN\nlocals {\n  line_card_secrets = [\n    \"line-channel-secret\",\n    \"line-channel-access-token\",\n    \"line-card-database-url\",\n  ]\n}\n\nresource \"google_secret_manager_secret\" \"app_secrets\" {\n  for_each = toset(local.line_card_secrets)\n  project = var.project_id\n  secret_id = each.value\n\n  replication {\n    auto {}\n  }\n\n  depends_on = [resource.google_project_service.services]\n}\n```\n\nThis way, during deployment, you only need to pass `--secrets=\"LINE_CHANNEL_SECRET=line-channel-secret,LINE_CHANNEL_ACCESS_TOKEN=line-channel-access-token\"` to `agents-cli deploy`, and the Agent Runtime will securely mount the secrets from Secret Manager when starting the container!\n\nIn [`deployment/terraform/shared/telemetry.tf`](https://github.com/kkdai/line-card-assistant/blob/main/deployment/terraform/shared/telemetry.tf), Terraform automatically creates a Cloud Logging Bucket with a 365-day retention period, enables `enable_analytics = true`, and directly maps telemetry logs to a **BigQuery Dataset (` telemetry_logs_dataset`)** via `google_logging_linked_dataset`:\n\n```\nresource \"google_logging_project_bucket_config\" \"telemetry_logs_bucket\" {\n  project = each.value\n  location = \"global\"\n  bucket_id = \"${var.project_name}-telemetry\"\n  retention_days = 365\n  enable_analytics = true\n}\n\nresource \"google_logging_linked_dataset\" \"telemetry_logs_dataset\" {\n  bucket = google_logging_project_bucket_config.telemetry_logs_bucket[each.key].bucket_id\n  link_id = replace(\"${var.project_name}_telemetry\", \"-\", \"_\")\n  project = each.value\n  location = \"global\"\n}\n```\n\nThis means all structured telemetry data output by OpenTelemetry can be analyzed directly in BigQuery using standard SQL without writing additional ETL scripts!\n\nIn the `observability.go` generated in Phase 1, the system was already configured with an OTLP HTTP Exporter pointing to `telemetry.googleapis.com` and Google ADC automatic authentication. However, by default, we only see top-level traces for the entire HTTP request or LLM call, but not the **execution time and routing decisions of each `FunctionNode` within the ADK 2.0 Graph Workflow**.\n\nTo present the processing of each business card as a complete waterfall span tree on **Google Cloud Trace**, we added [`internal/workflow/telemetry.go`](https://github.com/kkdai/line-card-assistant/blob/main/internal/workflow/telemetry.go):\n\n```\npackage workflow\n\nimport (\n    \"context\"\n\n    \"go.opentelemetry.io/otel\"\n    \"go.opentelemetry.io/otel/attribute\"\n    \"go.opentelemetry.io/otel/codes\"\n    \"go.opentelemetry.io/otel/trace\"\n)\n\nconst workflowTracerName = \"github.com/kkdai/line-card-assistant/internal/workflow\"\n\nconst (\n    AttrWorkflowNode = attribute.Key(\"adk.workflow.node\")\n    AttrWorkflowRoute = attribute.Key(\"adk.workflow.route\")\n    AttrInvocationID = attribute.Key(\"adk.invocation.id\")\n    AttrSessionID = attribute.Key(\"adk.session.id\")\n    AttrUserID = attribute.Key(\"adk.user.id\")\n    AttrCardID = attribute.Key(\"adk.card.id\")\n    AttrCardCompany = attribute.Key(\"adk.card.company\")\n    AttrCardDuplicate = attribute.Key(\"adk.card.is_duplicate\")\n    AttrHITLInterrupt = attribute.Key(\"adk.hitl.interrupt_id\")\n    AttrHITLAction = attribute.Key(\"adk.hitl.action\")\n    AttrSearchMatches = attribute.Key(\"adk.search.matches\")\n    AttrSearchKeywords = attribute.Key(\"adk.search.keywords\")\n)\n\nfunc startNodeSpan(ctx context.Context, nodeName string, attrs ...attribute.KeyValue) (context.Context, trace.Span) {\n    baseAttrs := make([]attribute.KeyValue, 0, len(attrs)+1)\n    baseAttrs = append(baseAttrs, AttrWorkflowNode.String(nodeName))\n    baseAttrs = append(baseAttrs, attrs...)\n    return otel.Tracer(workflowTracerName).Start(\n        ctx,\n        \"workflow.node.\"+nodeName,\n        trace.WithAttributes(baseAttrs...),\n    )\n}\n```\n\nNext, we inject `startNodeSpan` and `endNodeSpan` into all nodes in `internal/workflow/nodes_ingest.go` and `internal/workflow/nodes_search.go`. For example, in `HybridSearchNode`:\n\n```\nfunc newHybridSearchNode(cardStore store.CardStore) *workflow.FunctionNode {\n    return workflow.NewFunctionNode(\n        NodeHybridSearch,\n        func(ctx agent.Context, _ any) (ev *session.Event, err error) {\n            spanCtx, span := startNodeSpan(ctx, NodeHybridSearch, AttrInvocationID.String(ctx.InvocationID()))\n            defer func() {\n                routeAttr := \"\"\n                if ev != nil && len(ev.Routes) > 0 {\n                    routeAttr = ev.Routes[0]\n                }\n                endNodeSpan(span, err, AttrWorkflowRoute.String(routeAttr))\n            }()\n            // ... execute cardStore.HybridSearch(spanCtx, filter) ...\n            span.SetAttributes(AttrSearchMatches.Int(len(cards)))\n            // ...\n        },\n        workflow.NodeConfig{},\n    )\n}\n```\n\nWhen a user uploads a business card or performs a hybrid search, Cloud Trace automatically draws a hierarchical Span tree like the one below, including the time spent in the three child nodes of the parallel Fan-Out and the business attributes of each node:\n\n```\nflowchart TD\n    Root[\"invoke_workflow (line_card_assistant)<br/>trace_id: 4bf92f3577b34da6a3ce929d0e0e4736\"] --> R[\"workflow.node.RouterNode<br/>adk.workflow.route = 'ingest' (1.2ms)\"]\n    R --> V[\"workflow.node.VisionOCRNode<br/>adk.card.company = 'Taiwan AI Labs' (640ms)\"]\n    V --> C1[\"workflow.node.CleanAndDedupNode<br/>adk.card.is_duplicate = false (8.5ms)\"]\n    V --> C2[\"workflow.node.CompanyEnrichNode<br/>GoogleSearch Grounding (780ms)\"]\n    V --> C3[\"workflow.node.ContextMatchNode<br/>inferMetContext (310ms)\"]\n    C1 & C2 & C3 --> B[\"workflow.node.BuildPreviewNode<br/>Flex Bubble JSON (2.1ms)\"]\n    B --> H[\"workflow.node.HITLConfirmNode<br/>adk.hitl.interrupt_id = 'hitl_confirm_1'<br/>adk.hitl.action = 'pause' (0.8ms)\"]\n```\n\nWe also wrote automated unit tests in [`internal/workflow/telemetry_test.go`](https://github.com/kkdai/line-card-assistant/blob/main/internal/workflow/telemetry_test.go) using the OpenTelemetry SDK's `tracetest.NewInMemoryExporter()`, ensuring that \"card recognition + HITL modification loop + confirmation save\" and \"hybrid search with 0 results automatically degrading to fuzzy search\" 100% generate the expected Span nodes and attribute tags!\n\nWith Terraform IaC and OpenTelemetry tracing in place, we officially embedded the **`agents-cli eval` quality flywheel** created in Phase 4 into the GitHub Actions CI/CD pipeline, forming a two-stage automated delivery:\n\n```\nflowchart LR\n    subgraph PR[\"1. Pull Request Quality Gate (pr_checks.yaml)\"]\n        P1[\"go test -race -v ./...<br/>(Unit + OTel Span + Trajectory Tests)\"] --> P2[\"go test -tags=e2e<br/>(A2A + /app-info E2E Tests)\"]\n        P2 --> P3[\"agents-cli eval run<br/>(4 major metrics must = 1.0000)\"]\n    end\n\n    subgraph CD[\"2. Staging & Production CD (staging.yaml -> deploy-to-prod.yaml)\"]\n        S1[\"WIF Keyless Auth<br/>(google-github-actions/auth)\"] --> S2[\"agents-cli deploy<br/>Deploy to Staging Agent Runtime\"]\n        S2 --> S3[\"Vegeta Load Test<br/>30s pressure test + report upload to GCS\"]\n        S3 --> S4[\"GitHub Environment<br/>Production Manual Approval Gate\"]\n        S4 --> S5[\"agents-cli deploy<br/>Deploy to Production Agent Runtime\"]\n    end\n\n    PR -->|Merge to main| CD\n```\n\nIn [`.github/workflows/pr_checks.yaml`](https://github.com/kkdai/line-card-assistant/blob/main/.github/workflows/pr_checks.yaml), whenever a PR modifies `app/**`, `internal/**`, `cmd/**`, `sessions/**`, or `eval/**`, GitHub Actions will execute in order:\n\n`go test -race -v ./...`: Verify all Go unit tests, cross-instance HITL recovery, OpenTelemetry Node Spans, and trajectory tests.`go test -v -tags=e2e ./e2e/integration/...`: Start a real ADK Universal Launcher subprocess to verify `/.well-known/agent-card.json`, `/a2a/invoke`, and `/apps/app/app-info` endpoints.`agents-cli eval run` Quality Gatekeeper`workflow_trajectory_accuracy`, `ocr_and_enrichment_quality`, `duplicate_and_hitl_precision`, `search_and_email_grounding`) must maintain a perfect score!`.github/workflows/staging.yaml`)\nAfter the PR is merged into the `main` branch, [`.github/workflows/staging.yaml`](https://github.com/kkdai/line-card-assistant/blob/main/.github/workflows/staging.yaml) will:\n\n`uvx google-agents-cli@1.10.0 deploy` to deploy the latest commit to the Staging Vertex AI Agent Runtime, while mounting `LINE_CHANNEL_SECRET` and `LINE_CHANNEL_ACCESS_TOKEN` from Secret Manager and tagging `OTEL_RESOURCE_ATTRIBUTES=service.version=$`.` e2e/load_test/load_test.go``tsenart/vegeta/v12`) to perform a 30-second continuous pressure test on the Staging Agent Runtime, checking HTTP 200 success rate, P50/P95/P99 latency, and `429 Too Many Requests` ratio, and automatically archiving the results to `gs://${LOGS_BUCKET_NAME_STAGING}/load-test-results/`.`.github/workflows/deploy-to-prod.yaml`` production` environment, it is officially pushed to Production!`agents-cli deploy --dry-run` and Full E2E Tests All Passed!\nBefore submitting the code as a PR, we fully executed three major verifications locally:\n\n``` bash\n$ uvx google-agents-cli@1.10.0 deploy \\\n    --project line-vertex \\\n    --region asia-east1 \\\n    --dry-run \\\n    --no-confirm-project\n```\n\nPreflight output results:\n\n`aiplatform.googleapis.com` API`line-card-assistant` (`asia-east1`) ✅\n\n``` bash\n$ go test -v -tags=e2e ./...\n```\n\nIncludes the newly added `TestWorkflowOpenTelemetrySpans_IngestAndHITL`, `TestWorkflowOpenTelemetrySpans_SearchFallback`, and all tests under `e2e/integration` such as `TestA2AAgentCard`, `TestA2AMessageSend`, `TestA2AErrorHandling`, and `TestAppInfo` — all **PASS**!\n\n``` bash\n$ DRY_RUN=true make eval\n```\n\nThe 6 golden cases across the 4 custom metrics (`workflow_trajectory_accuracy`, `ocr_and_enrichment_quality`, `duplicate_and_hitl_precision`, `search_and_email_grounding`) still maintain **` 1.0000` (100% Pass)**!\n\nWith the completion of **Phase 5 ([Issue #8](https://github.com/kkdai/line-card-assistant/issues/8))**, the five stages planned in **[Roadmap Issue #9](https://github.com/kkdai/line-card-assistant/issues/9)** have been **100% achieved and closed**! Let's conclude with a summary table reviewing this journey from 0 to 1 with **Google ADK 2.0 Go + Google Cloud Agent Platform**:\n\n| Stage | Corresponding Issue / PR | Core Technical Breakthroughs | Corresponding Blog Post | \n|---|---|---|---|\n| **Prototype** | Initial Commit | Go `adk/v2/workflow` 12-node Graph Workflow, parallel`AddFanOut` /`AddFanIn` ,`NewRequestInputEvent` HITL loop | [In Action (1)](https://dev.to/2026/10/16/adk-graph-workflow-go/) | \n| **Phase 1** | [Issue #4](https://github.com/kkdai/line-card-assistant/issues/4) /[PR #10](https://github.com/kkdai/line-card-assistant/pull/10) | `agents-cli` (`adk_go` ) standardization,`app.NewRootAgent` dual-channel factory, ADK WebUI Playground, and A2A protocol support | [In Action (2)](https://dev.to/2026/10/17/adk-graph-workflow-go-2/) | \n| **Phase 2 (Part 1)** | [Issue #5](https://github.com/kkdai/line-card-assistant/issues/5) /[PR #11](https://github.com/kkdai/line-card-assistant/pull/11) | `artifact/gcsartifact` zero-state image bloat,`session/vertexai` cross-instance stateless HITL pause and resume | [In Action (3)](https://dev.to/2026/10/20/adk-graph-workflow-go-3/) | \n| **Phase 2 (Part 2) & Phase 3** | [Issue #5](https://github.com/kkdai/line-card-assistant/issues/5) ,[#6](https://github.com/kkdai/line-card-assistant/issues/6) /[PR #12](https://github.com/kkdai/line-card-assistant/pull/12) | Cloud SQL PostgreSQL ( `PostgresStore` ) card persistence, Cloud Run LINE Webhook Gateway (`internal/gateway` ) and Agent Runtime dual-layer decoupling | [In Action (4)](https://dev.to/2026/10/21/adk-graph-workflow-go-4/) | \n| **Phase 4** | [Issue #7](https://github.com/kkdai/line-card-assistant/issues/7) /[PR #13](https://github.com/kkdai/line-card-assistant/pull/13) | `agents-cli eval` quality flywheel,`/app-info` reflection introspection, 6 golden cases, and`setNodeTrace` trajectory repair (all 4 metrics at`1.0000` ) | [In Action (5)](https://dev.to/2026/10/22/adk-graph-workflow-go-5/) | \n| **Phase 5** | [Issue #8](https://github.com/kkdai/line-card-assistant/issues/8) /[PR #14](https://github.com/kkdai/line-card-assistant/pull/14) | `agents-cli deploy` , Terraform IaC (`deployment/terraform/` ), GitHub Actions CI/CD + WIF + Vegeta pressure testing, OpenTelemetry node-level Cloud Trace & BigQuery observation | **This article (In Action 6)** | \n\nThe complete open-source code, Terraform modules, and evaluation datasets have been synchronized to the GitHub Repo: **[`kkdai/line-card-assistant`](https://github.com/kkdai/line-card-assistant)**. If you are also building enterprise-grade AI Agents with Go and Google ADK 2.0, I hope this series serves as your best practical blueprint for moving from local development all the way to the Google Cloud Agent Platform production environment!", "url": "https://wpnews.pro/news/agent-platform-in-action-6-towards-production-grade-ai-agent-ops-building-full", "canonical_source": "https://dev.to/gde/agent-platform-in-action-6-towards-production-grade-ai-agent-ops-building-full-stack-38aa", "published_at": "2026-10-25 10:05:18+00:00", "updated_at": "2026-10-25 10:13:44.040836+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "mlops", "developer-tools", "ai-tools"], "entities": ["LINE", "agents-cli", "Google Cloud", "Vertex AI Agent Runtime", "Cloud Trace", "Terraform", "GitHub Actions", "ADK"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/agent-platform-in-action-6-towards-production-grade-ai-agent-ops-building-full", "markdown": "https://wpnews.pro/news/agent-platform-in-action-6-towards-production-grade-ai-agent-ops-building-full.md", "text": "https://wpnews.pro/news/agent-platform-in-action-6-towards-production-grade-ai-agent-ops-building-full.txt", "jsonld": "https://wpnews.pro/news/agent-platform-in-action-6-towards-production-grade-ai-agent-ops-building-full.jsonld"}}