[Agent Platform in Action (6)] Towards Production-Grade AI Agent Ops: Building Full-Stack Observable LINE Agents with agents-cli

Welcome to the sixth (and final) article in the "Agent Platform in Action" series! Looking back at the past five articles, we have taken LINE AI Card Assistant (kkdai/line-card-assistant) through the four major stages of evolution from a local prototype to an enterprise-grade AI Agent architecture:

google.golang.org/adk/v2/workflow covering Vision OCR, AddFanOut / AddFanIn parallel background research, and NewRequestInputEvent HITL interaction loops.agents-cli (adk_go) dual-channel architecture, supporting LINE Webhook, ADK WebUI Playground, and A2A protocols.artifact/gcsartifact and session/vertexai to solve cross-instance HITL recovery and image storage issues in stateless containers.PostgresStore) and building a stateless Cloud Run LINE Webhook Gateway ( internal/gateway). agents-cli eval to establish an automated quality flywheel covering 6 golden cases and 4 major evaluation metrics, all scoring 1.0000. In this final chapter today, we will complete the last piece of the puzzle for Roadmap #9 — Phase 5 (Issue #8): Official Agent Runtime Deployment, Terraform IaC, GitHub Actions CI/CD, and Full-Stack Cloud Trace Observability!

gcloud Commands" to "Declarative AgentOps" In many AI projects, even if the code and evaluation tests are written, production environments often become operational nightmares due to the following four issues:

go test can catch syntax and logic errors, but if someone accidentally breaks a Graph Node's state_delta or Prompt, how can we automatically intercept it before the PR is merged into main? VisionOCRNode or in the parallel Fan-Out's CompanyEnrichNode (Google Search Grounding)? We need every node of the ADK 2.0 Graph Workflow to expand into a clear nested Span tree in This is exactly what the Phase 5 Full-Stack Architecture Diagram at the beginning of the article depicts through four core pillars:

.github/workflows/pr_checks.yaml, staging.yaml, deploy-to-prod.yaml deployment/terraform/single-project/, cicd/, shared/ agents-cli deploy + internal/workflow/telemetry.go observability.go $\rightarrow$ Cloud Trace + Cloud Logging $\rightarrow$ telemetry.tf BigQuery Sinkagents-cli scaffold enhance In Phase 1, we introduced the Prototype-First philosophy of agents-cli: keep the project lightweight first, and then execute agents-cli scaffold enhance to layer on production infrastructure when ready for the cloud.

We execute the following directly in the project root directory:

uvx google-agents-cli@1.10.0 scaffold enhance . \
  --cicd-runner github_actions \
  --region asia-east1 \
  --yes

agents-cli automatically detected our agents-cli-manifest.yaml (template: adk_go, deployment_target: agent_runtime) and, without breaking any existing app/, internal/, eval/, or sessions/ code, added three major modules to the project:

deployment/terraform/cicd/`` deployment/terraform/shared/telemetry.tf and storage.tf``.github/workflows/ and e2e/ Next, we perform customized enhancements for the specific requirements of LINE AI Card Assistant (LINE credentials, Cloud SQL PostgreSQL permissions, Taiwan asia-east1 region)!

Let's open deployment/terraform/ and see how Terraform declaratively builds the entire Google Cloud Agent Platform:

service.tf) In deployment/terraform/single-project/service.tf, we use the google_vertex_ai_reasoning_engine resource from the google-beta provider to package the Go source code and declare it as a managed Agent Runtime instance. We also set lifecycle { ignore_changes = [spec[0].source_code_spec] } so that daily code updates are handled by the CI/CD's agents-cli deploy for fast rolling deployment, while infrastructure settings (IAM, Buckets, environment variables) are unified under Terraform control:

resource "google_vertex_ai_reasoning_engine" "app" {
  provider = google-beta
  display_name = var.project_name
  project = var.project_id
  region = var.region

  labels = {
    "agent-framework" = var.agent_framework
  }

  spec {
    identity_type = "AGENT_IDENTITY"
    agent_framework = var.agent_framework

    source_code_spec {
      inline_source {
        source_archive = data.archive_file.source.output_path
      }
      developer_connect_source {}
    }

    container_spec {
      image_uri = "us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-15:latest"
    }
  }

  lifecycle {
    ignore_changes = [
      spec[0].source_code_spec,
      spec[0].container_spec,
    ]
  }
}

iam.tf & variables.tf) To prevent LINE_CHANNEL_SECRET, LINE_CHANNEL_ACCESS_TOKEN, and DATABASE_URL from appearing in plain text in configuration files, we declare Google Cloud Secret Manager secrets in deployment/terraform/single-project/iam.tf and variables.tf. We grant roles/secretmanager.secretAccessor and roles/cloudsql.client to both the Vertex AI Service Agent (gcp-sa-aiplatform-re) and the Agent Identity Principal ( principal://${google_vertex_ai_reasoning_engine.app.spec[0].effective_identity}):

locals {
  line_card_secrets = [
    "line-channel-secret",
    "line-channel-access-token",
    "line-card-database-url",
  ]
}

resource "google_secret_manager_secret" "app_secrets" {
  for_each = toset(local.line_card_secrets)
  project = var.project_id
  secret_id = each.value

  replication {
    auto {}
  }

  depends_on = [resource.google_project_service.services]
}

This way, during deployment, you only need to pass --secrets="LINE_CHANNEL_SECRET=line-channel-secret,LINE_CHANNEL_ACCESS_TOKEN=line-channel-access-token" to agents-cli deploy, and the Agent Runtime will securely mount the secrets from Secret Manager when starting the container!

In deployment/terraform/shared/telemetry.tf, Terraform automatically creates a Cloud Logging Bucket with a 365-day retention period, enables enable_analytics = true, and directly maps telemetry logs to a BigQuery Dataset ( telemetry_logs_dataset) via google_logging_linked_dataset:

resource "google_logging_project_bucket_config" "telemetry_logs_bucket" {
  project = each.value
  location = "global"
  bucket_id = "${var.project_name}-telemetry"
  retention_days = 365
  enable_analytics = true
}

resource "google_logging_linked_dataset" "telemetry_logs_dataset" {
  bucket = google_logging_project_bucket_config.telemetry_logs_bucket[each.key].bucket_id
  link_id = replace("${var.project_name}_telemetry", "-", "_")
  project = each.value
  location = "global"
}

This means all structured telemetry data output by OpenTelemetry can be analyzed directly in BigQuery using standard SQL without writing additional ETL scripts!

In the observability.go generated in Phase 1, the system was already configured with an OTLP HTTP Exporter pointing to telemetry.googleapis.com and Google ADC automatic authentication. However, by default, we only see top-level traces for the entire HTTP request or LLM call, but not the execution time and routing decisions of each FunctionNode within the ADK 2.0 Graph Workflow.

To present the processing of each business card as a complete waterfall span tree on Google Cloud Trace, we added internal/workflow/telemetry.go:

package workflow

import (
    "context"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/codes"
    "go.opentelemetry.io/otel/trace"
)

const workflowTracerName = "github.com/kkdai/line-card-assistant/internal/workflow"

const (
    AttrWorkflowNode = attribute.Key("adk.workflow.node")
    AttrWorkflowRoute = attribute.Key("adk.workflow.route")
    AttrInvocationID = attribute.Key("adk.invocation.id")
    AttrSessionID = attribute.Key("adk.session.id")
    AttrUserID = attribute.Key("adk.user.id")
    AttrCardID = attribute.Key("adk.card.id")
    AttrCardCompany = attribute.Key("adk.card.company")
    AttrCardDuplicate = attribute.Key("adk.card.is_duplicate")
    AttrHITLInterrupt = attribute.Key("adk.hitl.interrupt_id")
    AttrHITLAction = attribute.Key("adk.hitl.action")
    AttrSearchMatches = attribute.Key("adk.search.matches")
    AttrSearchKeywords = attribute.Key("adk.search.keywords")
)

func startNodeSpan(ctx context.Context, nodeName string, attrs ...attribute.KeyValue) (context.Context, trace.Span) {
    baseAttrs := make([]attribute.KeyValue, 0, len(attrs)+1)
    baseAttrs = append(baseAttrs, AttrWorkflowNode.String(nodeName))
    baseAttrs = append(baseAttrs, attrs...)
    return otel.Tracer(workflowTracerName).Start(
        ctx,
        "workflow.node."+nodeName,
        trace.WithAttributes(baseAttrs...),
    )
}

Next, we inject startNodeSpan and endNodeSpan into all nodes in internal/workflow/nodes_ingest.go and internal/workflow/nodes_search.go. For example, in HybridSearchNode:

func newHybridSearchNode(cardStore store.CardStore) *workflow.FunctionNode {
    return workflow.NewFunctionNode(
        NodeHybridSearch,
        func(ctx agent.Context, _ any) (ev *session.Event, err error) {
            spanCtx, span := startNodeSpan(ctx, NodeHybridSearch, AttrInvocationID.String(ctx.InvocationID()))
            defer func() {
                routeAttr := ""
                if ev != nil && len(ev.Routes) > 0 {
                    routeAttr = ev.Routes[0]
                }
                endNodeSpan(span, err, AttrWorkflowRoute.String(routeAttr))
            }()
            // ... execute cardStore.HybridSearch(spanCtx, filter) ...
            span.SetAttributes(AttrSearchMatches.Int(len(cards)))
            // ...
        },
        workflow.NodeConfig{},
    )
}

When a user uploads a business card or performs a hybrid search, Cloud Trace automatically draws a hierarchical Span tree like the one below, including the time spent in the three child nodes of the parallel Fan-Out and the business attributes of each node:

flowchart TD
    Root["invoke_workflow (line_card_assistant)<br/>trace_id: 4bf92f3577b34da6a3ce929d0e0e4736"] --> R["workflow.node.RouterNode<br/>adk.workflow.route = 'ingest' (1.2ms)"]
    R --> V["workflow.node.VisionOCRNode<br/>adk.card.company = 'Taiwan AI Labs' (640ms)"]
    V --> C1["workflow.node.CleanAndDedupNode<br/>adk.card.is_duplicate = false (8.5ms)"]
    V --> C2["workflow.node.CompanyEnrichNode<br/>GoogleSearch Grounding (780ms)"]
    V --> C3["workflow.node.ContextMatchNode<br/>inferMetContext (310ms)"]
    C1 & C2 & C3 --> B["workflow.node.BuildPreviewNode<br/>Flex Bubble JSON (2.1ms)"]
    B --> H["workflow.node.HITLConfirmNode<br/>adk.hitl.interrupt_id = 'hitl_confirm_1'<br/>adk.hitl.action = '' (0.8ms)"]

We also wrote automated unit tests in internal/workflow/telemetry_test.go using the OpenTelemetry SDK's tracetest.NewInMemoryExporter(), ensuring that "card recognition + HITL modification loop + confirmation save" and "hybrid search with 0 results automatically degrading to fuzzy search" 100% generate the expected Span nodes and attribute tags!

With Terraform IaC and OpenTelemetry tracing in place, we officially embedded the agents-cli eval quality flywheel created in Phase 4 into the GitHub Actions CI/CD pipeline, forming a two-stage automated delivery:

flowchart LR
    subgraph PR["1. Pull Request Quality Gate (pr_checks.yaml)"]
        P1["go test -race -v ./...<br/>(Unit + OTel Span + Trajectory Tests)"] --> P2["go test -tags=e2e<br/>(A2A + /app-info E2E Tests)"]
        P2 --> P3["agents-cli eval run<br/>(4 major metrics must = 1.0000)"]
    end

    subgraph CD["2. Staging & Production CD (staging.yaml -> deploy-to-prod.yaml)"]
        S1["WIF Keyless Auth<br/>(google-github-actions/auth)"] --> S2["agents-cli deploy<br/>Deploy to Staging Agent Runtime"]
        S2 --> S3["Vegeta Load Test<br/>30s pressure test + report upload to GCS"]
        S3 --> S4["GitHub Environment<br/>Production Manual Approval Gate"]
        S4 --> S5["agents-cli deploy<br/>Deploy to Production Agent Runtime"]
    end

    PR -->|Merge to main| CD

In .github/workflows/pr_checks.yaml, whenever a PR modifies app/**, internal/**, cmd/**, sessions/**, or eval/**, GitHub Actions will execute in order:

go test -race -v ./...: Verify all Go unit tests, cross-instance HITL recovery, OpenTelemetry Node Spans, and trajectory tests.go test -v -tags=e2e ./e2e/integration/...: Start a real ADK Universal Launcher subprocess to verify /.well-known/agent-card.json, /a2a/invoke, and /apps/app/app-info endpoints.agents-cli eval run Quality Gatekeeperworkflow_trajectory_accuracy, ocr_and_enrichment_quality, duplicate_and_hitl_precision, search_and_email_grounding) must maintain a perfect score!.github/workflows/staging.yaml) After the PR is merged into the main branch, .github/workflows/staging.yaml will:

uvx google-agents-cli@1.10.0 deploy to deploy the latest commit to the Staging Vertex AI Agent Runtime, while mounting LINE_CHANNEL_SECRET and LINE_CHANNEL_ACCESS_TOKEN from Secret Manager and tagging OTEL_RESOURCE_ATTRIBUTES=service.version=$. e2e/load_test/load_test.go``tsenart/vegeta/v12) to perform a 30-second continuous pressure test on the Staging Agent Runtime, checking HTTP 200 success rate, P50/P95/P99 latency, and 429 Too Many Requests ratio, and automatically archiving the results to gs://${LOGS_BUCKET_NAME_STAGING}/load-test-results/..github/workflows/deploy-to-prod.yaml`` production environment, it is officially pushed to Production!agents-cli deploy --dry-run and Full E2E Tests All Passed! Before submitting the code as a PR, we fully executed three major verifications locally:

$ uvx google-agents-cli@1.10.0 deploy \
    --project line-vertex \
    --region asia-east1 \
    --dry-run \
    --no-confirm-project

Preflight output results:

aiplatform.googleapis.com APIline-card-assistant (asia-east1) ✅

$ go test -v -tags=e2e ./...

Includes the newly added TestWorkflowOpenTelemetrySpans_IngestAndHITL, TestWorkflowOpenTelemetrySpans_SearchFallback, and all tests under e2e/integration such as TestA2AAgentCard, TestA2AMessageSend, TestA2AErrorHandling, and TestAppInfo — all PASS!

$ DRY_RUN=true make eval

The 6 golden cases across the 4 custom metrics (workflow_trajectory_accuracy, ocr_and_enrichment_quality, duplicate_and_hitl_precision, search_and_email_grounding) still maintain 1.0000 (100% Pass)!

With the completion of Phase 5 (Issue #8), the five stages planned in Roadmap Issue #9 have been 100% achieved and closed! Let's conclude with a summary table reviewing this journey from 0 to 1 with Google ADK 2.0 Go + Google Cloud Agent Platform:

The complete open-source code, Terraform modules, and evaluation datasets have been synchronized to the GitHub Repo: kkdai/line-card-assistant. If you are also building enterprise-grade AI Agents with Go and Google ADK 2.0, I hope this series serves as your best practical blueprint for moving from local development all the way to the Google Cloud Agent Platform production environment!