[Agent Platform in Action (6)] Towards Production-Grade AI Agent Ops: Building Full-Stack Observable LINE Agents with agents-cli A developer wrapped up a six-part series on the LINE AI Card Assistant by adding production AgentOps to the open-source Go project, using agents-cli scaffold enhance to generate Terraform infrastructure-as-code, GitHub Actions CI/CD pipelines, and full-stack Cloud Trace observability. The final phase deploys the ADK 2.0 graph workflow to Vertex AI Agent Runtime via the google_vertex_ai_reasoning_engine resource, with each graph node expanding into a nested span tree for tracing. The project also established an automated eval flywheel covering six golden cases and four metrics, all scoring 1.0000. Welcome to the sixth and final article in the " Agent Platform in Action " series Looking back at the past five articles, we have taken LINE AI Card Assistant kkdai/line-card-assistant https://github.com/kkdai/line-card-assistant through the four major stages of evolution from a local prototype to an enterprise-grade AI Agent architecture: google.golang.org/adk/v2/workflow covering Vision OCR, AddFanOut / AddFanIn parallel background research, and NewRequestInputEvent HITL interaction loops. agents-cli adk go dual-channel architecture, supporting LINE Webhook, ADK WebUI Playground, and A2A protocols. artifact/gcsartifact and session/vertexai to solve cross-instance HITL recovery and image storage issues in stateless containers. PostgresStore and building a stateless Cloud Run LINE Webhook Gateway internal/gateway . agents-cli eval to establish an automated quality flywheel covering 6 golden cases and 4 major evaluation metrics, all scoring 1.0000 . In this final chapter today, we will complete the last piece of the puzzle for Roadmap 9 https://github.com/kkdai/line-card-assistant/issues/9 — Phase 5 Issue 8 https://github.com/kkdai/line-card-assistant/issues/8 : Official Agent Runtime Deployment, Terraform IaC, GitHub Actions CI/CD, and Full-Stack Cloud Trace Observability gcloud Commands" to "Declarative AgentOps" In many AI projects, even if the code and evaluation tests are written, production environments often become operational nightmares due to the following four issues: go test can catch syntax and logic errors, but if someone accidentally breaks a Graph Node's state delta or Prompt, how can we automatically intercept it before the PR is merged into main ? VisionOCRNode or in the parallel Fan-Out's CompanyEnrichNode Google Search Grounding ? We need every node of the ADK 2.0 Graph Workflow to expand into a clear nested Span tree in This is exactly what the Phase 5 Full-Stack Architecture Diagram at the beginning of the article depicts through four core pillars: .github/workflows/pr checks.yaml , staging.yaml , deploy-to-prod.yaml deployment/terraform/single-project/ , cicd/ , shared/ agents-cli deploy + internal/workflow/telemetry.go observability.go $\rightarrow$ Cloud Trace + Cloud Logging $\rightarrow$ telemetry.tf BigQuery Sink agents-cli scaffold enhance In Phase 1, we introduced the Prototype-First philosophy of agents-cli : keep the project lightweight first, and then execute agents-cli scaffold enhance to layer on production infrastructure when ready for the cloud. We execute the following directly in the project root directory: uvx google-agents-cli@1.10.0 scaffold enhance . \ --cicd-runner github actions \ --region asia-east1 \ --yes agents-cli automatically detected our agents-cli-manifest.yaml template: adk go , deployment target: agent runtime and, without breaking any existing app/ , internal/ , eval/ , or sessions/ code, added three major modules to the project: deployment/terraform/cicd/ deployment/terraform/shared/telemetry.tf and storage.tf .github/workflows/ and e2e/ Next, we perform customized enhancements for the specific requirements of LINE AI Card Assistant LINE credentials, Cloud SQL PostgreSQL permissions, Taiwan asia-east1 region Let's open deployment/terraform/ and see how Terraform declaratively builds the entire Google Cloud Agent Platform: service.tf In deployment/terraform/single-project/service.tf https://github.com/kkdai/line-card-assistant/blob/main/deployment/terraform/single-project/service.tf , we use the google vertex ai reasoning engine resource from the google-beta provider to package the Go source code and declare it as a managed Agent Runtime instance. We also set lifecycle { ignore changes = spec 0 .source code spec } so that daily code updates are handled by the CI/CD's agents-cli deploy for fast rolling deployment, while infrastructure settings IAM, Buckets, environment variables are unified under Terraform control: resource "google vertex ai reasoning engine" "app" { provider = google-beta display name = var.project name project = var.project id region = var.region labels = { "agent-framework" = var.agent framework } spec { identity type = "AGENT IDENTITY" agent framework = var.agent framework source code spec { inline source { source archive = data.archive file.source.output path } developer connect source {} } container spec { image uri = "us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-15:latest" } } lifecycle { ignore changes = spec 0 .source code spec, spec 0 .container spec, } } iam.tf & variables.tf To prevent LINE CHANNEL SECRET , LINE CHANNEL ACCESS TOKEN , and DATABASE URL from appearing in plain text in configuration files, we declare Google Cloud Secret Manager secrets in deployment/terraform/single-project/iam.tf https://github.com/kkdai/line-card-assistant/blob/main/deployment/terraform/single-project/iam.tf and variables.tf https://github.com/kkdai/line-card-assistant/blob/main/deployment/terraform/single-project/variables.tf . We grant roles/secretmanager.secretAccessor and roles/cloudsql.client to both the Vertex AI Service Agent gcp-sa-aiplatform-re and the Agent Identity Principal principal://${google vertex ai reasoning engine.app.spec 0 .effective identity} : Secret Manager secrets for LINE Messaging API credentials and Cloud SQL DSN locals { line card secrets = "line-channel-secret", "line-channel-access-token", "line-card-database-url", } resource "google secret manager secret" "app secrets" { for each = toset local.line card secrets project = var.project id secret id = each.value replication { auto {} } depends on = resource.google project service.services } This way, during deployment, you only need to pass --secrets="LINE CHANNEL SECRET=line-channel-secret,LINE CHANNEL ACCESS TOKEN=line-channel-access-token" to agents-cli deploy , and the Agent Runtime will securely mount the secrets from Secret Manager when starting the container In deployment/terraform/shared/telemetry.tf https://github.com/kkdai/line-card-assistant/blob/main/deployment/terraform/shared/telemetry.tf , Terraform automatically creates a Cloud Logging Bucket with a 365-day retention period, enables enable analytics = true , and directly maps telemetry logs to a BigQuery Dataset telemetry logs dataset via google logging linked dataset : resource "google logging project bucket config" "telemetry logs bucket" { project = each.value location = "global" bucket id = "${var.project name}-telemetry" retention days = 365 enable analytics = true } resource "google logging linked dataset" "telemetry logs dataset" { bucket = google logging project bucket config.telemetry logs bucket each.key .bucket id link id = replace "${var.project name} telemetry", "-", " " project = each.value location = "global" } This means all structured telemetry data output by OpenTelemetry can be analyzed directly in BigQuery using standard SQL without writing additional ETL scripts In the observability.go generated in Phase 1, the system was already configured with an OTLP HTTP Exporter pointing to telemetry.googleapis.com and Google ADC automatic authentication. However, by default, we only see top-level traces for the entire HTTP request or LLM call, but not the execution time and routing decisions of each FunctionNode within the ADK 2.0 Graph Workflow . To present the processing of each business card as a complete waterfall span tree on Google Cloud Trace , we added internal/workflow/telemetry.go https://github.com/kkdai/line-card-assistant/blob/main/internal/workflow/telemetry.go : package workflow import "context" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/codes" "go.opentelemetry.io/otel/trace" const workflowTracerName = "github.com/kkdai/line-card-assistant/internal/workflow" const AttrWorkflowNode = attribute.Key "adk.workflow.node" AttrWorkflowRoute = attribute.Key "adk.workflow.route" AttrInvocationID = attribute.Key "adk.invocation.id" AttrSessionID = attribute.Key "adk.session.id" AttrUserID = attribute.Key "adk.user.id" AttrCardID = attribute.Key "adk.card.id" AttrCardCompany = attribute.Key "adk.card.company" AttrCardDuplicate = attribute.Key "adk.card.is duplicate" AttrHITLInterrupt = attribute.Key "adk.hitl.interrupt id" AttrHITLAction = attribute.Key "adk.hitl.action" AttrSearchMatches = attribute.Key "adk.search.matches" AttrSearchKeywords = attribute.Key "adk.search.keywords" func startNodeSpan ctx context.Context, nodeName string, attrs ...attribute.KeyValue context.Context, trace.Span { baseAttrs := make attribute.KeyValue, 0, len attrs +1 baseAttrs = append baseAttrs, AttrWorkflowNode.String nodeName baseAttrs = append baseAttrs, attrs... return otel.Tracer workflowTracerName .Start ctx, "workflow.node."+nodeName, trace.WithAttributes baseAttrs... , } Next, we inject startNodeSpan and endNodeSpan into all nodes in internal/workflow/nodes ingest.go and internal/workflow/nodes search.go . For example, in HybridSearchNode : func newHybridSearchNode cardStore store.CardStore workflow.FunctionNode { return workflow.NewFunctionNode NodeHybridSearch, func ctx agent.Context, any ev session.Event, err error { spanCtx, span := startNodeSpan ctx, NodeHybridSearch, AttrInvocationID.String ctx.InvocationID defer func { routeAttr := "" if ev = nil && len ev.Routes 0 { routeAttr = ev.Routes 0 } endNodeSpan span, err, AttrWorkflowRoute.String routeAttr } // ... execute cardStore.HybridSearch spanCtx, filter ... span.SetAttributes AttrSearchMatches.Int len cards // ... }, workflow.NodeConfig{}, } When a user uploads a business card or performs a hybrid search, Cloud Trace automatically draws a hierarchical Span tree like the one below, including the time spent in the three child nodes of the parallel Fan-Out and the business attributes of each node: flowchart TD Root "invoke workflow line card assistant