# [Agent Platform in Action (6)] Towards Production-Grade AI Agent Ops: Building Full-Stack Observable LINE Agents with agents-cli

> Source: <https://dev.to/gde/agent-platform-in-action-6-towards-production-grade-ai-agent-ops-building-full-stack-38aa>
> Published: 2026-10-25 10:05:18+00:00

Welcome to the sixth (and final) article in the "**Agent Platform in Action**" series! Looking back at the past five articles, we have taken **[LINE AI Card Assistant (`kkdai/line-card-assistant`)](https://github.com/kkdai/line-card-assistant)** through the four major stages of evolution from a local prototype to an enterprise-grade AI Agent architecture:

`google.golang.org/adk/v2/workflow` covering Vision OCR, `AddFanOut` / `AddFanIn` parallel background research, and `NewRequestInputEvent` HITL interaction loops.`agents-cli` (`adk_go`) dual-channel architecture, supporting LINE Webhook, ADK WebUI Playground, and A2A protocols.`artifact/gcsartifact` and `session/vertexai` to solve cross-instance HITL recovery and image storage issues in stateless containers.`PostgresStore`) and building a stateless Cloud Run LINE Webhook Gateway (` internal/gateway`).` agents-cli eval` to establish an automated quality flywheel covering 6 golden cases and 4 major evaluation metrics, all scoring `1.0000`.
In this final chapter today, we will complete the last piece of the puzzle for **[Roadmap #9](https://github.com/kkdai/line-card-assistant/issues/9)** — **Phase 5 ([Issue #8](https://github.com/kkdai/line-card-assistant/issues/8)): Official Agent Runtime Deployment, Terraform IaC, GitHub Actions CI/CD, and Full-Stack Cloud Trace Observability**!

`gcloud` Commands" to "Declarative AgentOps"
In many AI projects, even if the code and evaluation tests are written, production environments often become operational nightmares due to the following four issues:

`go test` can catch syntax and logic errors, but if someone accidentally breaks a Graph Node's `state_delta` or Prompt, how can we automatically intercept it before the PR is merged into `main`?` VisionOCRNode` or in the parallel Fan-Out's `CompanyEnrichNode` (Google Search Grounding)? We need every node of the ADK 2.0 Graph Workflow to expand into a clear nested Span tree in This is exactly what the **Phase 5 Full-Stack Architecture Diagram** at the beginning of the article depicts through four core pillars:

`.github/workflows/pr_checks.yaml`, `staging.yaml`, `deploy-to-prod.yaml`
`deployment/terraform/single-project/`, `cicd/`, `shared/`
`agents-cli deploy` + `internal/workflow/telemetry.go`
`observability.go` $\rightarrow$ Cloud Trace + Cloud Logging $\rightarrow$ `telemetry.tf` BigQuery Sink`agents-cli scaffold enhance`
In Phase 1, we introduced the Prototype-First philosophy of `agents-cli`: keep the project lightweight first, and then execute `agents-cli scaffold enhance` to layer on production infrastructure when ready for the cloud.

We execute the following directly in the project root directory:

```
uvx google-agents-cli@1.10.0 scaffold enhance . \
  --cicd-runner github_actions \
  --region asia-east1 \
  --yes
```

`agents-cli` automatically detected our `agents-cli-manifest.yaml` (`template: adk_go`, `deployment_target: agent_runtime`) and, without breaking any existing `app/`, `internal/`, `eval/`, or `sessions/` code, added three major modules to the project:

`deployment/terraform/cicd/`` deployment/terraform/shared/telemetry.tf` and `storage.tf``.github/workflows/` and `e2e/`
Next, we perform customized enhancements for the specific requirements of **LINE AI Card Assistant** (LINE credentials, Cloud SQL PostgreSQL permissions, Taiwan `asia-east1` region)!

Let's open `deployment/terraform/` and see how Terraform declaratively builds the entire Google Cloud Agent Platform:

`service.tf`)
In [`deployment/terraform/single-project/service.tf`](https://github.com/kkdai/line-card-assistant/blob/main/deployment/terraform/single-project/service.tf), we use the `google_vertex_ai_reasoning_engine` resource from the `google-beta` provider to package the Go source code and declare it as a managed Agent Runtime instance. We also set `lifecycle { ignore_changes = [spec[0].source_code_spec] }` so that daily code updates are handled by the CI/CD's `agents-cli deploy` for fast rolling deployment, while infrastructure settings (IAM, Buckets, environment variables) are unified under Terraform control:

```
resource "google_vertex_ai_reasoning_engine" "app" {
  provider = google-beta
  display_name = var.project_name
  project = var.project_id
  region = var.region

  labels = {
    "agent-framework" = var.agent_framework
  }

  spec {
    identity_type = "AGENT_IDENTITY"
    agent_framework = var.agent_framework

    source_code_spec {
      inline_source {
        source_archive = data.archive_file.source.output_path
      }
      developer_connect_source {}
    }

    container_spec {
      image_uri = "us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-15:latest"
    }
  }

  lifecycle {
    ignore_changes = [
      spec[0].source_code_spec,
      spec[0].container_spec,
    ]
  }
}
```

`iam.tf` & `variables.tf`)
To prevent `LINE_CHANNEL_SECRET`, `LINE_CHANNEL_ACCESS_TOKEN`, and `DATABASE_URL` from appearing in plain text in configuration files, we declare **Google Cloud Secret Manager** secrets in [`deployment/terraform/single-project/iam.tf`](https://github.com/kkdai/line-card-assistant/blob/main/deployment/terraform/single-project/iam.tf) and [`variables.tf`](https://github.com/kkdai/line-card-assistant/blob/main/deployment/terraform/single-project/variables.tf). We grant `roles/secretmanager.secretAccessor` and `roles/cloudsql.client` to both the **Vertex AI Service Agent (`gcp-sa-aiplatform-re`)** and the **Agent Identity Principal (` principal://${google_vertex_ai_reasoning_engine.app.spec[0].effective_identity}`)**:

```
# Secret Manager secrets for LINE Messaging API credentials and Cloud SQL DSN
locals {
  line_card_secrets = [
    "line-channel-secret",
    "line-channel-access-token",
    "line-card-database-url",
  ]
}

resource "google_secret_manager_secret" "app_secrets" {
  for_each = toset(local.line_card_secrets)
  project = var.project_id
  secret_id = each.value

  replication {
    auto {}
  }

  depends_on = [resource.google_project_service.services]
}
```

This way, during deployment, you only need to pass `--secrets="LINE_CHANNEL_SECRET=line-channel-secret,LINE_CHANNEL_ACCESS_TOKEN=line-channel-access-token"` to `agents-cli deploy`, and the Agent Runtime will securely mount the secrets from Secret Manager when starting the container!

In [`deployment/terraform/shared/telemetry.tf`](https://github.com/kkdai/line-card-assistant/blob/main/deployment/terraform/shared/telemetry.tf), Terraform automatically creates a Cloud Logging Bucket with a 365-day retention period, enables `enable_analytics = true`, and directly maps telemetry logs to a **BigQuery Dataset (` telemetry_logs_dataset`)** via `google_logging_linked_dataset`:

```
resource "google_logging_project_bucket_config" "telemetry_logs_bucket" {
  project = each.value
  location = "global"
  bucket_id = "${var.project_name}-telemetry"
  retention_days = 365
  enable_analytics = true
}

resource "google_logging_linked_dataset" "telemetry_logs_dataset" {
  bucket = google_logging_project_bucket_config.telemetry_logs_bucket[each.key].bucket_id
  link_id = replace("${var.project_name}_telemetry", "-", "_")
  project = each.value
  location = "global"
}
```

This means all structured telemetry data output by OpenTelemetry can be analyzed directly in BigQuery using standard SQL without writing additional ETL scripts!

In the `observability.go` generated in Phase 1, the system was already configured with an OTLP HTTP Exporter pointing to `telemetry.googleapis.com` and Google ADC automatic authentication. However, by default, we only see top-level traces for the entire HTTP request or LLM call, but not the **execution time and routing decisions of each `FunctionNode` within the ADK 2.0 Graph Workflow**.

To present the processing of each business card as a complete waterfall span tree on **Google Cloud Trace**, we added [`internal/workflow/telemetry.go`](https://github.com/kkdai/line-card-assistant/blob/main/internal/workflow/telemetry.go):

```
package workflow

import (
    "context"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/codes"
    "go.opentelemetry.io/otel/trace"
)

const workflowTracerName = "github.com/kkdai/line-card-assistant/internal/workflow"

const (
    AttrWorkflowNode = attribute.Key("adk.workflow.node")
    AttrWorkflowRoute = attribute.Key("adk.workflow.route")
    AttrInvocationID = attribute.Key("adk.invocation.id")
    AttrSessionID = attribute.Key("adk.session.id")
    AttrUserID = attribute.Key("adk.user.id")
    AttrCardID = attribute.Key("adk.card.id")
    AttrCardCompany = attribute.Key("adk.card.company")
    AttrCardDuplicate = attribute.Key("adk.card.is_duplicate")
    AttrHITLInterrupt = attribute.Key("adk.hitl.interrupt_id")
    AttrHITLAction = attribute.Key("adk.hitl.action")
    AttrSearchMatches = attribute.Key("adk.search.matches")
    AttrSearchKeywords = attribute.Key("adk.search.keywords")
)

func startNodeSpan(ctx context.Context, nodeName string, attrs ...attribute.KeyValue) (context.Context, trace.Span) {
    baseAttrs := make([]attribute.KeyValue, 0, len(attrs)+1)
    baseAttrs = append(baseAttrs, AttrWorkflowNode.String(nodeName))
    baseAttrs = append(baseAttrs, attrs...)
    return otel.Tracer(workflowTracerName).Start(
        ctx,
        "workflow.node."+nodeName,
        trace.WithAttributes(baseAttrs...),
    )
}
```

Next, we inject `startNodeSpan` and `endNodeSpan` into all nodes in `internal/workflow/nodes_ingest.go` and `internal/workflow/nodes_search.go`. For example, in `HybridSearchNode`:

```
func newHybridSearchNode(cardStore store.CardStore) *workflow.FunctionNode {
    return workflow.NewFunctionNode(
        NodeHybridSearch,
        func(ctx agent.Context, _ any) (ev *session.Event, err error) {
            spanCtx, span := startNodeSpan(ctx, NodeHybridSearch, AttrInvocationID.String(ctx.InvocationID()))
            defer func() {
                routeAttr := ""
                if ev != nil && len(ev.Routes) > 0 {
                    routeAttr = ev.Routes[0]
                }
                endNodeSpan(span, err, AttrWorkflowRoute.String(routeAttr))
            }()
            // ... execute cardStore.HybridSearch(spanCtx, filter) ...
            span.SetAttributes(AttrSearchMatches.Int(len(cards)))
            // ...
        },
        workflow.NodeConfig{},
    )
}
```

When a user uploads a business card or performs a hybrid search, Cloud Trace automatically draws a hierarchical Span tree like the one below, including the time spent in the three child nodes of the parallel Fan-Out and the business attributes of each node:

```
flowchart TD
    Root["invoke_workflow (line_card_assistant)<br/>trace_id: 4bf92f3577b34da6a3ce929d0e0e4736"] --> R["workflow.node.RouterNode<br/>adk.workflow.route = 'ingest' (1.2ms)"]
    R --> V["workflow.node.VisionOCRNode<br/>adk.card.company = 'Taiwan AI Labs' (640ms)"]
    V --> C1["workflow.node.CleanAndDedupNode<br/>adk.card.is_duplicate = false (8.5ms)"]
    V --> C2["workflow.node.CompanyEnrichNode<br/>GoogleSearch Grounding (780ms)"]
    V --> C3["workflow.node.ContextMatchNode<br/>inferMetContext (310ms)"]
    C1 & C2 & C3 --> B["workflow.node.BuildPreviewNode<br/>Flex Bubble JSON (2.1ms)"]
    B --> H["workflow.node.HITLConfirmNode<br/>adk.hitl.interrupt_id = 'hitl_confirm_1'<br/>adk.hitl.action = 'pause' (0.8ms)"]
```

We also wrote automated unit tests in [`internal/workflow/telemetry_test.go`](https://github.com/kkdai/line-card-assistant/blob/main/internal/workflow/telemetry_test.go) using the OpenTelemetry SDK's `tracetest.NewInMemoryExporter()`, ensuring that "card recognition + HITL modification loop + confirmation save" and "hybrid search with 0 results automatically degrading to fuzzy search" 100% generate the expected Span nodes and attribute tags!

With Terraform IaC and OpenTelemetry tracing in place, we officially embedded the **`agents-cli eval` quality flywheel** created in Phase 4 into the GitHub Actions CI/CD pipeline, forming a two-stage automated delivery:

```
flowchart LR
    subgraph PR["1. Pull Request Quality Gate (pr_checks.yaml)"]
        P1["go test -race -v ./...<br/>(Unit + OTel Span + Trajectory Tests)"] --> P2["go test -tags=e2e<br/>(A2A + /app-info E2E Tests)"]
        P2 --> P3["agents-cli eval run<br/>(4 major metrics must = 1.0000)"]
    end

    subgraph CD["2. Staging & Production CD (staging.yaml -> deploy-to-prod.yaml)"]
        S1["WIF Keyless Auth<br/>(google-github-actions/auth)"] --> S2["agents-cli deploy<br/>Deploy to Staging Agent Runtime"]
        S2 --> S3["Vegeta Load Test<br/>30s pressure test + report upload to GCS"]
        S3 --> S4["GitHub Environment<br/>Production Manual Approval Gate"]
        S4 --> S5["agents-cli deploy<br/>Deploy to Production Agent Runtime"]
    end

    PR -->|Merge to main| CD
```

In [`.github/workflows/pr_checks.yaml`](https://github.com/kkdai/line-card-assistant/blob/main/.github/workflows/pr_checks.yaml), whenever a PR modifies `app/**`, `internal/**`, `cmd/**`, `sessions/**`, or `eval/**`, GitHub Actions will execute in order:

`go test -race -v ./...`: Verify all Go unit tests, cross-instance HITL recovery, OpenTelemetry Node Spans, and trajectory tests.`go test -v -tags=e2e ./e2e/integration/...`: Start a real ADK Universal Launcher subprocess to verify `/.well-known/agent-card.json`, `/a2a/invoke`, and `/apps/app/app-info` endpoints.`agents-cli eval run` Quality Gatekeeper`workflow_trajectory_accuracy`, `ocr_and_enrichment_quality`, `duplicate_and_hitl_precision`, `search_and_email_grounding`) must maintain a perfect score!`.github/workflows/staging.yaml`)
After the PR is merged into the `main` branch, [`.github/workflows/staging.yaml`](https://github.com/kkdai/line-card-assistant/blob/main/.github/workflows/staging.yaml) will:

`uvx google-agents-cli@1.10.0 deploy` to deploy the latest commit to the Staging Vertex AI Agent Runtime, while mounting `LINE_CHANNEL_SECRET` and `LINE_CHANNEL_ACCESS_TOKEN` from Secret Manager and tagging `OTEL_RESOURCE_ATTRIBUTES=service.version=$`.` e2e/load_test/load_test.go``tsenart/vegeta/v12`) to perform a 30-second continuous pressure test on the Staging Agent Runtime, checking HTTP 200 success rate, P50/P95/P99 latency, and `429 Too Many Requests` ratio, and automatically archiving the results to `gs://${LOGS_BUCKET_NAME_STAGING}/load-test-results/`.`.github/workflows/deploy-to-prod.yaml`` production` environment, it is officially pushed to Production!`agents-cli deploy --dry-run` and Full E2E Tests All Passed!
Before submitting the code as a PR, we fully executed three major verifications locally:

``` bash
$ uvx google-agents-cli@1.10.0 deploy \
    --project line-vertex \
    --region asia-east1 \
    --dry-run \
    --no-confirm-project
```

Preflight output results:

`aiplatform.googleapis.com` API`line-card-assistant` (`asia-east1`) ✅

``` bash
$ go test -v -tags=e2e ./...
```

Includes the newly added `TestWorkflowOpenTelemetrySpans_IngestAndHITL`, `TestWorkflowOpenTelemetrySpans_SearchFallback`, and all tests under `e2e/integration` such as `TestA2AAgentCard`, `TestA2AMessageSend`, `TestA2AErrorHandling`, and `TestAppInfo` — all **PASS**!

``` bash
$ DRY_RUN=true make eval
```

The 6 golden cases across the 4 custom metrics (`workflow_trajectory_accuracy`, `ocr_and_enrichment_quality`, `duplicate_and_hitl_precision`, `search_and_email_grounding`) still maintain **` 1.0000` (100% Pass)**!

With the completion of **Phase 5 ([Issue #8](https://github.com/kkdai/line-card-assistant/issues/8))**, the five stages planned in **[Roadmap Issue #9](https://github.com/kkdai/line-card-assistant/issues/9)** have been **100% achieved and closed**! Let's conclude with a summary table reviewing this journey from 0 to 1 with **Google ADK 2.0 Go + Google Cloud Agent Platform**:

| Stage | Corresponding Issue / PR | Core Technical Breakthroughs | Corresponding Blog Post | 
|---|---|---|---|
| **Prototype** | Initial Commit | Go `adk/v2/workflow` 12-node Graph Workflow, parallel`AddFanOut` /`AddFanIn` ,`NewRequestInputEvent` HITL loop | [In Action (1)](https://dev.to/2026/10/16/adk-graph-workflow-go/) | 
| **Phase 1** | [Issue #4](https://github.com/kkdai/line-card-assistant/issues/4) /[PR #10](https://github.com/kkdai/line-card-assistant/pull/10) | `agents-cli` (`adk_go` ) standardization,`app.NewRootAgent` dual-channel factory, ADK WebUI Playground, and A2A protocol support | [In Action (2)](https://dev.to/2026/10/17/adk-graph-workflow-go-2/) | 
| **Phase 2 (Part 1)** | [Issue #5](https://github.com/kkdai/line-card-assistant/issues/5) /[PR #11](https://github.com/kkdai/line-card-assistant/pull/11) | `artifact/gcsartifact` zero-state image bloat,`session/vertexai` cross-instance stateless HITL pause and resume | [In Action (3)](https://dev.to/2026/10/20/adk-graph-workflow-go-3/) | 
| **Phase 2 (Part 2) & Phase 3** | [Issue #5](https://github.com/kkdai/line-card-assistant/issues/5) ,[#6](https://github.com/kkdai/line-card-assistant/issues/6) /[PR #12](https://github.com/kkdai/line-card-assistant/pull/12) | Cloud SQL PostgreSQL ( `PostgresStore` ) card persistence, Cloud Run LINE Webhook Gateway (`internal/gateway` ) and Agent Runtime dual-layer decoupling | [In Action (4)](https://dev.to/2026/10/21/adk-graph-workflow-go-4/) | 
| **Phase 4** | [Issue #7](https://github.com/kkdai/line-card-assistant/issues/7) /[PR #13](https://github.com/kkdai/line-card-assistant/pull/13) | `agents-cli eval` quality flywheel,`/app-info` reflection introspection, 6 golden cases, and`setNodeTrace` trajectory repair (all 4 metrics at`1.0000` ) | [In Action (5)](https://dev.to/2026/10/22/adk-graph-workflow-go-5/) | 
| **Phase 5** | [Issue #8](https://github.com/kkdai/line-card-assistant/issues/8) /[PR #14](https://github.com/kkdai/line-card-assistant/pull/14) | `agents-cli deploy` , Terraform IaC (`deployment/terraform/` ), GitHub Actions CI/CD + WIF + Vegeta pressure testing, OpenTelemetry node-level Cloud Trace & BigQuery observation | **This article (In Action 6)** | 

The complete open-source code, Terraform modules, and evaluation datasets have been synchronized to the GitHub Repo: **[`kkdai/line-card-assistant`](https://github.com/kkdai/line-card-assistant)**. If you are also building enterprise-grade AI Agents with Go and Google ADK 2.0, I hope this series serves as your best practical blueprint for moving from local development all the way to the Google Cloud Agent Platform production environment!