If you have ever wanted to ship an AI-powered application without managing GPUs, model servers, or scaling infrastructure yourself, this guide is for you.
Managed inference simply means letting a cloud provider run the AI model for you: you send a request, the platform handles the compute, and you get a response back. On Google Cloud, the cleanest way to do this today is to pair the Gemini Enterprise Agent Platform (formerly Vertex AI) with Google Cloud Run, dividing responsibilities between the two services. The Agent Platform serves as the orchestration and intelligence engine, while Cloud Run hosts your custom application logic, front-end UIs, or Model Context Protocol (MCP) servers.
By the end of this article, you will be able to:
New to the underlying concept? Start with Google Cloud's primer: What is AI inference?
To follow along hands-on, you will need:
gcloud
CLI installed and authenticatedpip install google-adk
)You can also read this purely as an architecture walkthrough; every step is explained, not just shown.
This pattern splits your system into independent, auto-scaling tiers:
[ Client / Web UI ] ──> [ Cloud Run Service ] (App Logic / Tool Front End)
│
▼
[ Gemini Enterprise Agent Platform — Agent Runtime ]
(Orchestration, Intent Analysis, Memory)
│
▼
[ Managed Inference / Model Garden ]
(Gemini 3.x Pro / Flash models)
Why split it this way? Each tier scales independently and fails independently. Your web front end can handle a traffic spike without touching the model layer, and you can swap models without redeploying your application code. It also creates a clean security boundary, clients only ever talk to Cloud Run, never directly to the model.
Here is what each layer actually does:
Use the open-source Agent Development Kit (ADK) to define your agent's behavior in code and bind it to a model. The key idea to understand: tools are plain Python functions. The ADK reads each function's docstring to decide when and how to call it; so a clear docstring is not documentation nicety, it is part of your agent's logic.
from google.adk.agents import Agent
def call_internal_business_system(query: str) -> str:
"""Invokes secure business workflows deployed on Cloud Run."""
return "Data retrieved from secure internal backend."
root_agent = Agent(
name="enterprise_inference_agent",
model="gemini-3.5-flash", # Or another current model from Model Garden
instruction="You are a data processing assistant using managed inference.",
tools=[call_internal_business_system],
)
Breaking down the four fields:
name
— an identifier for your agent, used in logs and traces.model
— which Gemini model handles the reasoning. Flash models are faster and cheaper; Pro models handle more complex reasoning.instruction
— the agent's system prompt, shaping its behavior on every request.tools
— the Python functions the model is allowed to call. When a user request matches a tool's docstring, the model invokes it.
Note:Gemini 1.0 and 1.5 models (includinggemini-1.5-pro
) have been retired and now return errors. Always target a currently supported model, such asgemini-3.5-flash
,gemini-3.6-flash
, or a Gemini 3.x Pro release from Model Garden.
When deploying your orchestration backend or front-end dashboard, the tooling can package and push the container for you. Two small steps get you there.
In Google Cloud, services do not trust each other by default, your Cloud Run instance needs explicit permission to invoke Agent Platform endpoints. This command grants its service account that permission:
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
--member="serviceAccount:YOUR_RUN_SA@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
--role="roles/aiplatform.user"
In plain terms: "let this Cloud Run service call the AI platform." This is the step people most often forget; if your deployed service returns permission errors, come back here first.
The ADK ships with a one-command deployment path. Under the hood, it does three things: builds your container image, pushes it to Artifact Registry, and creates (or updates) the Cloud Run service.
adk deploy cloud_run \
--project="YOUR_PROJECT_ID" \
--region="us-central1" \
--service_name="agent-inference-backend" \
path/to/your/agent
Alternatively, the Agents CLI (agents-cli
) can scaffold the deployment configuration for a Cloud Run target. For example, agents-cli scaffold enhance --deployment-target cloud_run
and works from inside your preferred AI coding tool. Either route wires up your environment variables, including model targets and the public service URL.
Once the plumbing is in place, there are two primary ways to trigger managed inference. Choosing correctly comes down to one question: does a human need the answer right now?
Batch jobs are typically much cheaper per request, so a good rule of thumb is: default to batch, and reserve online inference for genuinely interactive experiences.
A demo can skip this section. Production cannot.
roles/aiplatform.user
to your Cloud Run service account, or nothing else will work.Try the smallest possible version: define a one-tool agent with the ADK, run adk deploy cloud_run
, and send it a request. Once that works, everything else in this article is an incremental addition.
Have you tried pairing the Agent Platform with Cloud Run, or are you still on a self-managed inference setup? I would love to hear what your architecture looks like in the comments.