A reference architecture for running the data plane of an AI assistant in the customer's own cloud account, with the provider keeping only a control plane
This document describes the second of three deployment patterns for an AI assistant platform. It is written for architects and platform engineers whose customer, or whose own organisation, has said: "our documents, our search index and our encryption key stay in our cloud account, under a key we can revoke without asking anyone."
The structure is the same as Pattern 1: a one-picture architecture, the request flow, the components, use cases, design considerations, what crosses the boundary, and when to choose it. The design is independent of any product; the technologies named can be replaced.
Terms used throughout. The provider is whoever builds and operates the platform software: a vendor, or an internal platform team. The customer is the organisation whose users and data the platform serves. In this pattern the customer also operates most of the system.
Prerequisites. On the customer's side: a cloud account, a Kubernetes service, managed Postgres and object storage, a key management service, and access to a cloud model provider in the chosen region. On the provider's side: a build pipeline that can sign artifacts and an OCI registry. The customer needs a team that can run Kubernetes and its supporting services in production.
| Pattern | Data lives in | Operated by | Crosses the boundary |
|---|---|---|---|
| 1. Shared service | the provider's account, inside a residency boundary | the provider | every user's traffic, inside the boundary |
| 2. Control/data split (this document) | the customer's cloud account | the customer runs the data plane; the provider runs a control plane | signed software in, optional health metadata out |
| 3. Isolated site | the customer's building | the customer, everything | two signed files in, one optional file out |
The three questions still decide: where does the data live, who operates it, what crosses the boundary. For this pattern the third question is the one to get right, and the next section shows why.
The obvious way to meet the customer's rule is to split by component: move the pieces that store customer content into the customer's account and keep the rest.
The customer runs retrieval over their own vector store, a connector module that reaches their systems from inside their network, a cache, their documents bucket, Vault and a KMS key they can revoke. The provider reaches these over one private link with mutual TLS. Software is pull-only. Stored data never leaves the customer's account.
Now trace one chat message and mark every arrow where customer content crosses an account boundary with ◆.
| # | Crossing | Why it exists |
|---|---|---|
| ① | The user's question travels through the provider's edge, gateway and Chat Gateway | The entry point is the provider's |
| ② | Retrieved passages come back over the link | The Orchestrator is the provider's |
| ③ | Results from the customer's systems come back over the link | Same reason |
| ④ | The full prompt goes to the model in the provider's account | The model is the provider's |
| ⑤ | The streamed answer sits in the provider's Redis for about five minutes | The Chat Gateway is the provider's |
Five crossings per question. None of them is a breach: each is encrypted, disclosed, inside the residency boundary and gone within minutes. But the design cannot support the sentence the customer wants: "your data never reaches our account." Stored data does not; data in flight does, five times.
Two rules follow, and they apply to any architecture:
The fix is to split by plane, not by component.
Three things change. There is no connectivity link, because there is nothing for the provider to reach, and the provider has no inbound path at all. The model call stays home: the Orchestrator calls the customer's own cloud model providers in the customer's account, under the customer's contract. There is nothing to crypto-shred at exit, because the provider holds no configuration, sessions or buffer. The ◆ marker appears nowhere.
| Hop | Path | Protocol | Who opens it | Authentication |
|---|---|---|---|---|
| ① | Client → customer edge | HTTPS/443 | client | opaque session cookie or token, resolved at the gateway |
| ② | Edge → API gateway | HTTPS/443 | load balancer | n/a (network hop) |
| ③ | Gateway → identity broker | HTTPS/443 | gateway | login and signing keys only; not on the chat path |
| ④ | Gateway → Chat Gateway | HTTP/8080 over mTLS | gateway | mesh identity |
| ⑤ | Chat Gateway → Orchestrator | gRPC over mTLS | Chat Gateway | mesh identity |
| ⑥ | Orchestrator → tools and actions | gRPC / HTTPS over mTLS | Orchestrator | mTLS; the sandbox holds no credentials; per-user tokens from the vault |
| ⑦ | Orchestrator → retrieval → pgvector | gRPC; then Postgres/5432 | Orchestrator, then retrieval | mTLS; database role from the customer's Vault |
| ⑧ | Orchestrator → model router → customer's model providers | HTTPS/443, private endpoint | Orchestrator | the customer's cloud identity |
| ⑨ | Orchestrator → connector hub → customer systems | gRPC; then HTTPS | Orchestrator, then connector | mTLS; connector credentials from Vault |
| ⑩ | Streamed answer → Chat Gateway → user | SSE over HTTPS | already open | same session as ① |
Every hop stays inside the customer's boundary. The provider's column in the detailed topology has no live-request arrows at all.
| Component | Runs in | Operated by | Common choice |
|---|---|---|---|
| Customer edge: domain, certificate, WAF, load balancer | customer account | customer | cloud load balancer and WAF |
| API gateway, identity broker, identity provider | customer account | customer | Kong, Keycloak, the customer's IdP |
| Chat Gateway, Orchestrator, retrieval, connector hub, Redis, mesh | customer's Kubernetes cluster | customer | EKS, AKS or GKE; Istio or Linkerd |
| Tools and actions: sandbox, action gateway, tool connectors, per-user credential vault, metering | customer account | customer | gVisor or Firecracker, MCP servers |
| Postgres with pgvector, documents bucket, Vault, KMS key | customer account | customer | managed Postgres, object storage, Vault |
| Model router and model accounts | customer account, customer's contract | customer | Bedrock, Azure OpenAI, Vertex AI |
| Delivery: Argo CD, release gate, Kyverno, optional registry mirror | customer account | customer | Argo CD, cosign, Kyverno |
| Observability, egress proxy, audit, SIEM | customer account | customer | OpenTelemetry stack |
| Release pipeline, signing, OCI registry, signed licence | provider | provider | GitHub Actions, cosign, Harbor or a cloud registry |
| Health dashboard (opt-in heartbeat), support intake | provider | provider | Grafana |
The defining case. Documents, index and keys in the customer's account, encrypted under a key the customer can revoke, with no content in the provider's infrastructure. The customer can operate Kubernetes and its supporting services, or has a managed-service team that can.
Because the customer runs the data plane, the provider's software has to run on whichever cloud the customer uses. The design keeps every cloud-specific dependency behind an interface: the model router, the identity binding for the registry pull, and the key management.
An internal platform team is the provider; a business unit with its own cloud account and its own compliance regime is the customer. The pull-only delivery and the egress proxy log give the business unit's auditors proof without trusting the platform team.
Delivery without access. The provider never pushes anything. Its pipeline signs every chart and image, attaches an SBOM and a vulnerability-scan attestation, and publishes to an OCI registry. In the customer's cluster, Argo CD polls the registry over the customer's own outbound HTTPS and applies new versions; a release gate lets the customer review the SBOM and attestation first; Kyverno verifies the signature with a bundled, offline trust root before any pod starts. The customer holds every admin credential.
For cross-cloud registry access, two methods work: federate the cluster's workload identity into a pull-only role at the provider, or issue a per-customer pull-only robot token the customer keeps in their own Vault. Both are scoped to that customer's repositories and can only pull. Identity inside the boundary. The customer runs the identity broker, and the identity provider sits inside their boundary. The IdP token is verified once at login; after that the browser holds an opaque, hash-stored session. CLI tokens and API keys resolve the same way. No sign-in assertion crosses to the provider.
Tools and actions in the customer's account. The sandbox is self-hosted in the customer's account with no platform credentials and default-deny egress; a managed sandbox service would be a third party that sees content. The action gateway applies the customer's policy and approval steps. Tool connectors run against an allowlist the customer sets. Per-user tokens are sealed in a credential vault with a key id. Metering writes to a local store with limits per organisation and workspace.
The model router points at the customer's own providers. Several providers behind one interface, each in the customer's account, pinned to a region the customer chooses, reached over private endpoints, with invocation logging as the customer's own setting. The prompt never touches the provider's account. The nuance to state plainly: the model provider still sees prompts, under the customer's contract rather than the vendor's.
Data and authorization. Postgres with pgvector, the documents bucket, Redis and Vault live in the customer's account. Authorization is one decision point: policy compiled into every query, with row-level security beneath it. Encryption uses a customer-owned KMS key. Revoke it and everything stops and becomes unreadable, including to provider support.
High availability and disaster recovery are the customer's: multi-zone, a second region per their policy, immutable backups. The provider states no recovery targets here because there is no provider data plane to fail over. The provider commits only to registry availability, and an optional local registry mirror removes even that runtime dependency.
Telemetry is local by default: Fluent Bit, OpenTelemetry, Loki, Jaeger, VictoriaMetrics, a local Grafana. Cloud and cluster audit logs stay in the account. The egress proxy log is the customer's proof: it shows exactly two external names ever reached for the provider (the registry and, if opted in, the heartbeat), plus the customer's own model, identity and system endpoints.
Support works two ways, both optional: a redacted support bundle the customer reads before sending, or a break-glass session the customer opens, approves, time-boxes and records in their own audit log. The break-glass session is the one path that could ever show content to the provider, and the diagram says so.
The customer pays for the whole data plane: cluster, databases, model consumption under their own contract, a second region if they want one. Managed cloud services soften the operating cost but do not remove it. The provider's cost is small and fixed: a pipeline, a registry, a dashboard.
Model capacity, quota and regional placement are the customer's. The provider cannot add quota or fix an outage at the customer's model provider. Everything else scales as in Pattern 1, inside the customer's cluster.
| Crossing | Direction | What travels | Who opens it | Can the customer switch it off? |
|---|---|---|---|---|
| Software pull | customer → provider registry; software flows in | signed charts, images, licence | the customer's Argo CD and nodes | yes; the running version keeps working, and a mirror removes the runtime dependency |
| Heartbeat | customer → provider | versions, health, error counts; fixed schema, no payloads | the customer's agent | yes; opt-in |
| Support bundle | customer → provider, manual | redacted diagnostics the customer read first | the customer | yes; never required |
| Break-glass session | customer opens; provider joins | whatever the customer shows; nothing retained | the customer | yes; per incident, time-boxed, recorded |
| Model call | customer → their own model provider | prompt with retrieved passages | the customer's Orchestrator | it is the product; but it stays under the customer's account, contract and region |
Compare this with the first design's five content crossings per question. That comparison is the whole argument for splitting by plane.
Choose the control/data split when the customer's rule is "our documents, index and keys in our account, under a key we can revoke, and no content in your infrastructure", and they can operate Kubernetes and its supporting services. If they want the provider to run everything, Pattern 1. If not even a prompt may leave their building, Pattern 3.
The provider prepares; the customer deploys.
The full design in ten customer bands plus the provider's two control-plane bands. The hop numbers match the request flow table above. Open the figure at full size to read the component notes.
This is part 2 of a three-part series. The other parts are also linked in the series box at the top of this article.