{"slug": "building-an-enterprise-genai-platform-reference-architecture-security-and", "title": "Building an Enterprise GenAI Platform: Reference Architecture, Security and Governance", "summary": "A proposed reference architecture for on-premises enterprise generative AI lays out the soft and hard components organizations need before building a full GenAI ecosystem, including an MLOps/GenAI Ops module, a GenAI gateway, an MCP gateway, an agentic framework, a RAG system, and RBAC/ABAC and OIDC authentication. The architecture specifies hard prerequisites such as Red Hat Ceph S3, MinIO Enterprise or Huawei object storage, MongoDB, Milvus or Qdrant vector databases, Redis Enterprise, Cloudera Kafka or Red Hat AMQ Streams, Keycloak or Microsoft ADFS, and an NVIDIA GPU farm. The MLOps module is described as needing model registry, vulnerability scanning, red teaming, shadow AI detection, DLP, drift detection, auditing, and prompt registry and versioning, with MLflow Prompt Registry named for prompt governance.", "body_md": "This article aims to give insights on implementing end-to-end GenAI Architecture for an enterprise running completely on-prem, for newcomers. Also consider it as a proposed reference architecture for the base components of an Enterprise GenAI system.\n\nPre-Requisites\n\nThis section gives a quick view of which foundations -as prerequisites- are necessary before implementing a huge GenAI ecosystem.\n\nSoft Artifacts\n\nSoftware used to manage, operate, and govern the whole reference architecture. This software must be acquired externally or developed in-house.\n\nMLOPS/GenAI Ops Module\n\nModel Serving Module: On-prem and cloud frontier models\n\nGenAI Gateway\n\nRBAC/ABAC System: For authorization\n\nOIDC System: For Authentication\n\nExternal GenAI Model Downloading Pipeline\n\nCost Management (FinOps Methodology): for genai and for traditional ai\n\nTotal AI Security\n\nAn enterprise level AI Playground\n\nMCP Gateway\n\nAgentic Framework\n\nData Governance Platform\n\nRAG System: For indexing and searching the content\n\nData Lineage: Tracking the data lifecycle\n\nDocument Search: Secure document search\n\nHard Artifacts\n\nNamely the foundational components which “the soft artifacts” require before running, such as databases, storage, or hardware artifacts. Before implementing such a “reference architecture”, the hard artifacts below should already be operational and running in a high-availability configuration as a prerequisite!\n\nModern Object storage: Red Hat Ceph S3 Storage or MinIO Enterprise or Object Storage (Huawei)\n\nDocument Database: MongoDB\n\nBig Data or LakeHouse System: Enterprise grade: Cloudera Big Data Ecosystem (vendor lock-in risk) OSS frameworks: “Apache Hadoop / Spark (Vanilla ecosystem)” or “Enterprise S3 + (Apache Iceberg or Trino)”\n\nVector Database: Milvus PROS: GPU support CONS: enterprise support only on cloud deployment / self on-prem deployment Qdrant (enterprise support) PROS: suitable for indexing JSON payloads alongside vectors. CONS: limited gpu support commonly only cpu\n\nDistributed Cache Server: Redis Enterprise\n\nStream/Event Driven Messaging: Cloudera Kafka or Red Hat AMQ Streams.\n\nTimeseries Monitoring and Elastic Search: Grafana and Kibana\n\nGenAI Accelerator Hardware Farm: Modern NVIDIA GPU infrastructure\n\nMutual TLS Certification Tools (for auth/authz reasons when ai-agents come into the scene): SecTrail\n\nAuthentication OIDC Framework: Keycloak or Microsoft ADFS\n\nRELATIONS OF THE COMPONENTS — High Level Representation\n\nMLOps Platform\n\nExternal Model Downloading\n\nEnd users and playground environment\n\nAI Agents\n\nApplications (enterprise or 3rd party)\n\nRAG framework\n\nEnterprise Core Services\n\nEXPLAINING THE SOFT ARTIFACTS STEP BY STEP\n\nMLOPS/GenAI Ops Module\n\nModel Registry\n\nThe module must cover the AI model types below and be compatible with managing both inhouse developed and 3rd party downloaded models\n\nTraditional AI, Model Registry Module\n\nGenAI Model, Model Registry Module\n\nSecurity/Vulnerability Scanning\n\nThe capabilities of GenAI models are increasing, and the attack surface is increasing. So the module must scan most types of GenAI and traditional AI models for vulnerabilities.\n\nRed teaming capabilities for scanning the system-wide attack surface\n\nContinuous model scanning\n\nShadow AI usage\n\nDLP\n\nContinuous GenAI activity tracking and assessment\n\nDrift Detection\n\nSystem should track and save “online or batch traditional ml predictions” and “feature values at prediction time”. With a periodic job, all historical predictions and features should be monitored for drift.\n\nAuditing\n\nAll activities must be audited and ready to be queried, reported, or used for alerts.\n\nPrompt Registry and versioning\n\nA “prompt registry and versioning” is an essential tool in cases of:\n\nGuardrail usage\n\nStructuring and deploying the system prompts when using LLM inference time, for the different needs/usecases (sharing the correct prompt templates over the enterprise for the applications/users/agents) In order to govern prompts “MLflow Prompt Registry” would be a good choice but some enterprises decide to develop in-house. In “MLflow Prompt Registry”, versioning, aliasing, diffs, and SDK support are available.\n\nModel Serving Module\n\nThe module should manage one-click REST API serving infrastructure with an easy UI.\n\ntraditional ML Models: one click python fastapi deployments with an easy to use sdk\n\nGenAI Models: one click deployments with an easy to use sdk. The OpenShift AI product claims to do that task, but I have not validated it through hands-on experience. Example inference frameworks: 1. VLLM: Community’s widely used framework PROS: easy to deploy, development community supported by Red Hat CONS: Optimized to work on just a single model 2. NVIDIA Triton Inference Server: PROS: Multiple models such as LLM/Vision/Embedding can run at the same time CONS: Learning curve and expensive effort to implement it as a one-click deployment\n\nCloud frontier models serving: The module needs to interact with external frontier model providers. Assume there is no on-prem deployment for cloud models, since cloud providers have their own deployment user interfaces and strategies. IMPORTANT #1: Telemetry metrics which is defined in inference framework (for vllm generated tokens per second, page size, vllm queue size/time etc.) should be exported to Prometheus like system to trigger any “Horizontal Pod Autoscaler” IMPORTANT #2: The module should work with MIG (multi instance gpu) partitions for relatively small genai models such as voice and video models.\n\n“MLOPS/GenAI Ops” Model Registry Agent Registry Security/Vulnerability Scanning Drift Detection Auditing Prompt Registry and Versioning\n\n“Model Serving Module” module interactions; With a GPU infrastructure operated on a Kubernetes/OpenShift environment Selecting inference framework vllm / nvidia triton Selecting suitable gpu instance for genai model and workload\n\nAI Gateway Module\n\nThe module must let enterprise-wide “real users/applications/on behalf of agents/autonomous agents” consume GenAI services securely.\n\nPossible consumers\n\nReal users/applications\n\nOn behalf of agents: The agents execute tasks on behalf of the end-user in an assistant role\n\nAutonomous agents: The agents execute enterprise tasks without needing a human nearby such as coding/reviewing/document discovery/classifying agents\n\nThe gateway responsibilities\n\nAuthentication: Must be checked with OIDC level access-tokens, any hard code api-keys should not be allowed.\n\nAuthorization: Must be controlled through the “Enterprise RBAC Platform” or implemented as a separate “MLOps Platform Module” at the client and consumed-model granularity.\n\nTraceability: Must trace and store diagnostic/audit/telemetry data Audit logs: GenAI activity -sensitive ones should be carefuly obfuscated- (request/response) should be stored in a cold storage such as S3. Diagnostic logs: To track any unpredicted gw/genai behaviours or end-user complaint all activity logs must be stored. For example last 1 week activity (db) in a hot storage but older than 1 week activity in a cold storage (S3) Telemetry logs: Should be exported to an external monitoring system (Grafana) to track the gw and genai inference latency metrics/token usage metrics and to set the alarms based on it. Frankly every ativity on gateway side or inference side should be exported to telemetry system in order to monitor the system health!\n\nRate Limiting and Throttling: For fair usage of the available resources, rate limiting and throttling are a must.\n\nSemantic routing: Selecting cheaper but suitable models for tasks.\n\nContext caching: For quick responses and not overwhelming the system resources, based on a “similarity threshold of user prompt/context” and based on “the criticality of the task!” answer from the cache.\n\nTraffic-Aware routing: A good gateway also knows the load on upstream GenAI inference services and should route requests to a less-loaded upstream service.\n\nCost management (finops) telemetry outputs of token usage\n\nSuggestions:\n\nLiteLLM: With on-prem deployment and enterprise support highly coverage for above requirements. A bit like an industry standart tool\n\nTrue Foundry: With on-prem deployment and enterprise support highly coverage for above requirements also including mcp gateway\n\nGathering Those Components Together\n\nBelow chart shows how an AI Gateway works with the below functionalities;\n\nAuthentication\n\nAuthorization\n\nTraceability/Observability\n\nRate Limiting and Throttling\n\nSemantic routing\n\nContext caching\n\nTraffic-Aware routing\n\nCost management (finops) telemetry outputs of token usage\n\nOIDC System\n\nThe system must authenticate the “real users” / “on behalf of agent” / “autonomous agent” but each of authentication must be managed with a different path\n\nAuthentication Paths Of OIDC System\n\nReal users\n\nWith a frontchannel or backchannel, an “access token”/”refresh token” pair should be created\n\nAI gateway and RAG framework should authenticate “user access token”. If the token is authenticated, the next stage will be authorization.\n\nOn Behalf of Agents\n\nEvery agent has its own mutual TLS certificate\n\nAn “on behalf of agent” must introduce itself to OIDC with the certificate, agent id (or Active Directory credentials) and “real user’s access token” and finally must acquire a very short-lived “on behalf of token”\n\nAI gateway and RAG framework should authenticate “on behalf of token”. If the token is authenticated, the next stage will be authorization.\n\nAutonomous Agents\n\nEvery agent has its own mutual TLS certificate\n\nAn “autonomous agent” must introduce itself to OIDC with the certificate, agent id (or Active Directory credentials) and finally must acquire a very short-lived “agent access token”\n\nAI gateway and RAG framework should authenticate “agent access token”. If the token is authenticated, the next stage will be authorization.\n\nKill Switch\n\nIn case of any uncontrolled agent activity (or less probable but real user activity) the OIDC system must set a kill switch in order to reject the authentication activity!\n\nRBAC/Object-Based Authorization or ABAC (attribute-based access control) System\n\nResponsibilities of the system\n\nThe system should answer below questions and must work with OIDC: Is the end-user allowed to request on that GenAI inference or to get that vectorized document/document chunk Is the “on behalf of agent” allowed to request on that GenAI inference or to get that vectorized document/document chunk Is the “autonomous agent” allowed to request on that GenAI inference or to get that vectorized document/document chunk\n\nFine grained permissions (ABAC): system should cover the fine grained permission requests such as “Is the end-user allowed to execute on X resource within Y,Z values …”\n\nExternal GenAI Model Downloading Pipeline\n\nThe system can download GenAI models from external sources such as HuggingFace or Github through a secure process, with no known blocking vulnerabilities after scanning. The execution path should be;\n\nend user requires a genai model (fill a form on MLOPS Module)\n\nreading the source genai model repositories vulnerability reports. If any blocking vulnerability is reported in the source repository, reject the model.\n\ndownloading the model in a secure sandbox location\n\nexecuting genai attack vector and anti virus scans. If any blocking vulnerability is found, quarantine the model and notify the MLOps system.\n\nIf the model is in quarantine, expect a form from the requesting user and then a documented supervisor review\n\nIf no blocking vulnerabilities are found, initiate a copy job for the model files from sandbox to enterprise S3 environment\n\nRegister it on the MLOPS platform as a usable genai model\n\nThe critical part here is scanning the genai model and the available tools are;\n\nOSS tools: ClamAV antivirus and ProtectAI PROS: Open source and free to use CONS: Does not cover all GenAI models, and the database is out of date\n\nPalo Alto Prisma AIRS Platform: PROS: Palo Alto has claimed ProtectAI and has started to support it in enterprise way.\n\nCrowdStrike Falcon Guardian: Not just for genai model security/vulnerability scan tool. A total AI security framework.\n\nGathering Those Components Together\n\nBelow chart shows how an external model pipeline should work;\n\nCost Management (FinOps Methodology)\n\nSince GenAI infrastructure is expensive to operate, monitoring consumption is important. But the Cost Management system should not stick to just GenAI cost. It should cover the items below and connect them together:\n\nCover traditional ML pipeline cost\n\nCover GenAI inference cost for both on-prem and on-cloud usage\n\nCover ML development cost on Jupyter Hub or Google Colab with regarding traditional resources such as cpu, memory\n\nTotal Cost Of Ownership for the hardware infrastructure\n\nTotal Cost of AI for the labor prices, each ai artifact’s cost etc.\n\nBusiness value calculation and whether testing that value justifies the total investment on ai assets totally\n\nIMPORTANT #1: Cost tracking must be granular and accountable on “end user” / “application” / “agent” level.\n\nGathering Those Components Together\n\nTotal AI Security\n\nTotal AI security must consist of the subjects below:\n\nShadow AI:\n\nAny unauthorized ai usage should be detected.\n\nDLP (Data Leakage Prevention):\n\nFor a highly regulated industry such as financing it is very important to reject any request in case including customer value. And some other companies be careful about their intellectual property in case of any leak to cloud ai systems.\n\nPII (Personally Identifiable Information):\n\nAny PII data should be masked before sending to cloud ai providers. and if exists in the genai response must be re-fill in the correct sequence.\n\nGuardrail:\n\nIt’s the first place protection layer before genai inference gets started. Guardrails check AI requests and responses against defined security and usage rules. They may use prompt-based checks or other detection methods.\n\nSome of the base attacks that can be captured with a proper Guardrail\n\nPrompt injection mitigation: Prompt injection occurs when an attacker manipulates the prompt to cheat the LLM for ignoring its original instructions\n\nSystem Prompt Stealing Prevention: System prompt stealing occurs when an attacker try to force the LLM to acquire the underneath instructions or logic.\n\nJailbreak: An attacker tries to bypass safety alignments to generate harmful content.\n\nToxicity and Sentiment Filtering: Scans both input and output for hate speech, racism, profanity, harassment etc.\n\nAI Red Teaming\n\nAI red teaming is important when decide to;\n\nTest bypassing safety controls\n\nTrigger harmful outputs\n\nDiscover data Leakage\n\nTest guardrails Red teaming can be done by “manual (with highly educated human)” or “automated (with some commercial tools, such as CrowdString Falcon or Palo Alto Prisma AIRS)” A recommended tool is “CrowdStrike Safemind”\n\nContinuous GenAI Inference Monitoring\n\nTracking request and responses of genai inference is very important (no matter batch or real time tracking). Because with the power of ai the attack surface is always getting increasing easily but most of the assets which we have is static such as guardrails. Though enterprises find a way to Continuously monitor the genai activity for vulnerabilities. Also when agents come into play “Continuous GenAI Inference Monitoring” staff would transform into “Agent Activity Real Time Monitoring”. For that topic, review Google Article — Inside Agent Compute Engine\n\nPentest With AI Support\n\nAI pentest, automates and scales complex cyberattacks, forcing security teams to use AI just to keep up with hackers. There is some commercial tools to execute that kind of test but a highly trained person can achieve same goal with open-weight exploit models. A recommended tool is “CrowdStrike Safemind”\n\nGathering Those Components Together\n\nBelow chart shows a quick reference for how an enterprise should face with “Total AI Security” with below subjects.\n\nShadow AI\n\nDLP\n\nPII\n\nGuardrail\n\nContinuous GenAI Inference Monitoring\n\nAI Red Teaming\n\nPentest With AI Support\n\nAn Enterprise Level AI Playground\n\nAn AI playground is an environment where, the environment -but before production, just a playground- where the users can write python code in order to develop or test ai models, implement some quick mvp genai solutions on their own. The playground should centralize data access, enforce strict enterprise security, optimize costly GPU resource allocation, and ensure seamless team collaboration for rapid machine learning development. For that functionality, the tools below are useful\n\nJupyterHub: oss, on-prem, runs on container based kubernetes environment\n\nGoogle Colab: Just on cloud no on-prem\n\nEnterprise RAG / Secure Document Search\n\nThe Enterprise RAG platform should discover, register, index, and search unstructured documents from enterprise data sources. The first focus is secure document search, keeping document metadata and permissions linked to the original source.\n\nDocument Discovery and Registry\n\nA scheduled discovery workflow finds documents from existing enterprise sources.\n\nThe source integration runs through MCP (p-Nyx in the draft), covering SharePoint, Enterprise Content Management, relational/document databases, and S3 object storage.\n\nThe discovery process should consider source RBAC permissions.\n\nMongoDB serves as the Document Registry and holds information needed to track documents, versions, source locations, and permissions.\n\nA Document Registry UI allows new sources to be registered and monitored.\n\nReal users or enterprise applications may submit new or updated documents through the Document Load process.\n\nDocument Load publishes new or modified document events through the Event-Driven Architecture to trigger the discovery workflow.\n\nSource permission changes should also publish events to trigger Document Permission Synchronization.\n\nDocument Permission Synchronization retrieves the latest permissions from the original source and updates the Document Registry and Vector Database permission metadata.\n\nDocument Indexing\n\nCompare discovered documents with their registered versions.\n\nStart an indexing workflow for new or changed documents. Skip unchanged documents.\n\nParse the document and store its content and metadata in enterprise object storage.\n\nChunk the parsed content, generate embeddings, and upsert the results into the Vector Database.\n\nEmbedding Generation\n\nAn embedding model converts document chunks into numerical vectors representing their semantic meaning.\n\nThe embedding model must support Turkish and multilingual document retrieval.\n\nQwen3-Embedding-0.6B is the primary candidate for evaluation. It is an open-weight multilingual embedding model with Turkish language support, suitable for on-prem deployment.\n\nBAAI/bge-m3 is an alternative candidate. It supports multilingual embeddings and both dense and sparse retrieval, making it suitable for Hybrid Search.\n\nThe final embedding model should be selected after evaluating retrieval accuracy and inference performance on Turkish enterprise documents.\n\nThe selected embedding model should be managed and versioned so that document indexing and search use compatible embeddings.\n\nVector Indexing\n\nStore embeddings and their associated metadata in the Vector Database.\n\nEach indexed chunk should keep references to its original document, version, source location, and permission attributes.\n\nThe Vector Database should support vector similarity search, keyword search, and metadata filtering for Hybrid Search.\n\nEnterprise RAG / Secure Document Search\n\nReal users and enterprise applications access Enterprise RAG through a REST API.\n\nAuthenticate requests using Enterprise OIDC.\n\nAuthorize requests using enterprise RBAC/ABAC.\n\nDocument and chunk access must follow the original source permissions.\n\nPermission changes should be synchronized through the Event-Driven Architecture and reflected in the Document Registry and Vector Database.\n\nChunking Strategy\n\nStatic chunking: Split documents into fixed-size chunks with some overlap between them. It is simple to implement and fast, but sometimes breaks a sentence or related content.\n\nSemantic chunking: Split documents based on meaning and topic changes instead of fixed sizes. It may keep related information together, but requires more processing and evaluation.\n\nRetrieval Strategy\n\nUse Hybrid Search by combining vector similarity search and keyword search.\n\nConvert the user’s search query into an embedding using a compatible embedding model.\n\nRetrieve relevant chunks using semantic similarity and keyword matching.\n\nApply document-level permission filters so that users only retrieve documents and chunks they are authorized to access.\n\nUse Document Registry metadata to identify the original documents and their versions.\n\nReranking will not be implemented in the initial version.\n\nPrompt Construction\n\nFor RAG answer generation, construct the LLM prompt using the user’s question and authorized retrieved document chunks.\n\nInclude instructions that the LLM should answer based on the provided document context.\n\nSend the prompt to the selected LLM through the existing AI Gateway.\n\nCitation Strategy\n\nGenerated answers should include references to the original documents used as context.\n\nCitation metadata should be stored alongside each document chunk in the Qdrant Vector Database using its JSON payload capability.\n\nEach Qdrant payload should contain the document ID, chunk ID, document version, document title, source location, and page or section information when available.\n\nThe Enterprise RAG API should retrieve citation metadata directly from the authorized Qdrant search results, without requiring additional MongoDB queries for each citation.\n\nMongoDB Document Registry remains the authoritative source for document metadata, versions, and lineage. Citation metadata stored in Qdrant must stay synchronized with the Document Registry.\n\nGenerated citations should be based on the actual retrieved document chunks, not on document references generated independently by the LLM.\n\nThe Enterprise RAG API should return the generated answer together with its source citations so users can verify the information.\n\nHallucination Prevention\n\nThe LLM should be instructed to generate answers based on the retrieved document context.\n\nIf the retrieved documents do not contain sufficient information, the system should indicate that there is not enough evidence to answer. As a system-prompt-level guardrail, a fallback instruction should be defined, such as: “If the retrieved context does not provide sufficient evidence to answer the user’s question, do not generate an answer based on assumptions or the model’s own knowledge. Instead, return a predefined fallback message indicating that insufficient evidence was found.”\n\nGenerated answers should include source citations so users can verify the information.\n\nThese controls reduce hallucination risk but do not guarantee that every generated answer is correct.\n\nGaps — Not Covered Subjects\n\nSemantic Chunking (a better approach instead of static chunking)\n\nMultimodal RAG\n\nOntology\n\nGraph RAG\n\nSemantic Layer\n\nNatural Language Queries\n\nGathering Those Components Together\n\nThe chart below shows a sample of how an enterprise RAG, Vector Search, and Document Discovery tasks might work;\n\nAgentic AI Applications and MCP\n\nThe Agentic AI platform should support On-Behalf-Of Agents and Autonomous Agents. Agents run on Kubernetes/OpenShift, use an authentication sidecar, consume GenAI models through the existing AI Gateway, and access enterprise services through the MCP Gateway (p-Nyx). User OIDC, Application OIDC, and Agent OIDC are separate identity-provider artifacts, not just different flows within one identity provider.\n\nAgent Control Plane and Agent Registry\n\nThe Agent Control Plane / Agentic Development Platform manages agent registration, tool registration, deployment, lifecycle, and identity enrollment.\n\nThe Agent Registry keeps each approved agent’s identity, owner, version, endpoint, deployment status, and capabilities.\n\nThe Tool Registry keeps tool definitions, owners, target services, operations, and related access-policy metadata.\n\nAgent Deployment deploys approved agent versions to Kubernetes/OpenShift and tracks changes and retirement.\n\nAt startup, the authentication sidecar contacts the Agent Control Plane to obtain its registered, trusted agent identity.\n\nBootstrap trust is required: before issuing that identity or a certificate, the Control Plane validates the sidecar’s Kubernetes/OpenShift workload identity (for example a bound service-account token), deployment identity, and approved Agent Registry record. A claimed agent_id alone is not proof of identity.\n\nOnce validated, the sidecar requests a short-lived mTLS certificate from SecTrail. Certificate renewal and revocation should follow the agent lifecycle.\n\nThe registry and tool metadata can provide policy inputs to Enterprise RBAC/ABAC. The Registry is not the authorization engine: RBAC/ABAC makes the access decision at invocation time.\n\nAgent Authentication\n\nUser OIDC authenticates real users; Application OIDC authenticates enterprise applications; Agent OIDC issues agent-facing tokens and validates delegated-agent identity.\n\nDo not send reusable Active Directory usernames or passwords from agents. Use the validated workload identity and short-lived mTLS certificates to authenticate agent workloads.\n\nFor an On-Behalf-Of Agent, the real user first obtains a user access token from User OIDC. The agent sidecar then requests a short-lived delegated token from Agent OIDC using OAuth 2.0 Token Exchange (RFC 8693). Agent OIDC verifies the user token, the authenticated agent identity, and delegation permissions, then issues a token identifying the agent and delegated user with a limited audience and scopes.\n\nFor an Autonomous Agent, the sidecar requests a short-lived, agent-only access token using its authenticated workload identity. No user identity is implied.\n\nEach receiving service validates the token issuer, audience, expiration, signature or introspection result, and required scopes. The specific trust relationship between independent OIDC providers must be configured explicitly.\n\nMCP and Domain-Service Authorization\n\nThe MCP Gateway (p-Nyx) is the common entry point for agents invoking registered enterprise tools. An optional Enterprise Service Bus remains available for domain-service integration.\n\nMCP validates the agent’s incoming token and checks Enterprise RBAC/ABAC for the agent, requested tool, operation, and delegation context.\n\nThe downstream domain service independently checks whether the authenticated identity can perform the requested operation on the requested business resource. Permission to call a tool does not automatically grant permission to access all business records.\n\nAgent Access Token (for MCP): Agent OIDC issues a short-lived access token to the agent or its sidecar. When this token has aud = p-Nyx MCP Gateway, its purpose is to authorize the Agent → MCP Gateway call. This is a standard OAuth access token, not a separate MCP-specific token type. The token may identify an autonomous agent or an agent acting on behalf of a user.\n\nDo not forward the Agent Access Token unchanged to a downstream Domain API. For example, an agent invokes the getStatement MCP tool with its Agent Access Token. MCP validates that token and checks whether the agent can invoke getStatement. But the banking statement API is a different protected service: a token issued for the MCP Gateway is not automatically valid for the banking API. Forwarding it unchanged would let a token intended for one service be reused at another and would bypass the intended service-specific security boundary.\n\nDownstream Domain API credential: After the MCP tool call is authorized, the MCP Gateway obtains or uses a separate credential accepted by the target Domain API (for example, an OAuth access token with aud = core-banking-api, obtained through an approved token exchange where supported). The Domain API validates that credential and independently checks authorization for the requested business operation and resource, such as whether the user may read a particular customer’s statement. For an On-Behalf-Of Agent, preserve the authorized user and agent delegation context as supported by the downstream service; for an Autonomous Agent, enforce its own service permissions. Do not assume the MCP authorization decision grants access to all domain data.\n\nTwo distinct permission checks: (1) MCP Gateway checks can this agent call this tool?; (2) the downstream Domain API checks can this identity perform this operation on this business resource? Both decisions should be auditable.\n\nTool and domain-service authorization decisions should be traceable in the audit records.\n\nObservability and Auditing\n\nCollect authentication and certificate events, identity enrollment, token exchange, tool invocations, RBAC/ABAC decisions, domain-service access decisions, and AI Gateway inference activity.\n\nCorrelate records with trace_id, agent_id, delegated user_id where applicable, tool_id, target_service, requested operation, decision, and timestamp.\n\nSensitive tokens, credentials, and confidential request data should not appear in ordinary telemetry logs.\n\nCommercial Platform Example\n\nMicrosoft Foundry Agent Service + Foundry Control Plane / Microsoft Agent 365 is a commercial reference worth evaluating for agent deployment, registry, governance, agent identity, tracing, and agent governance capabilities. It is not a claim of drop-in compatibility with the proposed SecTrail / separate OIDC / p-Nyx / on-prem OpenShift architecture; integration and hosting constraints require validation.\n\nAmazon Bedrock AgentCore is another candidate for runtime, identity, gateway and observability capabilities, with its own cloud integration model.\n\nAgent Development Framework\n\nThe enterprise Agentic AI platform should be SDK-agnostic. AI Product Teams can develop agents using their preferred framework without being forced to use a specific development SDK.\n\nSome of the available agent development frameworks are:\n\nLangGraph: Suitable for developing complex, stateful agent workflows with human-in-the-loop and durable execution capabilities.\n\nLangChain: Suitable for developing agents with LLM integrations, tool calling, and simpler agent workflows.\n\nMicrosoft Semantic Kernel: Suitable for AI applications developed within the Microsoft ecosystem, especially .NET applications.\n\nMicrosoft AutoGen: Suitable for multi-agent applications, agent collaboration, and event-driven agent orchestration.\n\nCrewAI: Suitable for developing multiple agents working together on tasks using role-based Crews and structured Flows.\n\nMicrosoft Agent Framework is also an alternative to consider for new Microsoft-based agent development, as it succeeds Semantic Kernel and AutoGen.\n\nThe enterprise should not enforce a specific development framework. However, all developed agents must follow the same enterprise registration, deployment, authentication, authorization, and monitoring requirements.\n\nAgent Registration and Deployment\n\nDevelopers can develop agents using any preferred SDK or framework.\n\nOnce development is completed, the agent must be registered in the Agent Registry Module, including its metadata, owner, development framework, version, and required capabilities.\n\nThe Agent Registry assigns a unique enterprise Agent ID to the registered agent.\n\nThe agent can be deployed using the Agent Control Plane or a separate enterprise CI/CD pipeline.\n\nAfter deployment, the agent’s Authentication Sidecar interacts with the Agent Control Plane to validate the deployed workload against its registered Agent ID and obtain a trusted workload identity.\n\nThe Authentication Sidecar then follows the enterprise certificate and Agent OIDC authentication process.\n\nAgent access to LLM inference and enterprise tools must follow the existing AI Gateway, MCP Gateway (p-Nyx), and RBAC/ABAC mechanisms.\n\nAgent execution and tool activities must be monitored and audited through the enterprise observability infrastructure.\n\nThe Agent Registry and Agent Control Plane should manage agents independently of the SDK used to develop them. An agent developed using LangGraph, LangChain, Semantic Kernel, AutoGen, CrewAI, or another framework should follow the same enterprise lifecycle and security requirements.\n\nOnly REST/SOAP Communication From MCP to Domain Services\n\nAI Agents must not directly access enterprise databases, file systems, S3 object storage, or other domain-level resources.\n\nAll access to enterprise domain resources must go through the MCP Gateway, using validated REST APIs or SOAP services.\n\nDomain services must be developed and maintained by their responsible domain-specific software development teams.\n\nThese teams have the required business and technical knowledge of their domains and are responsible for implementing business rules, input validation, authorization, and data access controls.\n\nThe MCP Gateway should expose approved domain operations as tools, rather than allowing agents to execute arbitrary database queries or directly manipulate underlying resources.\n\nThis approach prevents uncontrolled agent actions from bypassing domain business rules and causing unintended changes or damage to enterprise data.\n\nThe main principle: AI Agents can request business operations, but only domain services are allowed to execute those operations against underlying enterprise resources.\n\nA2A Communication\n\nNot covered\n\nGathering Those Components Together\n\nBelow chart shows a sample frame for agent readiness;\n\nThe module should help the enterprise evaluate AI models and applications before production and continuously monitor their quality after deployment.\n\nEvaluation\n\nEvaluate GenAI models and AI applications based on their accuracy, relevance, safety, hallucination risks, and business use-case requirements.\n\nEvaluation Strategies\n\nGolden Datasets: Prepare a set of test inputs with expected outputs, approved by domain experts. Run the same tests against different models or model versions and compare the results.\n\nLLM-as-a-Judge: Use another LLM to evaluate model outputs against predefined criteria, such as accuracy, relevance, completeness, and safety. Judge results should be periodically checked against human evaluations.\n\nHuman Evaluation: Domain experts manually review model outputs, especially for sensitive business use cases or when automated evaluation is not reliable.\n\nGroundedness / Faithfulness Evaluation: Check whether the generated answer is supported by the provided documents, images, audio, or video content. This is important for hallucination detection.\n\nTask-Specific Evaluation: Use evaluation metrics designed for the model’s actual task, such as text generation, image understanding, speech recognition, or video understanding.\n\nEvaluation by Model Type\n\nText LLMs: Use question-answer golden datasets, LLM-as-a-Judge, and human evaluation. Measure factual correctness, instruction following, relevance, hallucinations, and safety. For RAG applications, also evaluate document retrieval quality, groundedness, and citation correctness.\n\nVision-Language Models (VLMs): Use images with expected answers or annotations. Evaluate image understanding, visual question answering, OCR accuracy, object recognition, and whether the model invents objects or information not present in the image.\n\nSpeech Models: For Speech-to-Text (STT), compare transcriptions against reference transcripts using Word Error Rate (WER) or Character Error Rate (CER). For Text-to-Speech (TTS), evaluate speech naturalness, intelligibility, pronunciation, and voice quality using human ratings such as Mean Opinion Score (MOS).\n\nVideo Models: For video understanding, use video clips with expected descriptions, events, or answers. Evaluate event recognition, temporal understanding, timestamps, factual accuracy, and hallucinations. For video generation, evaluate visual quality, prompt adherence, temporal consistency, and human preference.\n\nGeneral Requirements\n\nEvaluation datasets and results should be versioned and managed through the MLOps/GenAI Ops platform.\n\nEvaluation criteria and acceptance thresholds should be defined according to the business use case.\n\nThe Evaluation Team should review results before a model or application is approved for production.\n\nBenchmarking\n\nCompare models using quality, latency, throughput, token consumption, and inference cost.\n\nBenchmark both on-prem and cloud models under comparable conditions.\n\nUse the existing MLOps/GenAI Ops, Model Serving, and AI Gateway modules for model versions, traffic routing, and telemetry.\n\nStore evaluation results and make them available to the Evaluation Team for review and approval.\n\nUseful strategies: Versioned evaluation datasets, centralized experiment tracking, release approval gates, and continuous production monitoring.\n\nInternal Copilots\n\nInternal Copilots should help enterprise employees with their daily tasks, such as document research, knowledge discovery, software development, email summarization, presentation creation, and business operations.\n\nMain Capabilities\n\nEnterprise Knowledge Assistant: Use Enterprise RAG and Secure Document Search to answer questions based on internal documents while respecting user permissions.\n\nAI Coding Assistant: Support developers with code generation, code reviews, debugging, and software documentation.\n\nEmail Assistant: Summarize emails, identify important information, and help employees prepare replies.\n\nDocument and Presentation Assistant: Help employees create documents, reports, and presentations using authorized enterprise information.\n\nBusiness Process Assistant: Help employees execute business operations through approved domain services using the Agentic AI Platform and MCP Gateway.\n\nSecurity Requirements\n\nAudit Log Protection: Audit logs must be obfuscated or masked to prevent leakage of employees’ private information and confidential enterprise data. Required operational and security traceability must still be preserved.\n\nNo Arbitrary Code Execution: Copilots must not execute arbitrary code, scripts, or shell commands on employees’ computers.\n\nInternal Guardrails: Apply enterprise guardrails to prevent prompt injection, sensitive information leakage, unsafe responses, and unauthorized actions.\n\nNo Internet Access: Internal Copilots must not have direct internet access. They should only communicate with approved internal enterprise services.\n\nPermission-Aware Access: Copilots must respect the employee’s existing RBAC/ABAC permissions. They must not retrieve documents, emails, or other resources that the employee cannot access.\n\nHuman Approval: Actions that modify enterprise data, send emails, or trigger business operations should require explicit user approval before execution.\n\nImplementation Approach\n\nInternal Copilots should reuse the existing Enterprise RAG, AI Gateway, Model Serving, Agentic AI, and Total AI Security components.\n\nCopilots can be developed in-house or acquired as commercial products, provided they comply with enterprise security and integration requirements.\n\nStart with a small number of use cases and expand based on employee adoption and business value.\n\nFuture Multimodal Capabilities\n\nThe enterprise AI platform should be ready to support AI models that process and generate content beyond text, such as images, speech, audio, and video.\n\nMain Capabilities\n\nVision-Language Models (VLM): Understand images, scanned documents, tables, and charts.\n\nSpeech Models: Support Speech-to-Text (STT), Text-to-Speech (TTS), and audio understanding.\n\nVideo Models: Support video summarization, event detection, and video understanding.\n\nMultimodal RAG: Extend Enterprise RAG to search and retrieve information from text, images, audio, and video.\n\nMultimodal Internal Copilots: Allow employees to interact with enterprise AI services using text, images, and voice.\n\nArchitecture Requirements\n\nReuse the existing Model Serving, AI Gateway, Enterprise RAG, and GPU infrastructure.\n\nSupport multimodal input/output formats and appropriate inference frameworks.\n\nExtend Total AI Security to inspect and protect multimodal content.\n\nEvaluate models using modality-specific evaluation strategies.\n\nExtend FinOps to track multimodal inference, GPU consumption, and storage costs.\n\nAI Ethics & EU AI Act\n\nWhat it means\n\nAI Ethics: What the company should do to use AI responsibly, fairly, transparently, and safely.\n\nEU AI Act: What the organization is legally required to do for AI systems that fall within the Act’s scope.\n\nThe enterprise AI platform should ensure that AI systems are developed and operated responsibly, following AI ethics principles, applicable regulations, and enterprise governance rules.\n\nFor an enterprise, special attention should be given to customer rights, operational safety, employee management, and personal data protection.\n\nAI Ethics Principles\n\nFairness and Non-Discrimination: AI systems should not create unfair treatment of clients or employees based on personal characteristics.\n\nTransparency: Clients and employees should be informed when interacting with AI systems or when AI is involved in decisions affecting them, as required by applicable rules.\n\nHuman Oversight: Important decisions affecting operational safety, clients, or employees should remain under appropriate human control.\n\nPrivacy and Data Protection: Personal and sensitive data should be protected according to GDPR and other applicable data protection regulations.\n\nAccountability: Every AI use case should have a responsible business owner, with traceable decisions and documented responsibilities.\n\nEU AI Act Compliance\n\nAI Risk Classification: Each AI use case should be assessed against EU AI Act risk categories, including prohibited practices and high-risk systems.\n\nSector-Specific Assessment: Pay special attention to AI used in recruitment, employee monitoring, biometric identification, customer services, and safety-critical operations.\n\nHigh-Risk AI Requirements: Where applicable, define risk management, technical documentation, testing, human oversight, monitoring, and audit requirements.\n\nAI Transparency: Apply relevant transparency requirements for client-facing chatbots and AI-generated content.\n\nContinuous Compliance: Reassess AI applications when their purpose, models, operating conditions, or applicable regulations change.\n\nHigh-Risk AI — Enterprise Examples\n\nRecruitment: AI evaluating and ranking job candidates.\n\nEmployee Performance: AI monitoring employee performance or influencing promotion, termination, or certain work-allocation decisions.\n\nBiometric Identification: AI identifying individuals through remote biometric recognition, where the system falls within the relevant legal category.\n\nProduct Safety: AI used as a safety component of a regulated product or system, where the applicable product-safety and conformity-assessment requirements are met.\n\nPublic Services: AI supporting access to essential public services or benefits, where the system falls within the relevant legal category.\n\nEach use case should be reviewed for EU AI Act classification, GDPR requirements, and applicable sector-specific regulations before production.\n\nIntegration With Enterprise AI Governance\n\nThe AI360 Use Case Assessment & Architectural Compliance Committee should review AI ethics and regulatory requirements during Incubation, MVP, Pilot, and Production stages.\n\nThe committee should involve Cybersecurity, Enterprise Architecture, relevant business teams, and Legal/Compliance representatives when required.\n\nUse the existing MLOps/GenAI Ops, Total AI Security, Evaluation, Auditing, and AI Gateway components to support technical compliance controls.\n\nFor AI used in safety-critical systems, involve the responsible safety and certification teams.\n\nAI applications should not proceed to production without the required risk classification, compliance review, and approval.\n\nPlatform Operating Model\n\nThis section looks for an answer to the question: “how should teams be better organized in order to execute the above reference architecture?”\n\nPlatform Team Responsibilities\n\nAI Architecture Team\n\nResponsibilities: - Implementing and operating the reference architecture. - Coordinating with other IT teams. - Supporting AI teams and troubleshooting issues. - Developing MVPs/PoCs for new tools or closing gaps with in-house solutions.\n\nTeam Composition: Mostly software developers and enterprise architects.\n\nModel Scout Team\n\nResponsibilities: - Scouting new AI models and monitoring vulnerabilities in existing models. - Creating awareness of upcoming AI models and important improvements. - Operating the Model Serving Module.\n\nTeam Composition: Mostly data scientists, with fewer data engineers.\n\nEvaluation Team\n\nResponsibilities: - Evaluating new GenAI models and matching them to suitable use cases. - Operating the Guardrail and Prompt Registry modules.\n\nTeam Composition: Mostly data scientists.\n\nAI Product Teams\n\nResponsibilities: - Developing AI applications and business use cases using the enterprise AI platform. - Testing and evaluating their AI applications before deploying them to production. - Operating, monitoring, and improving their AI applications with support from the AI Platform Team.\n\nTeam Composition: People from business units, Data Engineering (DE), and Data Science (DS) teams.\n\nAI360 Use Case Assessment & Architectural Compliance Committee\n\nResponsibilities: - Reviewing AI use cases against all requirements while ensuring alignment with enterprise architecture, AI architecture, frameworks, constraints, and best practices. - Assessing each use case across architecture, Data Science (DS), and Data Engineering (DE) dimensions throughout the following stages: Incubation MVP — Definition of Done (1/3) Pilot — Definition of Done (2/3) Production — Definition of Done (3/3)\n\nTeam Composition: Representatives from business units (use-case-specific), AI Architecture, Data Engineering (DE), Data Science (DS), IT/Enterprise Architecture, and Cybersecurity teams.\n\nRoadmap\n\nThis section looks for an answer to the question: “In which order should we implement that kind of an architecture?”\n\nTechnical Implementation Roadmap\n\nFirst 3 Months\n\nGoal: Implement the basic foundations as soon as possible.\n\nImplement MLOps / GenAI Ops with the following capabilities: Model Registry Auditing Prompt Registry and Versioning\n\nImplement Model Serving with the following capabilities: On-prem GenAI model serving On-prem traditional ML model serving\n\nImplement AI Gateway with the following capabilities: Authentication Authorization Traceability / Observability Rate Limiting and Throttling Semantic Routing Context Caching Traffic-Aware Routing\n\nImplement the initial Enterprise RAG and Secure Document Search pipeline.\n\nFirst 6 Months\n\nGoal: Expand the foundational AI components.\n\nComplete the remaining capabilities of MLOps / GenAI Ops.\n\nComplete the remaining capabilities of Model Serving.\n\nComplete the remaining capabilities of AI Gateway.\n\nImplement Local AI Coding Harnesses to improve development productivity.\n\nImplement the External Model Download Pipeline.\n\nImplement Total AI Security with the following capabilities: DLP PII Protection Guardrails\n\nImplement early-stage Cost Management (FinOps) with the following capabilities: Track traditional ML pipeline costs. Track ML development costs in JupyterHub / Google Colab environments. Track GenAI inference costs for both on-prem and cloud usage using NCU calculations.\n\nFirst Year\n\nGoal: Complete the AI architecture and make all components operational.\n\nComplete the remaining capabilities of Total AI Security.\n\nComplete the remaining capabilities of Cost Management (FinOps).\n\nImplement Agentic AI Applications and MCP.\n\nImplement Server-Side AI Coding Harnesses.\n\nImplement Evaluation, Benchmarking, A/B Testing, and Canary Deployment.\n\nEnterprise AI Adoption & Organizational Development Roadmap\n\nThis section looks for an answer to the question: “How do we build an AI culture for the Enterprise?” Because human nature is a bit defensive, it’s hard to change a person’s habits or working style immediately and if you think about the enterprise level, resistance scales much more.\n\nFirst 3 Months — Awareness and Alignment\n\nGoal: Build a common understanding of AI and identify initial business use cases.\n\nOrganize AI awareness sessions for developers, business teams, and management.\n\nProvide practical training on GenAI usage, limitations, and security risks.\n\nDefine initial responsibilities for the AI Platform Team and AI Product Teams.\n\nIdentify business use cases for initial RAG and Secure Document Search projects.\n\nFirst 6 Months — Skills and Adoption\n\nGoal: Help teams use the AI platform independently and effectively.\n\nProvide hands-on training for developers on RAG, AI APIs, and AI-assisted coding.\n\nTrain security and compliance teams on AI-specific risks.\n\nEstablish regular feedback sessions with early platform users.\n\nStart measuring AI adoption, developer productivity, and user satisfaction.\n\nFirst Year — Enterprise AI Working Culture\n\nGoal: Make AI a regular part of enterprise software development and business operations.\n\nProvide advanced training on Agentic AI, MCP, and AI-assisted software engineering.\n\nTrain teams on evaluating AI outputs and applying human oversight.\n\nEstablish an internal AI community for sharing experience and reusable solutions.\n\nOrganize management workshops on AI investment, business value, and accountability.\n\nReview organizational AI adoption and actual business benefits", "url": "https://wpnews.pro/news/building-an-enterprise-genai-platform-reference-architecture-security-and", "canonical_source": "https://pub.towardsai.net/building-an-enterprise-genai-platform-reference-architecture-security-and-governance-8d83c2d8cb39?source=rss----98111c9905da---4", "published_at": "2026-11-02 17:01:02+00:00", "updated_at": "2026-11-02 17:54:53.998523+00:00", "lang": "en", "topics": ["artificial-intelligence", "generative-ai", "mlops", "ai-agents", "ai-infrastructure"], "entities": ["Red Hat Ceph", "MinIO", "MongoDB", "Milvus", "Qdrant", "Redis Enterprise", "Cloudera", "Keycloak"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/building-an-enterprise-genai-platform-reference-architecture-security-and", "markdown": "https://wpnews.pro/news/building-an-enterprise-genai-platform-reference-architecture-security-and.md", "text": "https://wpnews.pro/news/building-an-enterprise-genai-platform-reference-architecture-security-and.txt", "jsonld": "https://wpnews.pro/news/building-an-enterprise-genai-platform-reference-architecture-security-and.jsonld"}}