Private AI Is Not Model Hosting: A Reference Architecture for Sovereignty, Identity, GPUs, and Operations on VCF 9.1

Most private AI conversations start too low in the stack.

The first question is usually some version of, “Can we run the model on-prem?” That is not a bad question, but it is not the architecture question. Running a model on-premises proves that you can place compute near data. It does not prove that you have a private AI platform.

That is the difference between model hosting and private AI.

Model hosting is a deployment task. Private AI is an operating model.

That distinction matters because Broadcom is positioning VMware Cloud Foundation 9.1 as a production AI private cloud platform, not just a place to run isolated inference servers. The VCF 9.1 announcement calls out production AI workloads, inference and agentic applications, multi-tenant AI isolation, high-speed networking, AI observability, policy controls, and data sovereignty.

This article lays out a reference architecture for private AI on VCF 9.1 using four practical design lenses: sovereignty, identity, GPU architecture, and operations.

The Core Shift: From “Where Does the Model Run?” to “How Is AI Governed?”

If a customer reduces private AI to “run the model in my data center,” the design will usually miss the real production requirements.

A model endpoint by itself does not create governance.
A GPU cluster by itself does not create sovereignty.
A private subnet by itself does not create trust.
A RAG pipeline by itself does not create safe data access.

The architecture has to treat AI as a platform service that touches infrastructure, data, identity, networking, security, lifecycle, and operations.

VCF 9.1 is relevant here because Broadcom is pushing the platform toward a unified private cloud operating model for VMs, containers, inference workloads, agentic applications, and traditional workloads. The release messaging highlights one infrastructure layer for AI, Kubernetes, and VMs, along with model and GPU metrics, policy injection, data sovereignty controls, and secure model access.

The practical design goal is not simply this:

“Deploy a model endpoint.”

The practical design goal is this:

“Deliver approved AI services to approved users, against approved data, through approved network paths, using governed accelerator capacity, with observable runtime behavior and auditable lifecycle controls.”

That is a different architecture.

Reference Architecture at a Glance

The diagram below shows the control model I would use to frame a private AI design conversation on VCF 9.1. The important thing to notice is that the model runtime is only one layer. Identity, policy, network boundaries, model governance, data access, GPU placement, and operations all participate in the production path.

This is why “private AI” should be designed more like a cloud platform than a special-purpose appliance. The model endpoint is a product of the platform. It should not become the platform.

Model Hosting Versus a Private AI Operating Model

A useful way to reset the conversation is to separate the proof-of-concept view from the production architecture view.

The second column is where most failed private AI programs get stuck. They can run a model, but they cannot safely offer it as a service.

Sovereignty Starts With Data Boundaries, Not Hardware Location

A private cloud is not automatically sovereign just because it is physically located in a corporate data center.

Sovereignty has to be designed into the data path. That means identifying where data originates, how it is classified, where it is indexed, what the model is allowed to retrieve, whether prompts and responses are retained, and which external connectors are allowed to participate.

For AI, the sovereignty boundary has more moving parts than a traditional application. A RAG service might touch file shares, object storage, ticketing systems, SQL databases, document repositories, vector indexes, embeddings, model endpoints, audit logs, and API gateways. Every one of those touchpoints can become a control gap.

VCF Private AI Services in 9.1 adds Model Context Protocol support with governance, which Broadcom positions as a way to integrate AI assistants with internal content repositories and external tools without building custom connectors. That is useful, but it also raises the bar for policy design. MCP connectors should not become an uncontrolled side door into enterprise systems.

A practical sovereignty design should define:

The architecture should assume that “no public cloud inference” is only the first control. The harder control is proving that enterprise data, model artifacts, prompts, embeddings, and tool calls stay inside approved boundaries.

Identity Is the Control Plane for Private AI Consumption

Private AI should not rely on shared API keys, local accounts, or manually distributed endpoint credentials. That might work for a lab, but it breaks down quickly when multiple teams start consuming AI services.

VCF 9.1 extends unified SSO, password, and certificate management with broader component coverage, fleet-level controls, and vault and CA integrations. VMware’s VCF security update also describes the VCF Identity Broker as the broker between VCF components and an external identity provider or directory service.

That identity layer matters because AI consumption is not limited to human users. You need to govern at least four identity types:

VCF 9.1 also introduces OAuth 2.0 authentication patterns for PowerCLI and VCF SSO. The documented workflow creates an API client, generates an API token, exchanges it for a short-lived bearer token, and uses that bearer token to authenticate to VCF components such as VCF Automation, VCF Operations, vSphere, and NSX.

That pattern is exactly the kind of control private AI needs. AI platform automation should not be built around stored passwords or personal administrator credentials. It should use scoped, revocable, auditable access.

A good private AI identity model on VCF should include:

• VCF SSO integrated with the enterprise identity provider.
• Group-based roles mapped to AI platform personas.
• Separate project ownership for data science, application, platform, and security teams.
• Service identities for model endpoints and RAG connectors.
• Short-lived tokens for automation.
• Auditable access to model deployment, endpoint creation, data indexing, and connector configuration.

Identity is not a supporting detail. It is the boundary between “AI sandbox” and “AI service.”

Network Architecture Determines Whether Private AI Can Scale Safely

A private AI architecture needs more than a VLAN with a firewall rule.

The network design has to separate tenants, protect shared services, control egress, and keep model endpoints reachable without turning every AI project into a custom network build.

For VCF Private AI Services, the Supervisor networking choice is a major design decision. VMware’s June 2026 guidance notes that the VDS and Foundation Load Balancer model can provide a streamlined path to get AI endpoints running quickly, while deploying the Supervisor with VPC networking unlocks more capabilities for security, multi-tenancy, and self-service.

That distinction is important.

A proof of concept may be fine with the simpler model. A production private AI platform should usually lean toward VPC networking if the organization expects multiple teams, shared services, governed ingress, and delegated consumption.

VCF 9.1 also introduces or expands network services around VPCs, including network services for Virtual Private Clouds, Connectivity Policy for cross-VPC communication, and VPC Network Span for targeting subnets to specific clusters.

A production layout might use:

The goal is to avoid two extremes.

The first extreme is the lab design, where everything is reachable because that makes the demo easy. The second extreme is the over-isolated design, where every new model endpoint requires manual firewall tickets, custom routing, and weeks of coordination.

The better pattern is governed self-service: predefined network zones, approved connectivity patterns, and clear exception handling.

GPU Design Is a Scheduling, Sharing, and Lifecycle Problem

GPU architecture is often where private AI planning gets expensive fast.

A customer may start with a simple assumption: “We need GPUs.” That is not specific enough. The architecture needs to distinguish between AI workstations, lightweight inference, shared model serving, high-throughput inference, multi-GPU workloads, and multi-host training or distributed inference.

VCF 9.1 adds several relevant capabilities and ecosystem updates. Broadcom’s Private AI Services update says VCF 9.1 supports DirectPath Enablement for NVIDIA AI infrastructure, giving a single VM high-performance, exclusive GPU access. The same update calls out support for NVIDIA Blackwell options, NVIDIA HGX with Blackwell GPUs and NVLink Switch, and NVIDIA ConnectX-7 and BlueField-3 with Enhanced DirectPath I/O for high-speed multi-host AI data transfer.

The vSphere update for VCF 9.1 also notes that NVIDIA GPUs used for vGPU can be configured with time slicing and MIG mode for more efficient sharing.

That gives architects a broader set of choices, but also more responsibility.

The mistake is treating GPU assignment as a one-time placement decision.

In production, GPU capacity needs a service model:

• Standard GPU profiles.
• Approval workflow for exclusive GPU access.
• Quotas by project or organization.
• Runtime metrics for utilization, memory, temperature, and throughput.
• Driver and firmware lifecycle ownership.
• Compatibility validation across VCF, ESX, NVIDIA AI Enterprise, guest OS, runtime, and model framework.
• Placement rules that understand data locality, network locality, and failure domains.

A GPU that is technically available is not always the right GPU to assign.

Model Governance Is More Than a Model Store

Model governance starts with a model store, but it should not end there.

VMware Private AI Foundation with NVIDIA has included Model Store and Model Runtime capabilities, with VMware describing the platform as supporting secure model delivery, model deployment and scalability, NVIDIA NIM, air-gapped deployment options, and reference workflows. VMware’s Private AI Services guidance also describes Model Store, Model Endpoints, Data Indexing and Retrieval, and Agent Builder as part of the Private AI Services model.

Those services help move the conversation away from “someone downloaded a model and ran it somewhere” toward a controlled model lifecycle.

A practical model lifecycle should include:

This is especially important for RAG and agentic systems.

A base model can be approved while the retrieval source is not.
A connector can be approved for one project and completely inappropriate for another.
An MCP tool can be safe with read-only access and risky with write access.
A vector index can quietly preserve data that the source system later removed.

The architecture has to govern the model, the data, the tools, and the runtime behavior together.

Operations Must See the Model, the GPU, and the Platform

Traditional infrastructure monitoring is not enough for private AI.

Knowing that the VM is powered on does not tell you whether the model is useful. Knowing that the Kubernetes pod is running does not tell you whether time-to-first-token is acceptable. Knowing that the GPU is allocated does not tell you whether it is efficiently used.

VCF Private AI Services 9.1 includes an AI Metrics Observability Dashboard. VMware describes model metrics such as cache utilization, tokens generated per request, token throughput, time to first token, and end-to-end request latency. It also lists GPU utilization metrics such as utilization, temperature, power usage, memory temperature, and memory clock, with a note that these dashboards require Grafana.

That is the right direction because AI operations needs three levels of visibility:

VCF 9.1 also adds or expands operational controls that matter adjacent to AI, including audit trail improvements, password policy checks, certificate management enhancements, and fleet-level operations.

Private AI operations should be run like a platform SRE function, not a one-off infrastructure queue.

Example Control Policy for a Private AI Project

The following YAML is not a VCF product schema. It is a reference control document that platform teams can translate into VCF roles, namespace quotas, VPC policies, registry rules, model approval workflows, runtime templates, and monitoring requirements.

The point is to make the operating model explicit before teams start deploying endpoints.

apiVersion: governance.dtd.local/v1
kind: PrivateAIProjectPolicy
metadata:
  name: claims-summarization-prod
  owner: insurance-platform-team
  environment: production

spec:
  sovereignty:
    allowedLocations:
      - us-central-private-dc1
    allowedDataClasses:
      - confidential
      - regulated-customer-data
    outboundInternet: deny
    promptRetention: encrypted-30-days
    responseRetention: encrypted-30-days
    embeddingRetention: encrypted-180-days
    approvedExternalTools:
      - none

  identity:
    humanGroups:
      platformAdmins: grp-vcf-ai-platform-admins
      modelOperators: grp-ai-mlops-operators
      appConsumers: grp-claims-app-service-owners
      auditors: grp-grc-ai-auditors
    serviceIdentities:
      modelRuntime: svc-claims-summary-runtime
      ragIndexer: svc-claims-summary-indexer
      ciPipeline: svc-claims-summary-release
    tokenPolicy:
      automationTokens: short-lived
      sharedSecretsAllowed: false
      emergencyAccessRequiresTicket: true

  modelGovernance:
    modelStore: harbor.private.example.com/model-store
    approvedModels:
      - name: mistral-small-approved
        version: "2026.06.15"
        approvalTicket: GRC-AI-1842
        allowedUseCases:
          - claims-summary
          - internal-analyst-assist
    prohibitedUse:
      - customer-facing-autonomous-decisions
      - policy-denial-generation
    rollbackModelVersion: "2026.05.20"

  runtime:
    endpointName: claims-summary-api
    runtimeEngine: vllm
    apiGateway: private-ml-api-gateway
    minReplicas: 2
    maxReplicas: 6
    requiredMetrics:
      - time_to_first_token
      - end_to_end_latency
      - token_throughput
      - request_error_rate
      - gpu_utilization
      - gpu_memory_pressure

  compute:
    acceleratorProfile: nvidia-vgpu-mig-20gb
    allowDirectPathGpu: false
    cpuInferenceFallback: false
    quota:
      maxGpuProfiles: 6
      maxCpuCores: 96
      maxMemoryGb: 768

  network:
    projectVpc: vpc-ai-claims-prod
    ingress:
      allowedSources:
        - claims-app-prod
        - analyst-portal-prod
      publicExposure: false
    egress:
      allowedDestinations:
        - claims-db-readonly
        - object-store-claims-documents
        - vector-db-claims-prod
      defaultAction: deny
    connectivityPolicy: restricted-community

  dataAccess:
    retrievalSources:
      - name: claims-db-readonly
        mode: read-only
        piiFiltering: required
      - name: object-store-claims-documents
        mode: read-only
        documentTypes:
          - pdf
          - docx
          - txt
    indexRefresh:
      schedule: "0 */6 * * *"
      maxStalenessHours: 8

  operations:
    auditRetentionDays: 365
    dashboardRequired: true
    alertRoutes:
      latency: ai-platform-sre
      gpuSaturation: vcf-operations
      policyViolation: security-operations
      retrievalFailure: data-platform-team

  lifecycle:
    patchWindow: "Sunday 02:00-06:00 local"
    modelReviewIntervalDays: 90
    recoveryTier: tier-1
    backupIncludes:
      - model-endpoint-config
      - model-store-metadata
      - vector-index
      - namespace-policy
      - audit-records

This kind of control document is valuable because it forces the architecture discussion to happen before deployment. It also gives each team a clear translation point:

• VCF admins translate it into platform constructs.
• Network teams translate it into VPC and security policy.
• Security teams translate it into audit and access controls.
• MLOps teams translate it into model lifecycle and runtime behavior.
• Data owners translate it into retrieval and retention policy.

Ownership Model: Who Runs Private AI?

A private AI platform fails when every team assumes another team owns the hard parts.

The cleanest operating model separates responsibilities without creating silos.

The operating model should be documented as part of the architecture. Otherwise, the first production incident will expose the gaps.

A Phased Path to Production

A practical VCF 9.1 private AI adoption path should move in phases. Skipping directly from lab inference to enterprise self-service is how organizations create shadow AI inside their own private cloud.

Define the AI Control Boundary

Start with the governance model before ordering more GPUs.

Define approved data classes, model sources, tenant boundaries, external connector rules, identity groups, logging requirements, and recovery expectations. This phase should produce a policy baseline that can be mapped into VCF roles, VPCs, namespaces, quotas, and catalog items.

Build the Platform Foundation

Stand up the required VCF components, identity integration, vSphere Supervisor or VKS layer, VPC networking model, storage policies, GPU-enabled hosts, model registry, and observability path.

This is where the architecture should validate hardware compatibility, NVIDIA AI Enterprise requirements where applicable, driver and firmware dependencies, and the Broadcom support matrix. Treat compatibility as an architecture input, not an installation footnote.

Deliver One Controlled AI Service

Pick a narrow workload with real value and manageable risk. A summarization or internal knowledge retrieval use case is often better than a broad “enterprise assistant” as the first production candidate.

The first service should prove:

• governed model onboarding;
• identity-based access;
• private endpoint exposure;
• controlled retrieval;
• GPU or CPU profile assignment;
• model and GPU metrics;
• backup and recovery;
• auditability.

Turn the Pattern Into a Catalog

Once the first service is stable, convert the architecture into repeatable catalog items and project templates.

This is where VCF Automation can become important. VMware describes VCF Private AI Foundation with NVIDIA as offering ready-to-use catalog items such as an AI workstation, AI Kubernetes cluster with GPU-capable worker nodes, and GPU-enabled Triton Inference Server through VCF Automation Quickstart.

The catalog should not be a free-for-all. It should expose standard patterns with approved defaults.

Operate, Optimize, and Govern

After the first few workloads go live, the platform team should start managing GPU utilization, endpoint sprawl, stale indexes, policy exceptions, model versions, and runtime cost.

This is where private AI becomes real operations work. The goal is not just to deploy more endpoints. The goal is to make the platform safer, more predictable, and easier to consume over time.

Private AI Reference Architecture Checklist

Before calling a VCF 9.1 AI deployment “production-ready,” I would expect the following checklist to be addressed.

If those answers are missing, the environment may still be useful. It is just not a mature private AI platform yet.

Conclusion: The Model Is the Smallest Part of Private AI

The easiest part of private AI is often getting a model to respond.

The harder part is making that response safe, governed, observable, repeatable, and supportable inside the enterprise operating model.

That is where VCF 9.1 becomes interesting. Broadcom is not only positioning it as a place to run AI workloads. The platform direction is about bringing AI into the same private cloud control model as VMs, Kubernetes, networking, identity, security, lifecycle, and operations.

That does not remove the architecture work. It changes where the architecture work belongs.

Private AI on VCF 9.1 should be designed as a platform service with clear sovereignty boundaries, identity controls, GPU consumption models, network segmentation, model governance, data access rules, and operational telemetry.

Running a model on-premises proves technical feasibility.

Operating private AI proves enterprise readiness.

External References

Broadcom Announces VMware Cloud Foundation 9.1
https://news.broadcom.com/releases/broadcom-announces-vmware-cloud-foundation-9-1

Streamline, Simplify, and Protect All Your AI Workloads with VCF 9.1
https://blogs.vmware.com/cloud-foundation/2026/05/05/streamline-simplify-and-protect-all-your-ai-workloads-with-vcf-9-1/

Platform Security in VCF 9.1
https://blogs.vmware.com/cloud-foundation/2026/05/05/platform-security-vcf-9-1/

OAuth 2.0 Authentication Using VMware Cloud Foundation PowerCLI 9.1
https://blogs.vmware.com/cloud-foundation/2026/06/01/oauth-2-0-authentication-using-vmware-cloud-foundation-powercli-9-1/

Deploying VMware Cloud Foundation Private AI Services: Navigating Supervisor Networking Stack
https://blogs.vmware.com/cloud-foundation/2026/06/11/deploying-vmware-cloud-foundation-private-ai-services-navigating-supervisor-networking-stack/

VCF Networking 9.1: Exploring Network Services for Virtual Private Clouds
https://blogs.vmware.com/cloud-foundation/2026/05/15/vcf-networking-9-1-exploring-network-services-for-virtual-private-clouds/

VCF Networking 9.1: Simpler VPC Connectivity Control
https://blogs.vmware.com/cloud-foundation/2026/05/15/vcf-networking-9-1-simpler-vpc-connectivity-control/

VCF 9.1 Networking: Precision Workload Placement with VPC Network Span
https://blogs.vmware.com/cloud-foundation/2026/06/09/vcf-9-1-networking-precision-workload-placement-with-vpc-network-span/

What’s New with vSphere 9.1
https://blogs.vmware.com/cloud-foundation/2026/05/12/whats-new-with-vsphere-9-1/

Generative AI with VMware Private AI Foundation with NVIDIA on VCF 9.0
https://blogs.vmware.com/cloud-foundation/2025/06/17/generative-ai-with-vmware-private-ai-foundation-with-nvidia-on-vcf-9-0/

Private AI Services: New in VMware Private AI Foundation with NVIDIA in VCF 9.0
https://blogs.vmware.com/cloud-foundation/2025/06/19/private-ai-services-new-in-vmware-private-ai-foundation-with-nvidia-in-vcf-9-0/

Install VMware Private AI Foundation with NVIDIA Using VCF Automation
https://blogs.vmware.com/cloud-foundation/2026/02/24/install-vmware-private-ai-foundation-with-nvidia-using-vcf-automation/

The post Private AI Is Not Model Hosting: A Reference Architecture for Sovereignty, Identity, GPUs, and Operations on VCF 9.1 appeared first on Digital Thought Disruption.