GPUs Are Not a Cloud: Why Neoclouds Need Vendor Neutral AI Infrastructure Orchestration

TL;DR

Neoclouds may begin by selling access to scarce GPU capacity, but long-term differentiation requires more than racks, drivers, and a booking portal. A production AI cloud must convert bare-metal servers, virtual machines, GPU pools, storage, networks, and external cloud resources into secure, repeatable, tenant-aware services.

The missing layer is vendor-neutral AI infrastructure orchestration. It should accept a customer or platform intent, select an appropriate substrate, provision Kubernetes and supporting services, enforce isolation, manage accelerator lifecycle and placement, collect operational evidence, and retire the environment safely. The goal is not to hide every hardware difference. The goal is to create a portable control plane over a hardware-aware data plane.

Introduction

A neocloud can own thousands of high-value accelerators and still fail to operate like a cloud.

Hardware inventory answers one question: what capacity exists? A cloud platform must answer a much larger set of questions. Who is allowed to consume that capacity? How quickly can a secure environment be created? Which tenant is responsible for the workload? What data and network boundaries apply? Which GPU sharing model is permitted? How are upgrades, failures, metering, recovery, and evidence handled? What happens when a workload must move to a different hardware generation, site, or provider?

These questions become more important as neoclouds move beyond large, dedicated training reservations and begin serving enterprise inference, regulated workloads, internal AI platforms, and sovereign AI programs. A long-running enterprise service is not a one-time allocation. It is a lifecycle commitment.

That is why the next competitive boundary for neoclouds is orchestration.

The useful architecture is not a thin portal placed in front of GPU servers. It is software that can turn bare metal, accelerators, virtual machines, and cloud capacity into governed Kubernetes and AI platform services without forcing the provider or customer into one hardware vendor’s operating model.

A Neocloud Must Sell a Platform, Not Just Capacity

A specialized GPU provider can create early value through availability, price, and performance. Those advantages matter, but they are difficult to defend forever. Hardware supply changes. New accelerator vendors emerge. Hyperscalers adjust pricing. Customers move from training projects to persistent inference services. Enterprise buyers expect identity integration, service levels, audit evidence, support boundaries, and predictable lifecycle management.

The operating model must therefore evolve from a resource business into a platform business.

The difference is not cosmetic. A portal can make a manual process look modern. Orchestration changes the process itself.

A real orchestration layer continuously compares desired state with actual state. It notices failed provisioning, unhealthy GPUs, drifted configuration, missing policy, exhausted quotas, broken dependencies, or unsupported combinations. It then corrects the condition, blocks the request, or escalates with enough evidence for an operator to act.

Scope and Assumptions

This article focuses on neoclouds and enterprise platform teams that need to deliver Kubernetes and AI services across a mixture of bare-metal servers, virtual machines, accelerator pools, and public or partner cloud resources.

The architecture assumes the following:

• Kubernetes is a primary platform API and workload substrate, but it is not the entire cloud.
• Some customers can share infrastructure safely, while others require a dedicated control plane, cluster, or physical host.
• Accelerator types, interconnects, drivers, firmware, and sharing capabilities will remain heterogeneous.
• Private AI and sovereign AI are service-placement and control models, not simply product labels.
• Vendor neutrality means replaceable integrations and portable contracts, not pretending that all hardware behaves identically.
• The provider is responsible for day-2 operations, including upgrades, observability, incident response, recovery, and evidence retention.

The goal is not to prescribe one product stack. It is to define the capabilities the orchestration layer must provide and the operating decisions it must make visible.

The Orchestration Layer at a Glance

The important point in the following diagram is where the abstraction boundary sits. Customers should request a service outcome. The orchestration layer should translate that outcome into infrastructure-specific actions while preserving the constraints that make the workload secure and operable.

This architecture treats orchestration as a provider control plane, not as a single scheduler. Kubernetes scheduling is one component. Cluster creation, hardware preparation, tenant isolation, GPU software lifecycle, workload admission, policy enforcement, telemetry, billing context, and recovery all sit around it.

Orchestration Is More Than Scheduling

A scheduler decides where an admitted workload should run. An orchestration platform decides whether the environment should exist, how it should be built, which controls apply, how it changes over time, and what evidence proves that the service is healthy.

This distinction prevents a common architecture mistake: installing Kubernetes, adding a GPU device plugin, and declaring the result to be an AI cloud.

That stack may run a workload. It does not yet provide a governed service.

The Six Planes a Neocloud Must Unify

A useful orchestration architecture can be understood as six connected planes. These planes can be implemented by several products and controllers, but they must behave as one service system.

Capacity and Infrastructure Plane

The capacity plane turns physical and virtual resources into trustworthy inventory. For bare metal, that includes out-of-band management, hardware inspection, firmware and BIOS state, RAID configuration, disk cleaning, operating-system imaging, network attachment, and health remediation. For virtual machines and external cloud instances, it includes image selection, instance lifecycle, networking, storage, quotas, and provider-specific failure handling.

This plane must know more than the number of GPUs in a rack. It needs CPU architecture, system memory, NUMA layout, local NVMe, NIC type, fabric location, power state, firmware compatibility, accelerator health, and maintenance status. A distributed training job may require eight healthy GPUs in one fabric domain, not eight arbitrary devices spread across a region.

The orchestration layer should expose this infrastructure through normalized capability classes while retaining the topology and compatibility data needed for correct placement.

Cluster and Runtime Plane

The cluster plane creates and maintains Kubernetes control planes, worker pools, networking, storage integrations, ingress, registries, secrets services, policy engines, and baseline observability.

Cluster lifecycle must be declarative. A provider should be able to state that a tenant needs a dedicated production cluster in a specific region with an approved Kubernetes channel, GPU-enabled worker pools, default-deny networking, backup, and a validated add-on bundle. Controllers then reconcile the environment and report whether it is ready.

This plane also owns upgrades, conformance testing, drift detection, certificate rotation, control-plane backup, worker replacement, and add-on compatibility. Without that lifecycle discipline, each customer environment becomes a snowflake that grows harder to patch and support.

Tenant and Security Plane

Kubernetes provides useful isolation primitives, but it does not provide a complete tenant model by itself. A neocloud must define the tenant as a first-class platform object and carry that identity through infrastructure, clusters, namespaces, networks, storage, secrets, telemetry, support, and billing.

The tenant plane decides:

• whether the customer receives a namespace, virtual control plane, dedicated cluster, or dedicated hardware
• which identity provider and roles apply
• which networks and endpoints are reachable
• whether egress is denied, inspected, or allow-listed
• where secrets and encryption keys live
• which images, models, and registries are approved
• which administrators can use break-glass access
• how tenant data, logs, snapshots, and deleted disks are handled

Isolation is not one switch. It is a bundle of boundaries with different strengths and costs.

Accelerator and AI Workload Plane

GPU operations are a lifecycle problem. Drivers, container runtimes, device plugins, health agents, firmware, partitioning modes, topology labels, telemetry, and scheduler integrations must remain compatible.

The orchestration layer should translate customer intent into an accelerator service class. A request might need exclusive devices, hardware-partitioned instances, time-shared access, a specific memory floor, a same-rack topology, or preemptible capacity. The platform should validate that the requested combination is supported before the workload is admitted.

For batch, training, and HPC-style AI jobs, queueing and quota policy are as important as node placement. The platform needs admission control, priority, borrowing, preemption, gang behavior, topology-aware placement, and clear queue-time objectives. For inference, it also needs replica placement, autoscaling, model-loading behavior, cold-start measurement, and service latency objectives.

AI Services and Delivery Plane

Enterprise customers rarely want raw Kubernetes alone. They may need managed notebooks, training jobs, distributed frameworks, model registries, inference endpoints, data connectors, vector services, model gateways, evaluation pipelines, or approved runtime images.

The orchestration layer should package these capabilities as versioned service components. Each component needs an owner, support boundary, upgrade path, security baseline, observability contract, and rollback method.

This is where a neocloud becomes more than GPU infrastructure. It creates repeatable platform outcomes that customers can consume without rebuilding the same AI stack for every project.

Evidence and Economics Plane

A provider cannot govern or price what it cannot attribute.

Every resource and event should carry tenant, environment, service class, workload, region, hardware pool, and owner context. Metrics, logs, traces, audit records, queue events, GPU health, network performance, storage behavior, and lifecycle actions must be correlated.

The same context should support showback, chargeback, capacity planning, incident review, and service-level reporting. A GPU-hour is not a sufficient economic unit when the service also reserves high-speed fabric, local storage, object storage, CPU, memory, power, software subscriptions, and operational support.

A stronger unit is cost per useful service outcome, such as completed training run, million inference requests, successful batch, protected endpoint, or tenant service class.

Treat the Platform as a Compiler for Infrastructure Intent

The cleanest mental model is to treat the orchestration layer as a compiler.

A customer or platform team submits intent. The platform validates policy, resolves the request into infrastructure and service components, executes the plan through controllers, and produces both a running environment and an evidence record.

This model has an important operational benefit. Failure becomes structured. Instead of returning a generic provisioning error, the platform can report that the requested topology is unavailable, the selected GPU partitioning mode conflicts with the isolation class, the region lacks approved storage, or the tenant has exceeded reserved capacity.

The customer receives a reason and a supported alternative. The operator receives the controller state and evidence needed to diagnose the condition.

A Practical AI Environment Service Contract

The following YAML is a conceptual platform contract, not a standard Kubernetes resource that can be installed as written. It demonstrates the level of intent a neocloud should capture before selecting servers or creating a cluster.

apiVersion: platform.dtd.example/v1alpha1
kind: AIEnvironment
metadata:
  name: claims-inference-prod
  namespace: tenant-acme
spec:
  serviceClass: regulated-inference

  placement:
    jurisdiction: eu
    allowedSubstrates:
      - bareMetal
      - privateVirtualization
    externalBursting: denied

  isolation:
    cluster: dedicated
    hardware: sharedWithProtectedSlices
    network: defaultDeny
    tenantManagedKeys: required

  accelerator:
    capabilityClass: high-memory
    count: 8
    allocation: exclusive
    topology: same-fabric-domain

  kubernetes:
    releaseChannel: stable
    lifecycleWindow: monthly
    controlPlaneBackup: daily

  data:
    residency: eu-only
    approvedStorageClass: encrypted-performance
    modelRegistry: private

  observability:
    tenantMetrics: enabled
    workloadTracing: enabled
    auditRetentionDays: 365

  recovery:
    workloadRpoMinutes: 60
    environmentRebuildTargetMinutes: 120

The reader should notice that the request does not name a server model, cloud instance, or GPU vendor. It expresses the capabilities and controls the workload requires. Provider-specific adapters remain free to map that contract to supported hardware.

The platform implementation must add several things around this object:

• admission rules that reject unsupported or noncompliant combinations
• controllers that provision infrastructure, clusters, policies, and AI services
• status conditions that report capacity, health, security, and readiness
• a versioned service-class catalog that limits arbitrary combinations
• rollback and cleanup behavior for partially completed requests
• evidence that records what was selected and which policies were applied

Successful execution should produce more than a Ready state. It should prove that the correct region, isolation tier, GPU topology, network policy, storage class, observability bundle, and recovery controls were validated.

Multi-Tenancy Is the Product Boundary

The most important neocloud design decision may be the isolation unit offered to customers.

Sharing increases utilization and can reduce cost, but it also expands the number of components that must enforce fairness and security. Dedicated environments cost more, but they simplify several trust and performance questions. The orchestration layer should make this an explicit service-class choice rather than an informal engineering decision.

Namespaces remain useful, but they are not a complete security boundary. Cluster-scoped objects, shared nodes, kernel exposure, storage systems, network plugins, privileged workloads, and administrative access all affect the real isolation level.

A provider should therefore define an isolation profile that includes at least:

• control-plane boundary
• worker-node boundary
• network and egress policy
• storage and backup boundary
• secrets and key ownership
• privileged workload policy
• accelerator sharing mode
• observability and log visibility
• operator access and break-glass process
• deletion, disk cleaning, and evidence-retention behavior

The commercial catalog should price these differences. A dedicated cluster with exclusive GPUs and tenant-owned keys is a different service from a shared development namespace using time-sliced accelerators.

GPU Management Must Become a Service-Class Problem

A neocloud cannot treat every GPU request as count: 1.

Accelerator requirements include memory, supported numeric formats, interconnect, CPU and NUMA locality, network fabric, storage throughput, driver branch, runtime compatibility, partitioning capability, and failure behavior. The orchestration layer must preserve those requirements while still presenting a usable service catalog.

The platform should separate four decisions that are often collapsed:

• Eligibility: Which accelerator families can run the workload?
• Allocation: Does the workload receive a whole device, partition, or shared slot?
• Admission: When is the job allowed to consume capacity, and can it borrow or preempt?
• Placement: Which host, rack, fabric domain, site, or provider can meet the topology and service objective?

Kubernetes device frameworks can expose hardware, and queueing systems can govern admission. The neocloud orchestration layer must connect those mechanisms to tenant contracts, hardware health, lifecycle compatibility, and commercial policy.

Sovereign AI Requires More Than Local Hardware

A provider can place GPUs inside a national boundary and still deliver a weak sovereign AI service.

Sovereignty depends on who can operate, change, inspect, recover, and exit the platform. A credible sovereign AI service class should define:

• where data, prompts, models, embeddings, logs, backups, and support artifacts may reside
• which legal entities and operators can access the environment
• who controls identity roots, encryption keys, certificates, and break-glass credentials
• where container images, model artifacts, packages, and updates originate
• who can approve platform upgrades and emergency changes
• whether telemetry leaves the jurisdiction
• how remote vendor support is brokered and recorded
• how the customer can export workloads, data, evidence, and configuration
• how the service can be restored when an external dependency is unavailable

This is another reason orchestration must sit above individual hardware and cloud providers. Private and sovereign environments should be expressed as policy and placement classes that can be implemented on approved substrates.

The strongest design does not guess one permanent destination for every AI workload. It preserves the ability to place workloads on premises, in a regional neocloud, in a private environment, or across approved clouds while maintaining reproducible operations and visible control boundaries.

Hardware Neutrality Must Preserve Hardware Awareness

Vendor neutrality is frequently misunderstood as removing vendor-specific information. That approach creates a lowest-common-denominator platform and can damage performance, reliability, and supportability.

The better pattern is a portable control plane over a hardware-aware data plane.

The service contract should use capability language such as:

• high-memory accelerator
• low-latency fabric domain
• hardware-isolated accelerator slice
• confidential-compute capable node
• high-throughput inference class
• distributed-training topology class
• approved sovereign supply-chain profile

Adapters then map those capabilities to supported combinations of servers, accelerators, NICs, storage, drivers, and runtimes. The mapping should be versioned and tested.

Customers gain portability because they request capabilities rather than vendor SKUs. Operators retain accuracy because the platform still knows which implementations satisfy the contract.

This design also creates a practical exit strategy. Replacing a server vendor, introducing a new accelerator, or adding another cloud provider should require a new adapter and validation profile, not a rewrite of every customer workflow.

The Operating Model Behind the Platform

Orchestration does not remove ownership. It makes ownership executable.

A frequent failure pattern is assigning all of this to a small Kubernetes team. That team may understand clusters, but it cannot alone own firmware, physical networking, customer contracts, sovereignty, AI runtime compatibility, security exceptions, and service economics.

The orchestration layer should reflect these boundaries through APIs, approvals, policy ownership, dashboards, and escalation paths.

A Phased Implementation Path

The platform should be built from the infrastructure inward. Starting with a polished self-service portal creates demand before the backend can deliver safely.

Each phase should include failure testing. Provisioning success in a clean lab is not enough. Test exhausted IP pools, unavailable topology, unhealthy GPUs, failed image pulls, expired certificates, lost management connectivity, partial cluster creation, storage latency, policy rejection, and cleanup after aborted requests.

A platform becomes trustworthy when failure is expected, bounded, observable, and recoverable.

What Neoclouds Should Measure

GPU utilization alone cannot show whether the platform is delivering a good service.

Two simple ratios are especially useful:

Useful Accelerator Efficiency = Useful Workload Time
                                --------------------
                                Allocated GPU Time

Provisioning Reliability = Successfully Ready Environments
                           -------------------------------
                           Total Accepted Requests

These ratios should be interpreted by service class. A reserved training cluster and a shared development pool should not have the same utilization target or queue objective.

Common Failure Modes and Caveats

Treating Kubernetes as the Complete Cloud

Kubernetes orchestrates workloads and exposes powerful APIs, but it does not automatically provide tenant contracts, bare-metal lifecycle, sovereign controls, service economics, customer support, or recovery ownership. Those capabilities must be designed around it.

Using Namespaces as the Default Answer for Every Tenant

Shared clusters can be efficient, but they require strong policy, network, storage, runtime, and administrative controls. Higher-risk tenants may need a dedicated control plane, cluster, or host. Isolation should be selected from workload and trust requirements, not from a desire to maximize density.

Hiding Hardware Differences Too Aggressively

Accelerators and fabrics differ in memory, features, topology, drivers, runtime support, and performance. A generic gpu label is not enough. Portability should come from capability contracts and adapters, not from discarding information the scheduler and operator need.

Automating Provisioning but Not Lifecycle

Day-0 automation creates an environment. Day-2 orchestration patches, rotates certificates, replaces failed nodes, upgrades operators, validates compatibility, backs up control planes, recovers service, and retires resources safely. The latter determines whether the cloud remains supportable.

Building Self-Service Before Guardrails

A portal without service classes, quotas, policy, and backend reconciliation accelerates sprawl. Self-service should expose supported outcomes, not every infrastructure knob.

Measuring Infrastructure Without Tenant Context

Cluster-wide metrics can show that a GPU is busy without showing which customer, workload, queue, service objective, or commercial commitment is affected. Tenant and service-class context must be attached at ingestion.

Calling a Region Sovereign Without Testing Control

Local placement is only one sovereignty input. Identity, keys, operators, software supply chain, telemetry, support access, recovery, and exit all matter. The service must prove those controls through architecture and evidence.

Centralizing the Control Plane Without Designing Its Failure Domain

A fleet-wide orchestration layer can become a high-impact dependency. Separate management and workload planes, protect credentials, back up desired state, test controller failure, limit blast radius, and define degraded operating modes.

A Decision Framework for Selecting an Orchestration Platform

A neocloud should evaluate orchestration software against operating evidence, not a feature checklist or demonstration.

The best platform is not the one that claims to abstract the most. It is the one that turns real service intent into a reliable environment while preserving the information needed for performance, security, support, and change.

Conclusion

Neoclouds were able to enter the market by solving an urgent capacity problem. The next stage is harder. Enterprise and sovereign AI customers need repeatable services, not just access to accelerators.

That requires an orchestration layer that treats bare metal, virtual machines, cloud resources, Kubernetes clusters, GPU pools, policies, and AI services as parts of one reconciled system. It must provision environments, isolate tenants, govern workload admission, preserve topology, manage lifecycle, collect evidence, and connect consumption to service economics.

Kubernetes is a strong foundation because it provides declarative APIs, extensibility, workload orchestration, and a broad ecosystem. It is not sufficient by itself. The neocloud must add the tenant model, service catalog, policy system, hardware lifecycle, accelerator governance, observability, recovery, and commercial context that turn infrastructure into a cloud platform.

The durable architectural position is straightforward: build a portable control plane over a hardware-aware data plane. That lets the neocloud introduce new servers, accelerators, sites, and cloud partners without rewriting the customer experience, while still respecting the differences that determine whether AI workloads perform and remain supportable.

GPUs are valuable capacity. Orchestration is what turns that capacity into a business.

External References

• Cloud Native Computing Foundation: CNCF Cloud Native AI White Paper
  Canonical URL: https://tag-runtime.cncf.io/wgs/cnaiwg/whitepapers/cloudnativeai/
• Kubernetes SIG Cluster Lifecycle: Concepts – The Cluster API Book
  Canonical URL: https://cluster-api.sigs.k8s.io/user/concepts
• Metal3: Bare Metal Operator – Metal3 User Guide
  Canonical URL: https://book.metal3.io/bmo/introduction.html
• Kubernetes: Multi-tenancy
  Canonical URL: https://kubernetes.io/docs/concepts/security/multi-tenancy/
• Kubernetes: Dynamic Resource Allocation
  Canonical URL: https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/
• Kubernetes SIG Scheduling: Kueue – Kubernetes-native Job Queueing
  Canonical URL: https://kueue.sigs.k8s.io/
• NVIDIA: About the NVIDIA GPU Operator
  Canonical URL: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html
• NVIDIA: Time-Slicing GPUs in Kubernetes
  Canonical URL: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html
• KubeVirt: Architecture – KubeVirt User Guide
  Canonical URL: https://kubevirt.io/user-guide/architecture/
• OpenTelemetry: Documentation
  Canonical URL: https://opentelemetry.io/docs/
• Cloud Native Computing Foundation: Where Should AI Workloads Run? A Sovereign and Sensible Approach
  Canonical URL: https://www.cncf.io/blog/2026/07/10/where-should-ai-workloads-run-a-sovereign-and-sensible-approach/
• Nutanix: The Rapid Rise and Future of Neoclouds
  Canonical URL: https://www.nutanix.com/theforecastbynutanix/technology/what-is-next-for-neoclouds

The post GPUs Are Not a Cloud: Why Neoclouds Need Vendor Neutral AI Infrastructure Orchestration appeared first on Digital Thought Disruption.