GPU Scheduling Is a Business Policy Problem: Designing NVIDIA Run:ai Quotas, Fairness, and Preemption

Introduction

A GPU cluster does not know which product launch is contractually committed, which research experiment can wait until tomorrow, or which inference endpoint supports a revenue-producing application. Kubernetes sees pods, resource requests, labels, and scheduling constraints. The business sees customers, deadlines, budgets, risk, and service commitments.

That gap is where many shared GPU platforms fail.

One organization solves the problem with fixed reservations. Every team receives a permanent block of GPUs, even when those GPUs sit idle. Another organization allows unrestricted sharing and assumes high utilization proves success. That approach looks efficient until a production service queues behind a large training run or a notebook user consumes scarce accelerators for days.

NVIDIA Run:ai provides the policy and orchestration mechanisms needed to manage shared AI infrastructure across departments, projects, users, workloads, and GPU pools. NVIDIA positions Run:ai as an enterprise platform for AI workloads and GPU orchestration, available through SaaS and self-hosted deployment models [1], [2]. The product provides the controls, but it cannot decide which business services deserve protection or which workloads may be interrupted. Those decisions must exist before the scheduler is configured.

The architectural position of this guide is direct: GPU allocation should be treated as an enterprise service policy, not merely a Kubernetes configuration.

For current product context, the NVIDIA Mission Control 2.2.0 administration guide describes Run:ai as the AI-specific workload orchestration and GPU resource-management layer in the platform stack [14]. NVIDIA Mission Control 2.3.1 release notes list NVIDIA Run:ai 2.25 in the updated software stack [15]. SaaS capabilities can move independently, so exact fields and feature availability should always be validated against the documentation for the selected deployment model.

TL;DR

A sustainable Run:ai design separates five decisions that are often collapsed into one quota number:

• Guaranteed quota defines the capacity a project should receive during contention.
• Over-quota access allows teams to borrow unused GPUs instead of leaving them stranded.
• Maximum consumption limits prevent a single project from consuming the entire pool.
• Fair-share scheduling distributes contested burst capacity across projects and departments instead of rewarding whichever job arrived first.
• Priority and preemptibility determine which workloads inside a project should start first and which borrowed workloads can be reclaimed safely.

Use whole GPUs as the default enterprise service. Offer MIG or fractional GPUs only through controlled exception services with clear workload qualifications. Measure queue time, completion time, useful GPU utilization, guarantee attainment, and preemption cost. Do not treat raw utilization as the only proof that the policy is working.

The Scheduler Cannot Invent Business Policy

Run:ai can translate policy into scheduling behavior, but the policy must come from the enterprise. The architecture team should define three layers before creating quotas or projects:

• Business entitlement: Which organizational unit receives a guaranteed service level, and why?
• Workload treatment: Which workload classes are protected, interruptible, time-limited, or eligible for fractional resources?
• Infrastructure eligibility: Which GPU generations, memory sizes, network fabrics, security zones, and node pools may execute each workload?

The diagram below shows the control chain. The important point is that no single scheduler setting represents the complete policy.

A platform team that starts with Kubernetes objects usually ends up debating YAML. A platform team that starts with service outcomes can explain why each project, priority class, and node pool exists.

Map Enterprise Structure to the Run:ai Hierarchy

Run:ai uses scopes and organizational constructs to connect users, workloads, and resources. Current documentation identifies account, cluster, department, and project scopes, with workloads running within projects and projects optionally grouped into departments [3], [4], [16].

The terms should not be treated as interchangeable.

A useful mapping is:

Enterprise account
  -> Department: business funding or governance owner
      -> Project: durable AI initiative or service boundary
          -> Users and groups: authorized participants
              -> Workloads: notebooks, training, fine-tuning, batch, and inference

Projects should be durable enough to accumulate meaningful quota, cost, and service metrics. They should also be narrow enough that workloads inside the project share a coherent business priority. When unrelated production inference and experimental research are placed in the same project, priority can order the workloads, but reporting, entitlement, and ownership become harder to defend.

Separate Guarantees, Burst Capacity, and Maximum Consumption

The most important Run:ai policy decision is not the quota value. It is the relationship among deserved quota, over-quota access, and the maximum GPU limit.

Guaranteed or Deserved Quota

A deserved quota represents the project’s protected resource entitlement during contention. It is not the same as permanently assigning physical GPUs to the project. When the project is idle, the platform can use that capacity elsewhere if over-quota consumption is enabled.

This distinction is what prevents fixed reservations from becoming stranded capacity.

A guarantee is also not absolute under every failure condition. If a 24-GPU pool loses four GPUs for maintenance or hardware failure, the scheduler cannot deliver 24 GPUs of guarantees from 20 physical GPUs. Production policy therefore needs a capacity reserve, failure-domain assumption, or degraded-service rule.

Over-Quota Borrowing

Over-quota access allows a project to consume unused resources beyond its deserved quota. Current Run:ai documentation describes fairness as the combination of deserved resources and a project’s share of unused resources within a node pool [5], [6]. Over-quota workloads are opportunistic. They should be designed so they can wait, checkpoint, restart, scale down, or be reclaimed.

Borrowing is not a policy defect. It is the mechanism that lets a shared platform remain productive when demand is uneven.

Maximum GPU Limits

A maximum GPU limit places a hard ceiling on the total capacity a project may consume in a node pool. In simplified planning terms:

Project allocation <= minimum of:
  - physical capacity available to the project
  - deserved quota plus fair share of unused capacity
  - configured maximum GPU limit

This is a planning model, not the scheduler’s full internal calculation. Its value is conceptual: the guarantee protects minimum service, borrowing improves utilization, and the maximum protects the rest of the platform.

A maximum of 14 GPUs does not guarantee that 14 GPUs will be available. It only says the project may consume up to 14 when fair-share entitlement and physical capacity allow it.

Why Neither Extreme Works

The third model is usually the right starting point because it recognizes both scarcity and variability.

Fair-Share Scheduling Is Not First-Come, First-Served

First-come, first-served scheduling rewards timing, not business importance or equitable access. A team can submit a large queue early and dominate the platform even if another team has a stronger service entitlement.

Run:ai calculates fair-share values for projects and departments per node pool. Current documentation also describes hierarchical queueing and optional time-based fair-share behavior that can consider GPU-hour consumption over time [5]. The operational objective is to allocate contested capacity according to entitlement and policy rather than arrival order alone.

This creates an important division of responsibility:

• Fairness determines allocation across projects and departments.
• Priority determines ordering among workloads inside the same project.
• Preemptibility determines whether a running workload may be interrupted.
• Maximum limits prevent fair-share borrowing from becoming unlimited consumption.

Run:ai documentation explicitly notes that workload priority applies within a project and does not replace fairness across projects [6]. This prevents a common anti-pattern in which every team labels its work “highest priority” and expects the label to override other projects’ guarantees.

A priority label is not an executive escalation mechanism. It is a scheduling instruction inside an already governed resource boundary.

Preemption Must Be a Contract, Not a Surprise

Priority and preemptibility are separate controls in current Run:ai scheduling behavior [7]. A high-priority workload may still be preemptible, and a lower-priority workload may be protected if policy requires it to be non-preemptible.

The business decision is not simply, “Can this pod restart?” The decision should include:

• How much work is lost when the workload stops?
• Does the framework checkpoint reliably?
• How long does restoration take?
• Does preemption create duplicate writes, inconsistent state, or abandoned external resources?
• Is the workload serving users or an automated pipeline?
• Is interruption allowed during all hours or only during declared contention windows?

Recommended Workload Treatment

NVIDIA’s current workload-type documentation assigns default categories and behavior to supported workload types, including build, train, and deploy mappings, but administrators can change mappings and policies [12]. Enterprise architecture should therefore define intentional workload classes rather than assume vendor defaults match internal service tiers.

The most valuable preemptible capacity usually comes from checkpointed training, hyperparameter trials, batch processing, and development inference. Fragile distributed jobs with long restart times may technically be restartable but economically poor preemption targets.

Whole GPU Should Be the Default Enterprise Service

Whole-GPU allocation should be the default service because it gives the platform a predictable unit for performance, capacity planning, troubleshooting, and chargeback. It also avoids treating every workload as a candidate for fine-grained sharing before its memory behavior, utilization profile, and latency sensitivity are understood.

MIG and fractional GPU allocation remain valuable, but they should be offered as qualified service variants.

Current Run:ai documentation states that GPU fractions and MIG mode cannot be used on the same node [10]. Current MIG profile guidance also states that Dynamic MIG has completed deprecation and that GPU or memory fractions are not supported with MIG profiles [11]. The practical implication is clear: build deliberate node pools for MIG and fractional services instead of attempting to mix every sharing mode on the same nodes.

A reasonable enterprise exception process should require:

• a measured GPU-memory profile;
• latency and throughput tests under contention;
• a declared minimum and maximum memory allocation;
• confirmation that the framework behaves correctly when co-located;
• an owner for performance complaints;
• a path back to whole-GPU service when isolation is insufficient.

Fractional allocation is a density tool. It is not a substitute for quota, fairness, or workload governance.

Node Pools Express Hardware and Risk Boundaries

Node pools translate business and technical eligibility into infrastructure placement. NVIDIA’s platform-administrator guidance describes node pools as a way to group worker nodes by hardware type, capability, or location and use those groups to protect scarce resources and production workloads [3], [9].

Useful node-pool boundaries include:

• GPU generation: H100, H200, B200, L40S, A100, or another supported class.
• GPU memory: 48 GB, 80 GB, 141 GB, or another service tier.
• Interconnect: single-node PCIe, NVLink, NVL72, Ethernet, RoCE, or InfiniBand eligibility.
• Security zone: regulated workloads, restricted datasets, or isolated tenant boundaries.
• Workload type: production inference, shared training, development, MIG, or fractional service.
• Topology requirement: nodes that can satisfy distributed placement or specialized fabric requirements.

Quotas, maximums, and fair-share behavior should be considered per node pool. A project with eight guaranteed GPUs in a shared H100 pool does not automatically have eight guaranteed GPUs in a restricted B200 pool.

Do not create a node pool for every team. Excessive pools fragment capacity and recreate fixed reservations under a different name. Create a pool when hardware capability, security boundary, topology, lifecycle, or service behavior genuinely differs.

Worked Example: Three Teams Sharing 24 GPUs

Consider a 24-GPU whole-GPU pool used by three departments or major projects:

• Customer AI: production inference and release-bound fine-tuning.
• Research: model training, hyperparameter optimization, and exploratory notebooks.
• Enterprise Data: embeddings, batch inference, RAG indexing, and analytics notebooks.

The pool is homogeneous for this example so the scheduling policy is easy to see. A production design may repeat the same policy structure across several node pools.

Policy Definition

The guaranteed quotas total 16 GPUs. The remaining eight GPUs are shared burst capacity. Guarantees are not physical reservations, so all 24 GPUs can be used when demand exists.

Normal Demand

At 10:00, consumption looks like this:

Research is using otherwise idle capacity. This is desirable. The platform should not force Research down to six GPUs merely to preserve an unused reservation for Customer AI.

Contention Event

At 10:15, the demand changes:

• Customer AI requests 12 GPUs for production inference and an approved release workload.
• Research still requests 12 GPUs.
• Enterprise Data requests 4 GPUs for an urgent indexing cycle.

Total demand is 28 GPUs, four more than the pool can supply.

The scheduler should first protect deserved quotas, then distribute the eight burst GPUs according to fair-share policy, over-quota weights, maximums, workload eligibility, and actual placement constraints. A reasonable whole-GPU outcome for the 3:2:1 weighting is:

Whole-GPU rounding, topology, gang scheduling, and the scheduler’s exact fair-share state may produce a different discrete allocation. The table is a policy demonstration, not a promise of exact internal arithmetic.

The contention flow should look like this:

What Gets Preempted

The cross-project allocation is determined by fairness, not by comparing every workload’s priority globally. After Research must return three borrowed GPUs, priority and preemptibility inside the Research project determine which workloads are interrupted.

A sound policy would reclaim three GPUs from low-priority, checkpointed hyperparameter trials before interrupting an active research notebook or a protected training run. The workload controller should checkpoint, stop or requeue the selected work, record the event, and verify that it resumes correctly.

What the Maximum Does Not Mean

Customer AI has a maximum of 14 GPUs, but it receives 12 in this scenario. The maximum only prevents consumption above 14. It does not guarantee 14.

If the business requires Customer AI to receive 12 GPUs under all normal contention, the architecture team must raise its deserved quota to 12, reduce other guarantees, add physical capacity, or place production inference in a separate production pool. Labeling the workload “very high priority” is not a valid substitute for capacity entitlement across projects.

That is the point at which GPU scheduling becomes a business-policy decision.

Measure Service Outcomes, Not Only GPU Utilization

A shared platform can report high GPU utilization while delivering poor service. A large queue of low-value training jobs can keep GPUs busy as production releases miss deadlines.

The operating dashboard should include at least the following metrics.

NVIDIA Run:ai provides metrics and telemetry across physical resources and organizational entities such as clusters, node pools, departments, projects, and workloads [13]. The enterprise should add service-level calculations that reflect its own objectives.

Useful SLO examples include:

• p95 queue time for an interactive notebook is less than 10 minutes during business hours;
• production inference replicas obtain guaranteed capacity during a declared contention test;
• checkpointed training resumes within 15 minutes after preemption;
• less than 5 percent of monthly training GPU-hours are lost to preemption rollback;
• no project remains below deserved quota for more than the defined recovery interval when the pool is healthy;
• burst capacity remains available to at least two departments over a rolling month rather than being monopolized by one queue.

These metrics expose whether fairness is producing useful outcomes, not merely busy hardware.

Governance Questions to Answer Before Configuration

The scheduler configuration should not begin until the organization can answer the following questions.

The answers should be published as a service policy that users can understand. Hidden scheduler behavior creates conflict because teams cannot distinguish a platform defect from an intentional governance decision.

Translate the Policy into Run:ai Configuration

A practical implementation sequence is:

1. Inventory workloads by business owner, workload class, GPU count, GPU memory, runtime, checkpoint behavior, and service objective.
2. Define the smallest useful set of node pools based on hardware, topology, security, and service behavior.
3. Map durable business units to departments and durable initiatives or services to projects.
4. Set deserved quotas per project and node pool using measured demand and business entitlement.
5. Enable over-quota consumption for workloads that can safely use opportunistic capacity.
6. Set over-quota weights and maximum GPU limits to prevent monopolization.
7. Define workload priority and preemptibility mappings inside each project.
8. Apply idle-time, duration, placement, and workload policies.
9. Test contention, preemption, recovery, quota attainment, and degraded capacity before production onboarding.
10. Review metrics monthly and revise the service policy through change control.

The following YAML is an illustrative governance artifact. It is not a Run:ai API schema and should not be submitted directly. Its purpose is to capture the decisions that must later be implemented through the supported Run:ai UI, API, CLI, or infrastructure-as-code workflow for the selected version.

servicePolicy:
  nodePool: shared-whole-gpu
  physicalCapacityGpus: 24
  allocationUnit: whole-gpu

  projects:
    customer-ai:
      guaranteedGpus: 8
      maximumGpus: 14
      overQuotaWeight: 3
      workloadClasses:
        production-inference:
          priority: very-high
          preemptible: false
        release-finetuning:
          priority: high
          preemptible: true

    research:
      guaranteedGpus: 6
      maximumGpus: 14
      overQuotaWeight: 2
      workloadClasses:
        interactive-notebook:
          priority: high
          preemptible: false
          idleTimeoutMinutes: 30
        checkpointed-training:
          priority: normal
          preemptible: true

    enterprise-data:
      guaranteedGpus: 2
      maximumGpus: 6
      overQuotaWeight: 1
      workloadClasses:
        batch-embedding:
          priority: low
          preemptible: true
        development-inference:
          priority: normal
          preemptible: true

A successful implementation should demonstrate that the policy artifact, Run:ai configuration, operational dashboards, and user documentation all describe the same service behavior.

Common Policy Failures

Treating Quota as a Physical Reservation

This leaves GPUs idle when the entitled project has no demand. Use deserved quota with over-quota borrowing unless physical isolation is a real requirement.

Treating the Maximum as a Guarantee

A maximum limits consumption. It does not reserve capacity. Guaranteed service must be represented by deserved quota and supported by sufficient physical capacity.

Using Priority to Override Other Projects

Priority does not replace cross-project fairness. If a service requires more protected capacity, change its guarantee or node-pool design.

Making Everything Non-Preemptible

The platform loses its ability to reclaim borrowed resources. Require evidence before granting non-preemptible status, especially for long-running work.

Enabling Fractions Everywhere

Fractional allocation increases density but can create noisy-neighbor behavior, confusing metrics, and unpredictable latency. Qualify workloads and keep whole GPU as the default.

Creating Too Many Node Pools

Small pools fragment capacity and reduce the value of fair sharing. Create pools for meaningful capability or risk differences, not organizational convenience.

Measuring Only Utilization

High utilization can coexist with long queues, missed deadlines, and repeated preemption loss. Measure service outcomes and completed useful work.

Ignoring Maintenance and Failure Capacity

A quota plan that consumes 100 percent of physical capacity as guarantees has no room for host failure, upgrades, fragmentation, or workload placement constraints.

Conclusion

NVIDIA Run:ai can turn a shared GPU platform into a governed enterprise service, but only when the organization defines what the scheduler is expected to protect, share, limit, and reclaim.

The durable model separates deserved quota, over-quota borrowing, maximum consumption, fair-share allocation, workload priority, and preemptibility. Each control solves a different problem. Combining them creates a platform that can preserve service levels without leaving expensive accelerators stranded.

Whole GPUs should remain the default service unit because they simplify performance expectations, accounting, and operations. MIG and fractional GPU services should be offered deliberately through qualified node pools and documented exception criteria. Production inference, interactive development, checkpointed training, and batch processing should not receive identical treatment simply because they all request a GPU.

The final test is operational. During a controlled contention event, can the platform prove that guarantees are honored, burst capacity is shared fairly, low-value borrowed work is reclaimed safely, production services remain stable, and interrupted workloads recover within policy? When the answer is measurable and repeatable, GPU scheduling has moved beyond Kubernetes configuration and become an enterprise business policy.

External References

[1] NVIDIA: Accelerate AI & Machine Learning Workflows | NVIDIA Run:ai
Canonical URL: https://www.nvidia.com/en-us/software/run-ai/

[2] NVIDIA: NVIDIA Run:ai Documentation
Canonical URL: https://docs.nvidia.com/run-ai/index.html

[3] NVIDIA: Quick Start for Platform Administrators
Canonical URL: https://run-ai-docs.nvidia.com/saas/getting-started/quick-starts/platform-admin-quick-start

[4] NVIDIA: Adapting AI Initiatives to Your Organization
Canonical URL: https://run-ai-docs.nvidia.com/saas/platform-management/aiinitiatives/adapting-ai-initiatives

[5] NVIDIA: The NVIDIA Run:ai Scheduler: Concepts and Principles
Canonical URL: https://run-ai-docs.nvidia.com/saas/platform-management/runai-scheduler/scheduling/concepts-and-principles

[6] NVIDIA: How the Scheduler Works
Canonical URL: https://run-ai-docs.nvidia.com/saas/platform-management/runai-scheduler/scheduling/how-the-scheduler-works

[7] NVIDIA: Workload Priority and Preemption
Canonical URL: https://run-ai-docs.nvidia.com/saas/platform-management/runai-scheduler/scheduling/workload-priority-control

[8] NVIDIA: Projects
Canonical URL: https://run-ai-docs.nvidia.com/saas/platform-management/aiinitiatives/organization/projects

[9] NVIDIA: Node Pools
Canonical URL: https://run-ai-docs.nvidia.com/saas/platform-management/aiinitiatives/resources/node-pools

[10] NVIDIA: GPU Fractions
Canonical URL: https://run-ai-docs.nvidia.com/saas/platform-management/runai-scheduler/resource-optimization/fractions

[11] NVIDIA: Configuring NVIDIA MIG Profiles
Canonical URL: https://run-ai-docs.nvidia.com/saas/platform-management/aiinitiatives/resources/mig-profiles

[12] NVIDIA: Supported Workload Types
Canonical URL: https://run-ai-docs.nvidia.com/saas/workloads-in-nvidia-run-ai/workload-types/supported-workload-types

[13] NVIDIA: Metrics and Telemetry
Canonical URL: https://run-ai-docs.nvidia.com/saas/platform-management/monitor-performance/metrics

[14] NVIDIA: NVIDIA Mission Control Software Administration Guide 2.2.0 – Overview
Canonical URL: https://docs.nvidia.com/mission-control/docs/systems-administration-guide/2.2.0/overview.html

[15] NVIDIA: NVIDIA Mission Control 2.3.1 Release Notes
Canonical URL: https://docs.nvidia.com/mission-control/docs/systems-quick-start-guide/2.3.1/nmc-release-notes.html

[16] NVIDIA: Authentication and Authorization
Canonical URL: https://run-ai-docs.nvidia.com/saas/infrastructure-setup/authentication/overview

The post GPU Scheduling Is a Business Policy Problem: Designing NVIDIA Run:ai Quotas, Fairness, and Preemption appeared first on Digital Thought Disruption.