GPU & NVIDIA Support

The host detail page showing the NVIDIA driver and container toolkit deployment toggles, plus the detected GPU inventory.

Trailer.dev has two separate GPU concerns that are easy to confuse:

• Host-level NVIDIA deployment: the agent can deploy the NVIDIA driver and the NVIDIA Container Toolkit into the host’s Docker, so containers can use the GPU at all.
• Per-workspace acceleration: a single workspace opts in to GPU sharing, DRI hardware acceleration, nested virtualization, or (for Windows VDI) full GPU passthrough.

The first is configured once per host. The second is configured per workspace.

Host-level NVIDIA driver and toolkit

The agent can deploy two host-scoped containers:

• The NVIDIA driver (DeployDriver). Runs the nvidia-driver init entrypoint as a privileged container with a restart policy. It mounts /run/nvidia (shared propagation), /lib/firmware, and /var/log so the driver it installs is visible to the rest of the host.
• The NVIDIA Container Toolkit (DeployToolkit). Runs nvidia-toolkit, installs the nvidia Docker runtime into the host’s /etc/docker, restarts Docker so the runtime is picked up, then stays running.

Both are configured per host:

Setting	Meaning	Default
Deploy driver	Deploy the NVIDIA driver container	off
Deploy toolkit	Deploy the NVIDIA Container Toolkit container	off
Driver image	NVIDIA driver container image	nvcr.io/nvidia/driver:580.126.16
Toolkit image	NVIDIA Container Toolkit image	nvcr.io/nvidia/k8s/container-toolkit:v1.17.8-ubuntu20.04

These settings live on the host and are delivered to the agent. On each reconcile the agent prunes whichever of the two is disabled and deploys (or upgrades) whichever is enabled. If the running container’s image does not match the desired version, the stale one is removed and the new version deployed.

Match the driver image tag to your host OS

NVIDIA publishes the driver container per operating system, so the image tag ends with the host’s OS name and version. A version-only tag will not resolve on its own: you append the OS name and version that match the host. For example, on Ubuntu 24.04 the driver image is:

nvcr.io/nvidia/driver:580.126.16-ubuntu24.04

Trailer detects and shows the host’s operating system and version on the host page, so you can pick the tag that matches. The container toolkit image uses the same scheme (its default already ends in -ubuntu20.04).

When the toolkit is (re)deployed the nvidia runtime becomes available, so the agent restarts itself to pick it up. This is what grants the agent GPU access, so you do not need to launch the agent (or standalone) container with --gpus all. Host-level deployment is Linux only.

flowchart TD
  A[Host NVIDIA config] --> B{deployDriver?}
  B -- yes --> C[Run nvidia-driver init container]
  B -- no --> D[Remove driver container]
  A --> E{deployToolkit?}
  E -- yes --> F[Run nvidia-toolkit, install nvidia runtime, restart Docker]
  E -- no --> G[Remove toolkit container]
  F --> H[Agent restarts to pick up nvidia runtime]

The driver container is optional. If an operator already manages the NVIDIA driver on the host, enable only the toolkit. The toolkit is always pointed at /run/nvidia/driver (via NVIDIA_DRIVER_ROOT and DRIVER_ROOT_CTR_PATH), so an operator-managed driver must be reachable there.

GPU inventory and metrics

Independent of whether Trailer deployed the driver, the agent reports the host’s NVIDIA GPUs on every heartbeat. Two data sources are merged:

• A scan of /sys/bus/pci/devices for NVIDIA display-class devices. This is the authoritative device list and works even with no driver loaded.
• nvidia-smi -q -x output, when a driver is present. This enriches each device with product name, temperature, utilization, memory, power, and virtualization mode. nvidia-smi is run on the host, or, if that fails, inside the deployed driver container.

For each GPU the agent records its PCI address (BDF), PCI device ID, bound kernel driver, IOMMU group, and (when nvidia-smi is available) UUID and live metrics. GPU metrics are collected on every heartbeat regardless of configuration. The CollectGpuMetrics dynamic config flag only controls whether the server persists the time-series metrics. See Resource monitoring for how metrics surface in the UI.

Choosing a CUDA version for image builds

When you build a GPU-enabled container image, you pick a CUDA version. That choice determines which CUDA runtime the image’s packages (for example pytorch-gpu or a vLLM build) are compiled against. Leave it unset for a CPU-only build.

The CUDA version only affects conda packages. It tells the conda solver that CUDA is available at the version you chose, so conda-forge selects the matching GPU builds. PyPI packages are not affected by this setting. If you need a GPU-enabled package that only comes from PyPI, request the appropriate PyPI package or wheel directly instead of relying on the CUDA version.

The builder shows the available CUDA versions in a dropdown, color-coded against the GPU driver on the build host you are looking at, with a legend. The colors are a quick read of whether the resulting image will run on that driver, and how well.

What compatibility means

NVIDIA drivers are backward compatible, and always have been. An image built for a given CUDA version runs on any host whose GPU driver supports that version or a newer one. A newer driver happily runs applications built against an older CUDA toolkit.

Since CUDA 11, minor version compatibility adds a second guarantee within a single major series. An image built with a newer minor toolkit runs on an older driver in the same major series, as long as the driver meets the family’s minimum:

CUDA major series	Minimum GPU driver
13.x	580 or newer
12.x	525 or newer
11.x	450 or newer

A newer minor than your driver buys you nothing

Minor version compatibility lets the image load, but any feature or API introduced after the driver’s own version is unavailable and fails at runtime. NVIDIA returns a “call requires a newer driver” style error. This is automatic and needs no extra package, but building a newer minor than the runtime host’s driver gives you no usable gain.

Crossing a major series is different. Running a newer-major-series image on an older-major driver needs NVIDIA’s separate forward compatibility path (a dedicated CUDA compatibility package). It is supported only on NVIDIA Data Center GPUs, select NGC-Ready RTX cards, and Jetson boards. It is not supported on consumer GeForce GPUs (for example an RTX 3080), where such an image generally will not run.

Match the version to the runtime host

Pick the CUDA version to match the GPU driver on the host where the workspace will actually run. The build host shown in the builder is only a reference. The runtime host’s driver is what has to support your choice. Going above the runtime driver’s version adds no performance.

flowchart TD
  A[Chosen CUDA version] --> B{At or below the driver's supported version?}
  B -- yes --> C[Runs: backward compatible]
  B -- no --> D{Same major series as the driver?}
  D -- yes --> E[Runs via minor version compatibility. Newer features unavailable]
  D -- no --> F[Needs data center forward compatibility. Will not run on consumer GPUs]

GPU architecture support

Driver compatibility is only half the story. Each CUDA version also targets a specific set of GPU architecture generations, and newer versions drop the oldest ones. For example, CUDA 13.x dropped Maxwell, Pascal, and Volta, so a CUDA 13.x image will not run on a Volta card even on a brand-new driver that reports CUDA 13.0. This is independent of the driver: it is about which architectures the toolkit still compiles for.

Each CUDA version targets a fixed set of architectures: it can be too old for a card (an architecture is only added in the version that introduces it) or too new (a later version drops the oldest ones). The builder knows each build host GPU’s architecture, so it flags any CUDA version that does not target it, in either direction. Such versions are dimmed and annotated with the architecture name (for example “no Volta support”), because they will not run on that GPU regardless of the driver.

For the full matrix of which CUDA versions support which GPU architectures and models, see CUDA GPUs supported (Wikipedia).

Reading the dropdown colors

Style	Meaning
Green	Backward compatible: at or below the host driver’s maximum supported CUDA version. The exact ceiling is annotated “highest supported”.
Amber	Minor version compatibility: a newer minor in the same major series. Runs, but features newer than the driver are unavailable.
Red	Needs a newer driver: a newer major series. Requires data center forward compatibility, and generally will not run on consumer GPUs.
Dimmed	The version does not target this host GPU’s architecture (too old for it, or a newer version that dropped it), so it will not run on that GPU at all.

Learn more

• CUDA Compatibility overview
• Why CUDA compatibility
• Minor version compatibility
• Forward compatibility

Why CUDA Version shows N/A

When Trailer deploys the NVIDIA container toolkit, it recreates the agent container with the NVIDIA runtime and the full driver capability set, which includes compute. The compute capability is what mounts the CUDA driver library into the container. With it in place, nvidia-smi reports the driver’s supported CUDA version and Trailer shows that value in the GPU details.

If the agent container is recreated by hand instead, for example from a docker run command or a Compose file that does not carry those settings, it comes back without the compute capability. nvidia-smi still enumerates the GPUs, temperature, power, and memory, but it reports CUDA Version: N/A. Trailer reports the same, because the CUDA driver library it reads that value from was never mounted:

NVIDIA-SMI 580.126.16    Driver Version: 580.126.16    CUDA Version: N/A

This does not affect your workspaces. Workloads run in workspace containers, which get their own GPU access, so their CUDA programs are unaffected. The impact is limited to reporting. Because the agent reads the CUDA version through nvidia-smi in its own container, a missing compute capability makes Trailer show CUDA Version as N/A for the host even though the driver supports a specific version. To restore accurate reporting, let Trailer’s toolkit deployment recreate the agent (it applies the correct settings), or add the settings to your own launch spec so a manual recreate keeps them:

docker run --runtime=nvidia \
  -e NVIDIA_VISIBLE_DEVICES=all \
  -e NVIDIA_DRIVER_CAPABILITIES=all \
  ...

Restarting the existing container is safe, since a restart keeps its settings. The capability is lost only when the container is recreated from a spec that omits it.

Per-workspace hardware acceleration

A workspace has a boolean Hardware Acceleration option in its configuration. When enabled, the agent attaches DRI/render devices to the container:

• On a Linux host: /dev/dri.
• On a Windows (WSL) host: /dev/dxg, /dev/dri/card0, /dev/dri/renderD128, plus a read-only bind mount of /usr/lib/wsl.

This is vendor-neutral device sharing for OpenGL/Vulkan/VA-API style workloads. It does not require the NVIDIA toolkit. Toggling hardware acceleration on or off triggers a workspace recreate.

A workspace also has a Nested Virtualization option. When enabled the agent attaches /dev/kvm and /dev/net/tun. This is what backs nested hypervisors and the Windows VDI runtime.

Per-workspace NVIDIA GPU sharing

A non-Windows workspace can have specific GPUs attached (each stored as a UUID plus BDF). For these workspaces the agent sets NVIDIA_VISIBLE_DEVICES to the attached GPUs’ UUIDs, so the NVIDIA Container Runtime exposes exactly those devices inside the container. This is the path that requires the toolkit and runtime described above.

If the workspace already sets NVIDIA_VISIBLE_DEVICES explicitly in its environment variables, the agent leaves that value alone.

Windows VDI GPU passthrough

GPU sharing via NVIDIA_VISIBLE_DEVICES and GPU passthrough are mutually exclusive. A Windows VDI workspace hands its attached GPUs to the guest VM via VFIO instead of advertising them to the container runtime.

For a Windows VDI workspace the agent:

• Auto-loads the vfio-pci kernel module if needed, then fails fast if it is still not loaded.
• Validates each requested GPU. Mediated modes (vGPU / vSGA), MIG-enabled GPUs, GPUs with no IOMMU group, and GPUs whose IOMMU group is not cleanly isolated to one physical card are rejected.
• Rebinds the whole IOMMU group (the GPU and its companion functions such as HDMI audio) to vfio-pci, then maps /dev/vfio/vfio plus each /dev/vfio/<group> device into the container.
• Records the passed-through PCI addresses in a container label so the devices are released back to the host driver when the workspace is deleted.

Because VFIO pins all of the guest’s RAM, the container’s memlock ulimit is raised to unbounded. See Windows virtual desktops for the rest of the Windows VDI runtime.

Supported vendors

Host-level driver/toolkit deployment, GPU inventory/metrics, and NVIDIA_VISIBLE_DEVICES sharing are NVIDIA-specific. Hardware acceleration (DRI device sharing) and VFIO passthrough are vendor-neutral at the device level, but the GPU detection and passthrough validation paths target NVIDIA display devices.