NVIDIA GPU Passthrough on Proxmox for Jellyfin Transcoding

What this architecture solves

Software transcoding can turn a media server into a CPU-bound workload. Passing a GPU directly to a virtual machine keeps the hypervisor separate from the application while allowing the container to use dedicated video encode and decode engines.

Proxmox host -> VFIO -> Linux VM -> NVIDIA driver -> Container Toolkit -> Jellyfin

The GPU is reserved for one VM. This is not a way to share a device between several VMs, and results depend on chipset support, IOMMU groups, GPU generation, codecs, and client devices.

Keep physical console access or an alternate remote path to the host. A wrong boot, driver, or PCIe configuration can make the VM or host temporarily unreachable.

The building blocks

IOMMU isolates PCIe devices and their memory access. Intel calls it VT-d and AMD calls it AMD-Vi. VFIO is the Linux framework that binds a PCIe device so QEMU/KVM can pass it to a VM.

Check the entire IOMMU group before assigning anything. A GPU commonly exposes video and audio functions, and every relevant function must be considered together.

NVENC encodes video and NVDEC decodes it using dedicated GPU engines. Jellyfin may need them when a client cannot play the original codec, needs a lower bitrate, requires subtitle burn-in, or performs HDR to SDR tone mapping.

Always check whether the client can use direct play first. Avoiding an unnecessary transcode is usually the best optimization.

Requirements and placeholders

You need IOMMU-capable CPU and firmware, a suitable IOMMU group, a recovery path to the Proxmox host, and a Linux VM using Q35 plus OVMF. Replace these placeholders with values from your own environment:

1. Enable and verify IOMMU

Start by inspecting kernel messages and group availability:

dmesg | grep -Ei 'DMAR|IOMMU|AMD-Vi'
find /sys/kernel/iommu_groups/ -type l | wc -l

Enable VT-d or AMD-Vi in firmware if no groups are available. Add the correct kernel parameter through the boot method your Proxmox installation uses:

# Intel
intel_iommu=on iommu=pt

# AMD
amd_iommu=on iommu=pt

After rebooting, verify the running kernel command line rather than assuming the bootloader update took effect:

cat /proc/cmdline

2. Inspect IOMMU groups

List devices by group before binding the GPU:

for device in /sys/kernel/iommu_groups/*/devices/*; do
  group=${device#*/iommu_groups/*}
  group=${group%%/*}
  printf 'Group %s: ' "$group"
  lspci -nns "${device##*/}"
done

Record video, audio, and numeric device IDs:

lspci -nn | grep -Ei 'VGA|3D|Audio.*NVIDIA'
lspci -k -s <PCI_ADDRESS>

An isolated GPU/audio group is ideal. Do not use ACS overrides as a shortcut unless you understand the isolation trade-off.

3. Bind the GPU to VFIO

The Proxmox host should not load nouveau or the proprietary NVIDIA driver for the device being passed through:

sudo tee /etc/modprobe.d/blacklist-gpu-passthrough.conf >/dev/null <<'EOF'
blacklist nouveau
blacklist nvidia
blacklist nvidia_drm
blacklist nvidia_modeset
EOF

sudo tee /etc/modprobe.d/vfio.conf >/dev/null <<'EOF'
options vfio-pci ids=<GPU_IDS>
EOF

printf '%s\n' vfio vfio_iommu_type1 vfio_pci | sudo tee -a /etc/modules
sudo update-initramfs -u -k all
sudo reboot

After reboot, the key result is that vfio-pci owns the device:

lspci -k -s <PCI_ADDRESS>

4. Configure the VM

Use Q35 for a modern PCIe bus and OVMF for UEFI firmware. Inspect the current VM configuration before changing it:

qm config <VM_ID>
qm set <VM_ID> -hostpci0 <PCI_ADDRESS>,pcie=1

x-vga=1 is sometimes needed, but makes the passed-through GPU the primary adapter. If the virtual display is also removed, noVNC may no longer be a recovery option. Keep SSH, serial console, or a documented rollback path.

Secure Boot

DKMS-built NVIDIA modules can be rejected when Secure Boot does not trust their signature. Prefer enrolling a MOK key or using distribution-signed packages. Disabling Secure Boot can be a documented risk decision in a controlled environment, but should not be the default solution.

5. Install and validate the driver in the VM

The VM has its own operating system and driver stack. Install the driver and matching kernel headers from your distribution repositories:

sudo apt update
sudo apt install -y linux-headers-$(uname -r) nvidia-driver
sudo reboot
nvidia-smi

Do not proceed to Docker until nvidia-smi works in the VM. Check kernel messages, DKMS state, Secure Boot, and version compatibility first.

6. Expose the GPU to Docker

Install NVIDIA Container Toolkit from NVIDIA’s current documentation, configure Docker, and test it before adding Jellyfin:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
docker run --rm --gpus all nvidia/cuda:<TAG> nvidia-smi

Use a maintained image tag. If this test fails, debug the chain in order: GPU visibility in the VM, working driver, configured runtime, restarted Docker service.

7. Declare GPU access in Compose

Compose syntax varies across versions. This intentionally generic example shows the relevant controls:

services:
  jellyfin:
    image: jellyfin/jellyfin:latest
    environment:
      NVIDIA_VISIBLE_DEVICES: all
      NVIDIA_DRIVER_CAPABILITIES: compute,video,utility
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    volumes:
      - ./config:/config
      - ./cache:/cache
      - /srv/media:/media:ro

video enables NVENC/NVDEC and utility allows diagnostics such as nvidia-smi. Keep media read-only when the service does not need write access.

docker compose up -d jellyfin
docker exec jellyfin nvidia-smi

8. Configure and validate Jellyfin

Select NVIDIA acceleration in Jellyfin and enable only codecs supported by your GPU. AV1, HEVC, and tone mapping support vary by GPU generation and FFmpeg build.

Force a real transcode for testing. Direct play is desirable, but it does not test GPU acceleration. During a conversion, watch the device:

watch -n 1 nvidia-smi

Success criteria:

• Jellyfin-related FFmpeg process is visible.
• GPU utilization and memory increase during conversion.
• CPU is no longer the dominant encoding resource.
• Playback remains stable without sustained buffering.
• Temperature and power stay within vendor limits.

Troubleshooting

IOMMU missing: revisit firmware, boot parameters, and the actual boot method. Confirm /proc/cmdline after reboot.

GPU not bound to VFIO: re-check video and audio IDs with lspci -nn, verify no graphics driver owns the device, then rebuild initramfs.

VM starts without console: review Q35, OVMF, PCIe assignment, and x-vga. Restore virtual display temporarily through host access if needed.

nvidia-smi fails: check kernel/headers alignment, DKMS output, Secure Boot, and package sources.

Docker cannot see the device: test each layer independently, starting with nvidia-smi inside the VM.

Jellyfin still uses CPU: confirm an actual transcode, inspect FFmpeg logs, validate codec support, and prefer direct play when possible.

Capacity testing and operations

Capacity is shaped by source/destination codec, resolution, HDR, subtitles, bitrate, network, GPU temperature, and FFmpeg version. Measure it rather than publishing a universal stream count:

1. Define a repeatable media and client sample.
2. Record codec, resolution, and bitrate for each test.
3. Add one stream at a time.
4. Track GPU, VRAM, CPU, latency, temperature, and buffering.
5. Stop before service quality degrades.

Keep VM configuration backups, alternate host access, and a rollback procedure for bootloader, initramfs, drivers, and PCIe assignment. The goal is not merely making a GPU appear in a dashboard; it is maintaining an understandable and recoverable chain from hypervisor to container.