Difference between revisions of "GPU passthrough to virtual machines"

From ZoneMinder Wiki
Jump to navigationJump to search
(No syntax highlighting!)
(Fix intro, add sig, remove refs)
Line 1: Line 1:
Using a GPU reduces the load on the CPUs and RAM.  For Zoneminder, /dev/shm at 50% reduction and load is about 30% what it was before deploying a NVidia GTX 1050.   
Using a GPU reduces the load on the CPUs and RAM.  I saw /dev/shm at 50% reduction and load about 30% what it was before deploying a NVidia GTX 1050.   


==Nvidia GPU in VMware==
==Nvidia GPU in VMware==
The versions shown here are essential because it did not work at all prior to some updates, crashed the host and the VM would not start.  The versions are the latest current at 6 Aug 2020.
The versions shown here are essential because it did not work at all prior to some updates, crashed the host and the VM would not start.  The versions are the latest current at now [[User:Gerdesj|Gerdesj]] ([[User talk:Gerdesj|talk]]) 17:29, 6 August 2020 (UTC).


*Host: Dell T320, 1 socket Xeon E5-2407 2.2 GHz CPU, BIOS 2.9.0
*Host: Dell T320, 1 socket Xeon E5-2407 2.2 GHz CPU, BIOS 2.9.0
Line 11: Line 11:


===ESXi host===
===ESXi host===
ssh into the host and edit /etc/vmware/passthru.map.  Change the word bridge to link.  This avoids a PSOD <ref>https://www.reddit.com/r/vmware/comments/f3xsgj/nvidia_gpu_esx_65_dell_t320_pci_passthrough_crash/ - Reddit post - NVIDIA GPU / ESX 6.5 / DELL T320 / PCI Pass-through crash on shutdown</ref> on the host when restarting the VM with the GPU passed through to it.
ssh into the host and edit /etc/vmware/passthru.map.  Change the word bridge to link.  This avoids a PSOD on the host when restarting the VM with the GPU passed through to it. See: https://www.reddit.com/r/vmware/comments/f3xsgj/nvidia_gpu_esx_65_dell_t320_pci_passthrough_crash/


  # NVIDIA
  # NVIDIA
  10de  ffff  link  false
  10de  ffff  link  false


Pass the GPU through to the host using the DirectPath I/O mechanism <ref>https://blogs.vmware.com/apps/2018/09/using-gpus-with-virtual-machines-on-vsphere-part-2-vmdirectpath-i-o.html - Using GPUs with Virtual Machines on vSphere – Part 2: VMDirectPath I/O</ref> and reboot, then connect both devices to the VM.  There will be an audio card and the video card itself.
Pass the GPU through to the host using the DirectPath I/O mechanism and reboot, then connect both devices to the VM.  There will be an audio card and the video card itself.  see: https://blogs.vmware.com/apps/2018/09/using-gpus-with-virtual-machines-on-vsphere-part-2-vmdirectpath-i-o.html


===Ubuntu VM===
===Ubuntu VM===
Line 60: Line 60:


In another console, you could run nvidia-smi and see a process using the GPU.
In another console, you could run nvidia-smi and see a process using the GPU.
==References==
<references />

Revision as of 12:29, 6 August 2020

Using a GPU reduces the load on the CPUs and RAM. I saw /dev/shm at 50% reduction and load about 30% what it was before deploying a NVidia GTX 1050.

Nvidia GPU in VMware

The versions shown here are essential because it did not work at all prior to some updates, crashed the host and the VM would not start. The versions are the latest current at now Gerdesj (talk) 17:29, 6 August 2020 (UTC).

  • Host: Dell T320, 1 socket Xeon E5-2407 2.2 GHz CPU, BIOS 2.9.0
  • VMware: ESXI 6.5.0 patch level 16576891
  • GPU: MSI Geforce GTX 1050 Ti (this card does not require any host BIOS settings changing, nor Memory Mapped I/O settings on the VM)
  • Cameras: Four Reolink RLC-520. Encoding at 2048 x 1536, 10 fps, High H.264 profile
  • VM: Ubuntu 20.04 LTS server with no extras. Four vCPUs, 6 GB RAM, 30GB root and EFI, 300GB XFS for /var

ESXi host

ssh into the host and edit /etc/vmware/passthru.map. Change the word bridge to link. This avoids a PSOD on the host when restarting the VM with the GPU passed through to it. See: https://www.reddit.com/r/vmware/comments/f3xsgj/nvidia_gpu_esx_65_dell_t320_pci_passthrough_crash/ 

# NVIDIA
10de  ffff  link   false

Pass the GPU through to the host using the DirectPath I/O mechanism and reboot, then connect both devices to the VM. There will be an audio card and the video card itself. see: https://blogs.vmware.com/apps/2018/09/using-gpus-with-virtual-machines-on-vsphere-part-2-vmdirectpath-i-o.html

Ubuntu VM

The VM must use EFI so the install must use the Ubuntu server installer and not the minimal installer which will not work with efiboot. VM type set to Ubuntu 64 bit.

In Advanced settings for the VM, set the following flag to false. This setting disables informing the VM it is a VM. This avoids a problem where the GPU fails to initialise properly:<syntaxhighlight lang="ini"> hypervisor.cpuid.v0 = FALSE </syntaxhighlight>

Nvidia drivers and CUDA

These instructions stay within the drivers etc provided by Ubuntu LTS. NVidia as upstream also provide drivers and these will be newer but may break something.

Use this command to decide which driver to install: 

# ubuntu-drivers devices

Install the "headless" version of the driver and reboot: 

# apt install nvidia-headless-440

Run this to confirm it is working after rebooting: 

# nvidia-smi

If you just need decoding eg for Zoneminder - this provides libnvcuvid.so: 

# apt install libnvidia-decode-440

Testing

Check ffmpeg has cuda support: 

# ffmpeg -hwaccels
ffmpeg version 4.2.4-1ubuntu0.1 Copyright (c) 2000-2020 the FFmpeg developers
...
Hardware acceleration methods:
vdpau
cuda
vaapi
drm
opencl
cuvid

There should be no error messages relating to libraries when you run something like this, which streams from a camera to /dev/null and uses CUDA: 

# ffmpeg -hwaccel cuda -i "rtmp://HOSTNAME_OR_IP/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=PASSWORD"  -an -f rawvideo -y /dev/null


In another console, you could run nvidia-smi and see a process using the GPU.

Retrieved from "https://wiki.zoneminder.com/index.php?title=GPU_passthrough_to_virtual_machines&oldid=16233"