GPU passthrough in VMWare
Using a GPU should reduce the load on the CPUs and potentially enable a system to process more or larger camera feeds. It can be tricky to get GPU passthrough for some NVidia cards working on VMWare. This article is based on a running system.
Nvidia GPU in VMware
I have not tested whether all mitigations are still needed, since I first wrote this article. I suggest ignoring the VMware related changes (except for the actual passthrough step!) first and then add them in if they are still an issue. Note that one of them is to avoid a PSOD so ensure you have access to the system if it needs rebooting from a crash.
- Host: Dell T320, 1 socket Xeon E5-2407 2.2 GHz CPU, BIOS 2.9.0
- VMware: ESXI 6.5.0 patch level 16576891
- GPU: MSI Geforce GTX 1050 Ti (this card does not require any host BIOS settings changing, nor Memory Mapped I/O settings on the VM)
- Cameras: Four Reolink RLC-410. Encoding at 2048 x 1536, 10 fps, High H.264 profile
- VM: Ubuntu 20.04 LTS server with no extras. Four vCPUs, 6 GB RAM, 30GB root and EFI, 300GB XFS for /var
- Zoneminder: 1.34 and 1.36
ssh into the host and edit /etc/vmware/passthru.map. Change the word bridge to link. This avoids a PSOD on the host when restarting the VM with the GPU passed through to it. See: https://www.reddit.com/r/vmware/comments/f3xsgj/nvidia_gpu_esx_65_dell_t320_pci_passthrough_crash/
# NVIDIA 10de ffff link false
Pass the GPU through to the host using the DirectPath I/O mechanism and reboot, then connect both devices to the VM. There will be an audio card and the video card itself. see: https://blogs.vmware.com/apps/2018/09/using-gpus-with-virtual-machines-on-vsphere-part-2-vmdirectpath-i-o.html
Ubuntu 20.04 VM
The VM must use EFI so the install must use the Ubuntu server installer and not the minimal installer which will not work with efiboot. VM type set to Ubuntu 64 bit.
In Advanced settings for the VM, set the following flag to false. This setting disables informing the VM it is a VM. This avoids a problem where the GPU fails to initialise properly:
hypervisor.cpuid.v0 = FALSE
Nvidia drivers and CUDA
These instructions stay within the drivers etc provided by Ubuntu 20.04 LTS. NVidia as upstream also provide drivers and these will be newer but may break something. The OS provided ffmpeg has cuda support built in.
Use this command to decide which driver to install:
# ubuntu-drivers devices
Install the "headless" version of the driver and reboot, optionally install nvidia-utils to get nvidia-smi:
# apt install nvidia-headless-440 # apt install nvidia-utils-440
Run this to confirm it is working after rebooting, if you have the utils installed:
If you just need decoding eg for Zoneminder - this provides libnvcuvid.so: libnvidia-decode-440. It should get installed automatically.
If it does not work properly then reboot the ESXi host. For example the two devices seem to pass through OK but only the audio devices seem to work. You may have crashed the GPU in some way. Messing around with drivers and too many restarts of the VM seems to cause this for me.
Check ffmpeg has cuda support:
# ffmpeg -hwaccels ffmpeg version 4.2.4-1ubuntu0.1 Copyright (c) 2000-2020 the FFmpeg developers ... Hardware acceleration methods: vdpau cuda vaapi drm opencl cuvid
There should be no error messages relating to libraries when you run something like this, which streams from a camera to /dev/null and uses CUDA:
# ffmpeg -hwaccel cuda -i "rtmp://HOSTNAME_OR_IP/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=PASSWORD" -an -f rawvideo -y /dev/null
In another console, you could run nvidia-smi and see a process using the GPU.
Camera parameters for reference. Reolinks have three streams - main, sub and ext. main is the clear stream and sub is the lowest quality one. These cameras also have a RTSP stream but that appears to be pretty flakey compared to RTMP. I want to watch these cameras so I monitor at high resolution. If you are building a security system then monitor sub and record main (bold in the URLs in the table.) Monitoring at say 640 x 480 7 frames per second will allow you to monitor a huge number of cameras.
All other settings on defaults. Initial setup done on 1.34. See below for notes on 1.36
|Source Path||rtmp://HOSTNAME_OR_IP/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=PASSWORD |
|Target colorspace||32 bit colour|
|Save JPEGs||Frames + Analysis images (if available)|
|Video Writer||H264 Camera Passthrough|
Changes between 1.34 and 1.36
I had to change the buffers settings to stop zmc crashing. Maximum Image Buffer Size (frames) from 25 to 0. The default is 0 if you create a new monitor in 1.36 which autotunes the buffer. Image Buffer Size (frames) to 5 which smoothes the live view. /dev/shm is no longer a concern in 1.36 as it was before (see the release notes: https://forums.zoneminder.com/viewtopic.php?f=1&t=30781 )