Things have been without any issue for the past 24 hours.
Things I had to do.
- Passthrough a NVDEC capable video card directly to the VM and Install it's drivers.
I have a dummy plug connected to it to ensure that the card is initialized, but I do not span the desktop to the dummy display on the Nvidia card.
- Configure the Camera in BlueIris to use NVDEC decoding and check "Also BVR" next to the decoding properties.
No more issues and the Quadro M2000 GPU stays at around 3% video decode load with 1x 1080p and 1x 4K stream, sitting at around 39C despite it's airflow being somewhat restricted.
P.S. Disabling the Vmware SVGA adapter was unnescicarry and I've since undone that so that I can access the VM's console through vsphere or Vmware Workstation.