Fails to load if nvidia mig driver used for gpu

How to reproduce this bug?

in values yaml muti2vec-clip: ... enable_cuda: true ... #added this line to make it work cuda_visible_devices: 0 ... nvidia.com/mig-1g.5gb: 1

Also had to modiify file transformersInferenceDeployment.yaml to allow the line cuda_visible_devices -helm chart 16.4.0

What is the expected behavior?

Was expecting it to work without adding cuda_visible_devices line, the same as it worked when using nvidia.com/gpu: 1

What is the actual behavior?

If cuda_visible_device is not there it fails to start with error cuda_visible_device must be there to use nvidia mig driver.

Supporting information

No response

Server Version

1.20.5

Code of Conduct

• I have read and agree to the Weaviate's Contributor Guide and Code of Conduct