Hi guys, I am facing an issue where a containerized bento failed to access GPU in k8s. Any pointers for further troubleshooting is deeply appreciated. Background: - Spinning up new k8s cluster to test out new cluster version updates before updating on production - Configs are identical across production cluster and new cluster - Tested with the same bentoml image that is currently running on production cluster in new k8s cluster,

CUDA version insufficient error

occured when instantiating onnx model runner. Items checked: - Driver version. Driver CUDA version is 11.6.2, which matches the selected CUDA version when building the bento. Also tested with CUDA 12 drivers but having the same

CUDA version insufficient error

- Exec into pod and run nvidia-smi. On the production bento pod,

nvidia-smi

ran successfully. However on test cluster bento pod, it returns

Permission Denied

error - Test the node with different GPU deployment. To isolate that it is not due to bad driver installation on that node, another GPU deployment (not bentoml) is deployed on the node and it does utilize the GPU correctly and able to run

nvidia-smi

Any ideas for further troubleshooting is deeply appreciated.