nvtop doesn't display the correct GPU usage of processes on MIG devices

When I start a Python script using Pytorch and Cuda to compute anything (e.g., matrix multiplication) on a GPU instance of a MIG-capable Nvidia GPU (in my case, an A100), nvtop displays that a process is using the GPU, but only displays 0% GPU usage.

System Specifications

• nvtop Version: 3.0.2 (also tested with 3.0.0 (ppa), and 1.2.2 (apt))
• Distro: Ubuntu 22.04
• GPU: Nvidia A100, split into different configurations using MIG
• Python 3.10
• PyTorch 2.1.1

Steps to reproduce

• Create the following python script

  python复制

  import torch
  
  device = torch.device('cuda')
  tensor1 = torch.rand((3_000, 3_000,)).to(device)
  tensor2 = torch.rand((3_000, 3_000,)).to(device)
  
  for _ in range(3_000):
      tensor3 = tensor1 @ tensor2

• Find the UUID of your GPU instance. It should start with MIG-

  bash复制

  nvidia-smi -L

• Run the script

  bash复制

  CUDA_VISIBLE_DEVICES=<UUID of the MIG device> python <name of your python script>

• While the program is running, it is displayed in nvtop, but with 0% GPU usage

  

• The script takes about 9 seconds to finish on a 7g.40gb GPU instance (which is the whole GPU).

If you have any more questions regarding my setup or want me to test some things, I'll be more than happy to answer your questions or test out different configurations. I don't have a different type of MIG-capable GPU or a different server to test on.