Low inference speed in A100 PCIE GPU

Hi @GeeeekExplorer, thank you very much for your works!

I tried running bench.py on a A100 80GB PCIE GPU, but got the below throughput with nano-vllm (10% of your reported number on RTX4070)

Total: 133966tok, Time: 1012.83s, Throughput: 132.27tok/s

When I tried with vllm (v0.21.0), the results were:

Total: 133966tok, Time: 14.06s, Throughput: 9530.43tok/s

For reference, the library version is:

>>> torch.__version__
'2.11.0+cu130'
>>> transformers.__version__
'5.7.0'
>>> vllm.__version__
'0.20.1rc1'

Do you have an idea why this happened ?