[Help] lora微调合并后模型推理速度明显慢了好多

Author: daydayup-zynCreated Oct 13, 2023Updated Aug 9, 2024

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

只用lora微调了20条数据集,合并模型后,推理速度相比原来的基座模型慢了好多,这个是什么原因?跟微调过程中的参数有关系吗?

Expected Behavior

No response

Steps To Reproduce

Environment

markdown
- OS:
- Python:
- Transformers:
- PyTorch:
- CUDA Support (`python -c "import torch; print(torch.cuda.is_available())"`) :

Anything else?

No response