请问seq_cls分类任务lora训练如何加速推理

Author: lln1997Created Jun 16, 2026Updated Sep 15, 2026

Labelsquestionstale

Checklist / 检查清单

I have searched existing issues, and this is a new question or discussion topic. / 我已经搜索过现有的 issues，确认这是一个新的问题与讨论。

Question Description / 问题描述

目前的推理代码如下，请问有支持batch推理或者vllm的batch推理嘛

import os
import json
from swift import BaseArguments, InferRequest, TransformersEngine, get_template
from tqdm import tqdm

adapter_path = 'xxx'
args = BaseArguments.from_pretrained(adapter_path)
engine = TransformersEngine(args.model, adapters=[adapter_path])
template = get_template(
    engine.processor, args.system, template_type=args.template, use_chat_template=args.use_chat_template)
engine.template = template
with open('xxx', 'r') as f,open('xxx', 'w') as f1:
    for line in tqdm(f):
        data = json.loads(line)
        data['messages'].append({"role": "assistant", "content": "<think>\n\n</think>\n\n"})
        infer_request = InferRequest(data["messages"])
        resp_list = engine.infer([infer_request])
        response: int = resp_list[0].choices[0].message.content
        data['predict'] = response  
        f1.write(json.dumps(data, ensure_ascii=False) + '\n')
        f1.flush()

Source: modelscope/ms-swift

View original on GitHub View discussion on GitHub