昇腾卡上用sglang启动无法开启mtp
System Info / 系統信息
sglang版本：0.5.15 启动指令：SGLANG_ENABLE_SPEC_V2=1 sglang serve --model-path /nas/disk1/GLM-OCR/ --port 8898 --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --served-model-name GLM-OCR > glm_ocr_sglang.log 2>&1 &
Who can help? / 谁可以帮助到您？
No response
Information / 问题信息
- My own modified scripts / 我自己修改的脚本和任务
- The official example scripts / 官方的示例脚本
Reproduction / 复现过程
启动成功后服务会崩溃，报错信息如下：
[2026-08-05 09:25:37] INFO: Started server process [36077]
[2026-08-05 09:25:37] INFO: Waiting for application startup.
[2026-08-05 09:25:37] INFO: Application startup complete.
[2026-08-05 09:25:37] INFO: Uvicorn running on http://0.0.0.0:8898 (Press CTRL+C to quit)
[2026-08-05 09:25:38] INFO: 127.0.0.1:59208 - "GET /model_info HTTP/1.1" 200 OK
/sgl-workspace/sglang/python/sglang/srt/entrypoints/openai/serving_chat.py:642: DeprecationWarning: max_tokens is deprecated in favor of the max_completion_tokens field
max_output_tokens = request.max_completion_tokens or request.max_tokens
/sgl-workspace/sglang/python/sglang/srt/entrypoints/openai/protocol.py:980: DeprecationWarning: max_tokens is deprecated in favor of the max_completion_tokens field
"max_new_tokens": self.max_completion_tokens or self.max_tokens,
[transformers] Accessing Glm46VImageProcessorFast from .models.glm46v.image_processing_glm46v. Returning Glm46VImageProcessor instead. Behavior may be different and this alias will be removed in future versions.
Warning: since the loaded file is not a zipfile, only "torch.device" and "str" type parameters are currently supported for parameter types of map_location. If parameter types of map_location is "Callable[[torch.Tensor, str], torch.Tensor]" or "Dict[str, str]", which is only support for zipfile, all tensors are currently loaded onto the CPU, which may introduce problems.
[2026-08-05 09:25:40] get env HOSTNAME = glm-ocr-01275668-n-1-7d-76dd8c679c-vdlgn
[2026-08-05 09:25:40] get env SGLANG_VIT_ENABLE_CUDA_GRAPH = 0
[2026-08-05 09:25:42] get env ASCEND_OPP_PATH = /usr/local/Ascend/cann-9.0.0/opp
/sgl-workspace/sglang/python/sglang/srt/environ.py:1220: UserWarning: Environment variable SGL_IN018162_N_1_7D_SERVICE_PORT is deprecated, please use SGLANG_IN018162_N_1_7D_SERVICE_PORT
warnings.warn(
/sgl-workspace/sglang/python/sglang/srt/environ.py:1220: UserWarning: Environment variable SGL_IN018162_N_1_7D_PORT_8898_TCP_PORT is deprecated, please use SGLANG_IN018162_N_1_7D_PORT_8898_TCP_PORT
warnings.warn(
/sgl-workspace/sglang/python/sglang/srt/environ.py:1220: UserWarning: Environment variable SGL_IN018162_N_1_7D_PORT_8898_TCP is deprecated, please use SGLANG_IN018162_N_1_7D_PORT_8898_TCP
warnings.warn(
/sgl-workspace/sglang/python/sglang/srt/environ.py:1220: UserWarning: Environment variable SGL_IN018162_N_1_7D_PORT_8898_TCP_ADDR is deprecated, please use SGLANG_IN018162_N_1_7D_PORT_8898_TCP_ADDR
warnings.warn(
/sgl-workspace/sglang/python/sglang/srt/environ.py:1220: UserWarning: Environment variable SGL_IN018162_N_1_7D_PORT is deprecated, please use SGLANG_IN018162_N_1_7D_PORT
warnings.warn(
/sgl-workspace/sglang/python/sglang/srt/environ.py:1220: UserWarning: Environment variable SGL_IN018162_N_1_7D_SERVICE_HOST is deprecated, please use SGLANG_IN018162_N_1_7D_SERVICE_HOST
warnings.warn(
/sgl-workspace/sglang/python/sglang/srt/environ.py:1220: UserWarning: Environment variable SGL_IN018162_N_1_7D_PORT_8898_TCP_PROTO is deprecated, please use SGLANG_IN018162_N_1_7D_PORT_8898_TCP_PROTO
warnings.warn(
[2026-08-05 09:25:57] get env HOME = /root
[2026-08-05 09:25:57] get env TE_AUTO_RESTART_COUNTER = 0
[2026-08-05 09:25:59] get env PYTHONPATH = /usr/local/Ascend/cann-9.0.0/python/site-packages:/usr/local/Ascend/cann-9.0.0/opp/built-in/op_impl/ai_core/tbe:/usr/local/Ascend/ascend-toolkit/latest/python/site-packages:/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe:
[2026-08-05 09:26:02] Scheduler hit an exception: Traceback (most recent call last):
File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 4659, in run_scheduler_process
scheduler.run_event_loop()
File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 1500, in run_event_loop
dispatch_event_loop(self)
File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 4510, in dispatch_event_loop
scheduler.event_loop_overlap()
File "/usr/local/python3.11.15/lib/python3.11/site-packages/torch/utils/contextlib.py", line 124, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 1601, in event_loop_overlap
batch_result = self.run_batch(batch)
^^^^^^^^^^^^^^^^^^^^^
File "/sgl-workspace/sglang/python/sglang/srt/utils/nvtx_utils.py", line 109, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 3371, in run_batch
batch_result = self.model_worker.forward_batch_generation(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sgl-workspace/sglang/python/sglang/srt/speculative/eagle_worker_v2.py", line 1194, in forward_batch_generation
verify_input: EagleVerifyInput = self.draft_worker.draft(batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sgl-workspace/sglang/python/sglang/srt/speculative/eagle_worker_v2.py", line 502, in draft
forward_batch, can_cuda_graph = prepare_for_draft(
^^^^^^^^^^^^^^^^^^
File "/sgl-workspace/sglang/python/sglang/srt/speculative/eagle_worker_common.py", line 302, in prepare_for_draft
draft_input.positions = batch.seq_lens.repeat_interleave(topk, dim=0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: query:../torch_npu/csrc/core/npu/NPUEvent.cpp:94 NPU function error: acl::AclQueryEventRecordedStatus(event, &currStatus), error code is 507035
[ERROR] 2026-08-05-09:26:02 (PID:36234, Device:0, RankID:-1) ERR00100 PTA call acl api failed
[Error]: The vector core execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EZ9999: Inner Error!
EZ9999[PID: 36234] 2026-08-05-09:26:01.676.352 (EZ9999): The error from device(chipId:3, dieId:0), serial number is 7, there is an exception of aivec error, core id is 3, error code = 0x800000, dump info: pc start: 0x1240001e3298, current: 0x1240001e4310, vec error info: 0xd901f8e354, mte error info: 0x8e030296b9, ifu error info: 0x7812a62808980, ccu error info: 0xd067ab6f5e81715e, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c140acac00.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:645]
TraceBack (most recent call last):
The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30296b9, fixp_error1 info: 0x8e, fsmId:0, tslot:4, thread:0, ctxid:0, blk:5, sublk:0, subErrType:4.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:658]
The error from device(chipId:3, dieId:0), serial number is 7, there is an exception of aivec error, core id is 4, error code = 0x800000, dump info: pc start: 0x1240001e3298, current: 0x1240001e4310, vec error info: 0x50ab7d6b7, mte error info: 0x8e030266b9, ifu error info: 0x3936987ac3fc0, ccu error info: 0xffaa2c5e67e4ec0b, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c140acac00.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:645]
The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30266b9, fixp_error1 info: 0x8e, fsmId:0, tslot:4, thread:0, ctxid:0, blk:6, sublk:0, subErrType:4.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:658]
The error from device(chipId:3, dieId:0), serial number is 7, there is an exception of aivec error, core id is 5, error code = 0x800000, dump info: pc start: 0x1240001e3298, current: 0x1240001e4310, vec error info: 0x5312e65829, mte error info: 0x8e030296b9, ifu error info: 0x565b441938e40, ccu error info: 0x10e9dee1de00a3f, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c140acac00.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:645]
The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30296b9, fixp_error1 info: 0x8e, fsmId:0, tslot:4, thread:0, ctxid:0, blk:7, sublk:0, subErrType:4.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:658]
The error from device(chipId:3, dieId:0), serial number is 7, there is an exception of aivec error, core id is 2, error code = 0x800000, dump info: pc start: 0x1240001e3298, current: 0x1240001e4310, vec error info: 0xf915746034, mte error info: 0x8e030266b9, ifu error info: 0x5fe986aa41cc0, ccu error info: 0xcc50d87d483bf4eb, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c140acac00.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:645]
The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30266b9, fixp_error1 info: 0x8e, fsmId:1, tslot:4, thread:0, ctxid:0, blk:4, sublk:0, subErrType:4.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:658]
Kernel task happen error, retCode=0x31, [vector core exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1729]
rtEventQueryStatus execution failed, reason=vector core exception[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
[Query][Status]query event recorded status failed, runtime result = 507035[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
[2026-08-05 09:26:02] SIGQUIT received. signum=None, frame=None. It usually means one child failed. ('Warning: torch.save with "_use_new_zipfile_serialization = False" is not recommended for npu tensor, which may bring unexpected errors and hopefully set "_use_new_zipfile_serialization = True"', 'if it is necessary to use this, please convert the npu tensor to cpu tensor for saving') [2026-08-05 09:26:02] Sleeping 5 seconds before crash diagnostics to let GPU activity settle. [rank0]:[W805 09:26:04.755850070 NPUStream.cpp:596] Warning: NPU warning, error code is 507035[Error]: [Error]: The vector core execution is abnormal. Rectify the fault based on the error information in the ascend log. EE9999: Inner Error! EE9999[PID: 36234] 2026-08-05-09:26:04.801.700 (EE9999): rtDeviceSynchronizeWithTimeout execution failed, reason=vector core exception[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65] TraceBack (most recent call last): wait for compute device to finish failed, runtime result = 507035.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148] (function npuSynchronizeUsedDevices) [W805 09:26:04.757177660 NPUStream.cpp:578] Warning: NPU warning, error code is 507035[Error]: [Error]: The vector core execution is abnormal. Rectify the fault based on the error information in the ascend log. EE9999: Inner Error! EE9999[PID: 36234] 2026-08-05-09:26:04.803.141 (EE9999): rtDeviceSynchronizeWithTimeout execution failed, reason=vector core exception[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65] TraceBack (most recent call last): wait for compute device to finish failed, runtime result = 507035.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148] (function npuSynchronizeDevice) Fatal Python error: Segmentation fault [2026-08-05 09:26:08] Pyspy failed (py-spy dump --pid 36234). Error: Error: Failed to copy Py_Version symbol
Caused by: 0: Permission denied (os error 13) 1: Permission denied (os error 13)
[2026-08-05 09:26:08] All pyspy dump attempts failed for PID 36234. [2026-08-05 09:26:08] CUDA user-triggered coredump is not enabled. Set CUDA_ENABLE_USER_TRIGGERED_COREDUMP=1 before CUDA initialization. [2026-08-05 09:26:08] CUDA coredump pipe not found for PID 36234: /model/corepipe.cuda.glm-ocr-01275668-n-1-7d-76dd8c679c-vdlgn.36234. Ensure CUDA_ENABLE_USER_TRIGGERED_COREDUMP=1 was set before this process initialized CUDA. [2026-08-05 09:26:08] Waiting 60.0 seconds for CUDA coredumps before exiting. !!!!!!! Segfault encountered !!!!!!! File "", line 0, in gsignal File "", line 0, in THPModule_npu_shutdown(_object*, _object*) File "Objects/methodobject.c", line 514, in cfunction_vectorcall_O File "./Include/internal/pycore_call.h", line 92, in _PyObject_VectorcallTstate File "Objects/call.c", line 299, in PyObject_Vectorcall File "Python/ceval.c", line 4769, in _PyEval_EvalFrameDefault File "./Include/internal/pycore_ceval.h", line 73, in _PyEval_EvalFrame File "Python/ceval.c", line 6434, in _PyEval_Vector File "./Modules/atexitmodule.c", line 98, in atexit_callfuncs File "Python/pylifecycle.c", line 1779, in Py_FinalizeEx File "Python/pylifecycle.c", line 1751, in Py_FinalizeEx File "Python/pylifecycle.c", line 2940, in Py_Exit File "Python/pythonrun.c", line 771, in handle_system_exit File "Python/pythonrun.c", line 828, in _PyErr_PrintEx File "Python/pythonrun.c", line 489, in PyRun_SimpleStringFlags File "Modules/main.c", line 255, in pymain_run_command File "Modules/main.c", line 596, in pymain_run_python File "Modules/main.c", line 684, in Py_RunMain File "Modules/main.c", line 714, in pymain_main File "Modules/main.c", line 738, in Py_BytesMain File "", line 0, in _start File "", line 0, in 0xffffffffffffffff
[2026-08-05 09:27:08] kill_process_tree called: parent_pid=36077, include_parent=True, pid=36077
Expected behavior / 期待表现
开启mtp启动模型
Source: zai-org/GLM-OCR