[Tracking] CI Test Failures and Fixes

# [Tracking] CI Test Failures and Fixes This issue tracks CI test failures, flaky tests, and infrastructure issues across SGLang's CI pipeline. **Note:** We mainly focus on scheduled CI on the `main` branch. **Last auto-update:** 2026-09-17 23:06 UTC — 5 broken, 9 flaky, 1031 recently fixed ## Quick Links - **[CI Coverage Overview](https://github.com/sgl-project/sglang/actions/workflows/ci-coverage-overview.yml)** - Check which tests are covered, by which backend/suite, and whether they are disabled - **[CI Failure Monitor](https://github.com/sgl-project/sglang/actions/workflows/ci-failure-monitor.yml)** - Automated failure analysis --- ## Ongoing Issues ### NV CI | Date | Test | Backend | Error | Notes | CI Status | Assignee | Related | |------|------|---------|-------|-------|-----------|----------|---------| | 2026-09-17 | `test_bench_serving_1gpu_part2.py` | call-pr-test-extra / extra-a-test-1-gpu-large / extra-a-test-1-gpu-large (1) | AssertionError: 58.996521273235864 not greater than 60 | performance (2% fail, 33/2229) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/35214306405/job/105179150076) | | 2026-09-16 | `test_bench_serving_2gpu.py` | call-pr-test-extra / extra-a-test-2-gpu-large / extra-a-test-2-gpu-large (0) | AssertionError: 916.0374041096846 not greater than 2200 | performance (2% fail, 17/1017) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/35088936689/job/104770520620) | | 2026-09-07 | `test_declared_fallbacks.py` | base-a-test-cpu / base-a-test-cpu (11) | NameError: name 'Optional' is not defined | environment (0% fail, 1/1435) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/34092676218/job/101649469267) | | 2026-05-13 | `test_deepseek_v4_pd_disagg_b200.py` | stage-c-test-dsv4-8-gpu-b200 | RuntimeError: Server process exited with code -9 | server (100% fail, 4/4) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/25814445163/job/75839184349) | | 2026-09-17 | `test_diffusion_1_gpu.py` | call-multimodal-gen-tests / multimodal-gen-test-1-gpu (0) | Failed: [request 1/1] Diffusion testcase 'qwen_image_t2i_cache_dit_enabled' f... | unknown (3% fail, 144/4559) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/35267607515/job/105359377263) | | 2026-09-17 | `test_diffusion_2_gpu.py` | call-multimodal-gen-tests / multimodal-gen-test-2-gpu (0) | Failed: [request 1/1] Diffusion testcase 'flux2_modelopt_fp8_tp2_t2i' failed ... | unknown (2% fail, 64/3834) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/35250503291/job/105301958387) | | 2026-09-17 | `test_disaggregation_decode_radix_cache_swa.py` | call-pr-test-extra / extra-b-test-8-gpu-h200 / extra-b-test-8-gpu-h200 (1) | RuntimeError: Server process exited with code -9 | server (2% fail, 5/247) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/35214306405/job/105179150157) | | 2026-05-22 | `test_dsa_models_mtp.py` | stage-c-test-8-gpu-h200 (0) | AssertionError: 179.9824287273711 not greater than 180 | performance (9% fail, 35/375) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/26282270975/job/77394696209) | | 2026-09-17 | `test_inkling.py` | base-b-test-1-gpu-large / base-b-test-1-gpu-large (0) | | unknown (0% fail, 11/9302) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/35272318340/job/105378652391) | | 2026-09-17 | `test_mamba2_track_ssm_indices.py` | base-a-test-cpu / base-a-test-cpu (4) | AttributeError: 'types.SimpleNamespace' object has no attribute 'forward_mode' | environment (0% fail, 3/5932) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/35264250756/job/105347619207) | | 2026-09-16 | `test_ministral4_models.py` | call-pr-test-extra / extra-a-test-2-gpu-large / extra-a-test-2-gpu-large (1) | AssertionError: 0.42222 not greater than or equal to 0.45 : Model mistralai/M... | accuracy (1% fail, 22/1619) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/35088936689/job/104770520531) | | 2026-08-28 | `test_moe_preprocess.py` | call-jit-kernel-tests / jit-kernel-unit-test | Job timed out while running this test | timeout (0% fail, 1/2575) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/33177338872/job/98869518293) | | 2026-09-17 | `test_numa_utils.py` | base-c-test-4-gpu-b200 / base-c-test-4-gpu-b200 (0) | AssertionError: 2 != 1 : Expected all visible 4-GPU B200 devices on one NUMA ... | correctness (0% fail, 4/4162) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/35257456921/job/105376761798) | | 2026-09-17 | `test_unified_radix_cache_kl_full.py` | base-b-test-2-gpu-large / base-b-test-2-gpu-large (1) | | unknown (0% fail, 21/7056) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/35261389005/job/105341770261) | ---

Recently Fixed (click to expand)

| Date | Issue | Fix | |------|-------|-----| | 2026-09-17 | `test_base_grammar_backend.py` — AssertionError: expected call not found. | - | | 2026-09-17 | `test_basic_sanity.py` — | - | | 2026-09-17 | `test_decode_bookkeeping_ownership.py` — AssertionError: ["New bookkeeping mutation(s) beyond the ... | - | | 2026-09-17 | `test_dflash.py` — | - | | 2026-09-17 | `test_diffusion_1_gpu_5090.py` — exit code 1 | - | | 2026-09-17 | `test_diffusion_1_gpu_b200.py` — sglang.multimodal_gen.test.server.test_server_common.Perf... | - | | 2026-09-17 | `test_diffusion_stage_profiler_sync.py` — AssertionError: producer tail leaked into consumer: {'pro... | - | | 2026-09-17 | `test_diffusion_unit.py` — AssertionError: Tuples differ: ('ulysses', False) != ('kv... | - | | 2026-09-17 | `test_disaggregation_decode_radix_cache.py` — AssertionError | - | | 2026-09-17 | `test_disaggregation_dsv4.py` — RuntimeError: Server process exited with code -9 | - | | 2026-09-17 | `test_disaggregation_nixl.py` — RuntimeError: Server process exited with code -9 | - | | 2026-09-17 | `test_disaggregation_wire.py` — AttributeError: C128_STATE | - | | 2026-09-17 | `test_dspark_target_hidden_projection.py` — AttributeError: 'types.SimpleNamespace' object has no att... | - | | 2026-09-17 | `test_dsv4_hicache_l2.py` — AttributeError: 'types.SimpleNamespace' object has no att... | - | | 2026-09-17 | `test_flux2_fp8_norm_quant_gate.py` — exit code 1 | - | | 2026-09-17 | `test_hicache_controller_request_id_unit.py` — IndexError: tuple index out of range | - | | 2026-09-17 | `test_hicache_staged_write_back_dispatch.py` — NameError: name 'transfer_mamba_state' is not defined | - | | 2026-09-17 | `test_hicache_storage_3fs_backend.py` — AssertionError: 0.0 not greater than 0.6 : Initial accura... | - | | 2026-09-17 | `test_inkling_unified.py` — AssertionError: avg_kl_div=0.013210544538507412 > thresho... | - | | 2026-09-17 | `test_kv_translate_ownership.py` — AttributeError: '_Inner' object has no attribute '_loc_ma... | - | | 2026-09-17 | `test_model_overrides.py` — AssertionError: Items in the first set but not the second: | - | | 2026-09-17 | `test_npu_mamba_async_layout.py` — NameError: name 'transfer_mamba_state' is not defined | - | | 2026-09-17 | `test_openai_server.py` — | - | | 2026-09-17 | `test_pp_hybrid_kv_transfer.py` — AttributeError: 'HybridLinearKVPool' object has no attrib... | - | | 2026-09-17 | `test_pp_pd_consensus.py` — AttributeError: 'types.SimpleNamespace' object has no att... | - | | 2026-09-17 | `test_qwen_image_fp8_norm_quant.py` — exit code 1 | - | | 2026-09-17 | `test_req_time_stats.py` — AssertionError: 4.599999904632568 != 4.6 within 7 places ... | - | | 2026-09-17 | `test_routed_experts_dp_readback.py` — | - | | 2026-09-17 | `test_rust_extension.py` — AssertionError: RuntimeError not raised | - | | 2026-09-17 | `test_scheduler_control.py` — | - | | 2026-09-17 | `test_server_args_docs_coverage.py` — AssertionError: ['--cuda-graph-prefill-max-context', '--d... | - | | 2026-09-17 | `test_server_info.py` — KeyError: 'disaggregation_mode' | - | | 2026-09-17 | `test_spec_eagle.py` — | - | | 2026-09-17 | `test_spec_eagle_topk.py` — | - | | 2026-09-17 | `test_spec_eagle_triton.py` — | - | | 2026-09-17 | `test_storage_prefetch_lifecycle.py` — AttributeError: 'types.SimpleNamespace' object has no att... | - | | 2026-09-17 | `test_vision_openai_server_a.py` — | - | | 2026-09-16 | `test_admission_abort_not_enqueued.py` — AttributeError: 'types.SimpleNamespace' object has no att... | - | | 2026-09-16 | `test_awq.py` — AssertionError: 0.63671875 not greater than 0.64 | - | | 2026-09-16 | `test_bcg_with_speculative_decoding.py` — | - | | 2026-09-16 | `test_benchmark_datasets_api.py` — exit code 1 | - | | 2026-09-16 | `test_block_fp8_as_mxfp8.py` — ModuleNotFoundError: No module named 'sglang.kernels.ops.... | - | | 2026-09-16 | `test_constrained_decoding.py` — KeyError: 'text' | - | | 2026-09-16 | `test_constrained_decoding_spec_reasoning.py` — | - | | 2026-09-16 | `test_data_parallelism.py` — RuntimeError: GPU(s) still not idle after waiting 30s bef... | - | | 2026-09-16 | `test_decode_queue_cleanup.py` — TypeError: '<=' not supported between instances of 'Magic... | - | | 2026-09-16 | `test_decode_radix_lock_ref.py` — TypeError: '<=' not supported between instances of 'Magic... | - | | 2026-09-16 | `test_deepep_small.py` — | - | | 2026-09-16 | `test_deepep_v2_expand_kernels.py` — AssertionError: Lists differ: [0, 0[591 chars], 0, 0, 0, ... | - | | 2026-09-16 | `test_deepep_v2_wire_dtype.py` — AttributeError: 'DeepEPv2DispatchOutput' object has no at... | - | | 2026-09-16 | `test_deepseek_v3_mtp.py` — | - | | 2026-09-16 | `test_deepseek_v4_flash_fp4_b200.py` — | - | | 2026-09-16 | `test_deepseek_v4_flash_fp4_h200.py` — | - | | 2026-09-16 | `test_deepseek_v4_rope_policy.py` — exit code 1 | - | | 2026-09-16 | `test_deepseek_v4_unified_fp8_q_pair.py` — exit code 1 | - | | 2026-09-16 | `test_diffusion_component_accuracy.py` — RuntimeError: Weight transfer matched 705/813 (86.72%); b... | - | | 2026-09-16 | `test_disaggregation_basic.py` — RuntimeError: Server process exited with code 1 | - | | 2026-09-16 | `test_disaggregation_different_tp.py` — RuntimeError: Server process exited with code -9 | - | | 2026-09-16 | `test_disaggregation_lora.py` — | - | | 2026-09-16 | `test_dp_attention.py` — | - | | 2026-09-16 | `test_dp_attention_local_control_broadcast.py` — AssertionError: 28 != 32 : Some requests failed: 28/32 su... | - | | 2026-09-16 | `test_dsa_glm52_pd_mtp_cp_layersplit.py` — RuntimeError: Server process exited with code -9 | - | | 2026-09-16 | `test_dsa_glm52_tp_mtp.py` — | - | | 2026-09-16 | `test_dsa_metadata_replay.py` — AssertionError: Tensor-likes are not equal! | - | | 2026-09-16 | `test_dsv41_verify_candidate_graph.py` — AssertionError: False is not true | - | | 2026-09-16 | `test_eagle_reject_sampling.py` — | - | | 2026-09-16 | `test_flashinfer_autotune_sync.py` — exit code 1 | - | | 2026-09-16 | `test_fp8_bpreshuffle_scale.py` — ModuleNotFoundError: No module named 'sglang.kernels.ops.... | - | | 2026-09-16 | `test_fp8_utils.py` — AttributeError: 'Fp8LinearMethod' object has no attribute... | - | | 2026-09-16 | `test_full_cuda_graph_prefill.py` — | - | | 2026-09-16 | `test_fused_moe_native.py` — exit code 1 | - | | 2026-09-16 | `test_gated_launch.py` — exit code 1 | - | | 2026-09-16 | `test_hicache_storage_mooncake_backend.py` — RuntimeError: GPU(s) still not idle after waiting 30s bef... | - | | 2026-09-16 | `test_hicache_variants.py` — | - | | 2026-09-16 | `test_ipc_a2a_jit_recovery.py` — RuntimeError: GPU(s) still not idle after waiting 30s bef... | - | | 2026-09-16 | `test_kimi_k3_b300_low_latency.py` — | - | | 2026-09-16 | `test_kimi_linear_models.py` — RuntimeError: GPU(s) still not idle after waiting 30s bef... | - | | 2026-09-16 | `test_laguna_hidden_dim_unit.py` — exit code 1 | - | | 2026-09-16 | `test_llama8b_nvfp4_kv_cache_sm120.py` — AssertionError: np.float64(0.617199391171994) not greater... | - | | 2026-09-16 | `test_mamba2.py` — AttributeError: 'MockMamba2ModelRunner' object has no att... | - | | 2026-09-16 | `test_mla_int8_deepseek_v3.py` — | - | | 2026-09-16 | `test_mlp_sync_pad_unpad.py` — TypeError: ForwardBatch.init_mlp_sync_metadata() missing ... | - | | 2026-09-16 | `test_nano_nemotron_vl.py` — exit code 1 | - | | 2026-09-16 | `test_nemotron_h_weight_loading.py` — exit code 1 | - | | 2026-09-16 | `test_npu_mxfp8_deepep_dispatch.py` — AttributeError: '_DeepEPDispatcherImplLowLatency' object ... | - | | 2026-09-16 | `test_openai_server_hidden_states.py` — | - | | 2026-09-16 | `test_pair_pool_decode.py` — ModuleNotFoundError: No module named 'sglang.kernels.ops.... | - | | 2026-09-16 | `test_parallel_state.py` — exit code 1 | - | | 2026-09-16 | `test_pooler_score_and_pool.py` — exit code 1 | - | | 2026-09-16 | `test_q8kv8_sparse_prefill_backend.py` — exit code 1 | - | | 2026-09-16 | `test_qwen35_fp4_mtp.py` — | - | | 2026-09-16 | `test_qwen3_next_models_mtp.py` — | - | | 2026-09-16 | `test_resolution_hook_registry.py` — AssertionError: Tuples differ: ([], ['validate_mega_moe_t... | - | | 2026-09-16 | `test_resolution_is_reproducible.py` — exit code 1 | - | | 2026-09-16 | `test_run_rust_tests.py` — exit code 1 | - | | 2026-09-16 | `test_run_sglang_radix_tree_rust_tests.py` — exit code 1 | - | | 2026-09-16 | `test_runai_model_streamer_loader.py` — exit code 1 | - | | 2026-09-16 | `test_sampling_batch_info.py` — exit code 1 | - | | 2026-09-16 | `test_scheduler_chunked_req_gate.py` — AttributeError: 'Scheduler' object has no attribute 'disa... | - | | 2026-09-16 | `test_server_args.py` — ValueError: DeepEP v2 per-rank decode CUDA graph exceeds ... | - | | 2026-09-16 | `test_serving_chat.py` — AssertionError: 3 != 1 | - | | 2026-09-16 | `test_spec_eagle_fa3.py` — | - | | 2026-09-16 | `test_spec_eagle_page.py` — | - | | 2026-09-16 | `test_spec_eagle_topk_page.py` — | - | | 2026-09-16 | `test_streaming_session.py` — | - | | 2026-09-16 | `test_trace.py` — AssertionError: '[PP 1]' not found in '[TP 2] [PP 0] [DP ... | - | | 2026-09-16 | `test_triton.py` — AttributeError: 'MockMLAModelRunner' object has no attrib... | - | | 2026-09-16 | `test_unified_radix_finished_req.py` — AttributeError: 'types.SimpleNamespace' object has no att... | - | | 2026-09-16 | `test_vision_seqlens.py` — exit code 1 | - | | 2026-09-15 | `test_checkpoint_prefix.py` — ModuleNotFoundError: No module named 'vllm' | - | | 2026-09-15 | `test_decode_retraction_backup.py` — ImportError: cannot import name 'get_device_accessible_pt... | - | | 2026-09-15 | `test_deepseek_v4_flash_fp4_b200_trtllm.py` — AssertionError: np.float64(0.025) not greater than or equ... | - | | 2026-09-15 | `test_diffusion_bcg.py` — Exception: retry() exceed maximum number of retries. | - | | 2026-09-15 | `test_disaggregation_aarch64.py` — RuntimeError: Server process exited with code -9 | - | | 2026-09-15 | `test_disaggregation_chunked_prefill_abort.py` — RuntimeError: Server process exited with code -9 | - | | 2026-09-15 | `test_disaggregation_decode_offload.py` — TimeoutError: Endpoint http://127.0.0.1:11100/health did ... | - | | 2026-09-15 | `test_disaggregation_dp_attention.py` — RuntimeError: Server process exited with code -9 | - | | 2026-09-15 | `test_disaggregation_dwdp_gpt_oss.py` — RuntimeError: Server process exited with code 1 | - | | 2026-09-15 | `test_disaggregation_hisparse.py` — RuntimeError: Server process exited with code -9 | - | | 2026-09-15 | `test_disaggregation_inkling_mxfp8.py` — RuntimeError: Server process exited with code -9 | - | | 2026-09-15 | `test_disaggregation_optimistic_prefill.py` — RuntimeError: Server process exited with code -9 | - | | 2026-09-15 | `test_disaggregation_pp.py` — timeout after 1200s | - | | 2026-09-15 | `test_disaggregation_rust_server.py` — RuntimeError: Server process exited with code -9 | - | | 2026-09-15 | `test_disaggregation_unified_memory.py` — RuntimeError: Server process exited with code -9 | - | | 2026-09-15 | `test_draft_entry_hook_parity.py` — AssertionError: Lists differ: [] != ['BailingMoeForCausal... | - | | 2026-09-15 | `test_dsa_glm52_hisparse.py` — | - | | 2026-09-15 | `test_dsa_glm52_nvfp4_dp_mtp.py` — | - | | 2026-09-15 | `test_dsa_pool_host_unit.py` — ImportError: cannot import name 'get_device_accessible_pt... | - | | 2026-09-15 | `test_e2e_pd.py` — RuntimeError: Server process exited with code -9 | - | | 2026-09-15 | `test_epd_disaggregation.py` — TimeoutError: Endpoint http://127.0.0.1:11200/health did ... | - | | 2026-09-15 | `test_ffn_model_loaders.py` — ModuleNotFoundError: No module named 'vllm' | - | | 2026-09-15 | `test_flashinfer_a2a.py` — | - | | 2026-09-15 | `test_flux2_gated_resnorm.py` — | - | | 2026-09-15 | `test_gdn_prefill_backend_policy.py` — AttributeError: 'GDNAttnBackend' object has no attribute ... | - | | 2026-09-15 | `test_generation_auxiliary_output.py` — AttributeError: 'types.SimpleNamespace' object has no att... | - | | 2026-09-15 | `test_glm4_moe_models.py` — | - | | 2026-09-15 | `test_glm53_flash_b200.py` — | - | | 2026-09-15 | `test_glm53_flash_h200.py` — AssertionError: 0.928 not greater than or equal to 0.93 | - | | 2026-09-15 | `test_hicache.py` — ImportError: cannot import name 'get_device_accessible_pt... | - | | 2026-09-15 | `test_hicache_page_first_write_back.py` — ImportError: cannot import name 'get_device_accessible_pt... | - | | 2026-09-15 | `test_hicache_spec_file_storage.py` — | - | | 2026-09-15 | `test_hicache_spec_mooncake_storage.py` — | - | | 2026-09-15 | `test_hicache_storage.py` — | - | | 2026-09-15 | `test_hicache_storage_file_backend.py` — | - | | 2026-09-15 | `test_hicache_storage_lora.py` — | - | | 2026-09-15 | `test_hicache_storage_runtime_attach_detach.py` — | - | | 2026-09-15 | `test_hiradix_cache_unit.py` — TypeError: cudaGetErrorString(): incompatible function ar... | - | | 2026-09-15 | `test_hisparse_unit.py` — ImportError: cannot import name 'get_device_accessible_pt... | - | | 2026-09-15 | `test_inkling_small_nvfp4.py` — | - | | 2026-09-15 | `test_kda_fused_verify_dispatch.py` — TypeError: KDAAttnBackend._can_run_fused_chain_verify() m... | - | | 2026-09-15 | `test_kimi_k3_b300.py` — | - | | 2026-09-15 | `test_lora_qwen3_vl_30b_a3b_instruct_logprob_diff.py` — exit code -9 | - | | 2026-09-15 | `test_mamba_path_state_cap.py` — TypeError: cudaGetErrorString(): incompatible function ar... | - | | 2026-09-15 | `test_mimo_v2.py` — | - | | 2026-09-15 | `test_mimo_v2_flash.py` — | - | | 2026-09-15 | `test_minimax_sparse_pool_host_unit.py` — ImportError: cannot import name 'get_device_accessible_pt... | - | | 2026-09-15 | `test_mla_fp8.py` — AssertionError: 0.0 not greater than or equal to 0.8 | - | | 2026-09-15 | `test_mooncake_transfer_batching.py` — AttributeError: 'types.SimpleNamespace' object has no att... | - | | 2026-09-15 | `test_nixl_backend_basic.py` — AttributeError: 'NixlKVManager' object has no attribute '... | - | | 2026-09-15 | `test_openai_function_calling.py` — | - | | 2026-09-15 | `test_pp_single_node.py` — | - | | 2026-09-15 | `test_pp_with_hicache.py` — | - | | 2026-09-15 | `test_prefill_adder.py` — AttributeError: 'types.SimpleNamespace' object has no att... | - | | 2026-09-15 | `test_processor_async_call_sites.py` — AssertionError: these processors reach preprocessing with... | - | | 2026-09-15 | `test_quantized_kv_pool.py` — AttributeError: '_FakeQuantMethod' object has no attribut... | - | | 2026-09-15 | `test_qwen35_hicache.py` — | - | | 2026-09-15 | `test_qwen4_exp_models.py` — | - | | 2026-09-15 | `test_rust_unified_radix_cache_unittest.py` — TypeError: cudaGetErrorString(): incompatible function ar... | - | | 2026-09-15 | `test_scripted_runtime_core.py` — AssertionError: scripted-runtime script failed: | - | | 2026-09-15 | `test_self_e2e_pd_baseline.py` — RuntimeError: Server process exited with code -9 | - | | 2026-09-1