CI: replace explicit example skips with hermetic smoke tests
Author: richardoo-707Created Sep 25, 2026Updated Sep 26, 2026
Context
Several example directories had empty test_ci.sh files. Bash exits successfully for an empty script, so PR and scheduled example checks reported green without running a test.
The empty-script fix adds real smoke tests where the examples can run deterministically. The following examples remain explicitly skipped until they can satisfy the same contract:
examples/language/deepseek: even the tiny configuration eagerly callsAutoConfig.from_pretrained(..., trust_remote_code=True), making CI depend on network access, the Hugging Face cache, and remote code.examples/community/roberta: the example requires external preprocessed data, tokenizer/config assets, multi-host setup, and legacy ColossalAI APIs.examples/inference/benchmark_ops: the entry points are full performance grids with warmups/repetitions and do not expose a bounded correctness-oriented smoke mode.
Acceptance criteria
- Each skipped directory has a deterministic smoke command that performs meaningful work and fails on regressions.
- The smoke test does not download model weights, datasets, tokenizers, or executable remote code.
- It runs in the example CI image and completes within five minutes on at most two GPUs.
- It uses synthetic/local fixtures and returns non-zero when the example is broken.
- Replace the corresponding explicit SKIP script with the smoke command.
Until these conditions are met, CI must print an explicit warning linked to this issue instead of silently succeeding.
Source: hpcaitech/ColossalAI