The Alpine Linux CI Runner does not pass the ZFS Test Suite (yet)

Recently I have been working on fixing the remaining failing tests on the Alpine Linux CI runner. With the current master branch (HEAD at aa26ca67b1b7663ad4d08cdddfbe3352b688b6c4) I get the following:

Tests with results other than PASS that are unexpected:
    FAIL cli_root/zfs_get/zfs_get_006_neg (expected PASS)
    FAIL rsend/send-c_stream_size_estimate (expected PASS)
    SKIP cli_root/zfs_get/zfs_get_009_pos (expected PASS)

A few issues need to be discussed before the Alpine runner can be finished and enabled in the default test matrix. I already have a branch whose patches make the Alpine CI run fully green.

I used Claude Code extensively for this. I mirrored the CI runner in a local VM, reproduced each failure there, and worked through the triage with it; the commits in my testing branch were written by Claude. I treat its output as diagnostic data to be checked, not as an answer, and where I could check the reasoning myself I did. What is left in this issue is what that work could not settle — design questions for the project rather than things I can decide in a branch, plus one item in section 4 that is past what I can verify.

Where things stand

Two branches on my fork, both on top of current master:

• alpine/baseline — https://github.com/alex-moch/zfs/tree/alpine/baseline: master plus a single commit that trims the workflow set and the OS list so iteration is cheap.
• alpine/combined — https://github.com/alex-moch/zfs/tree/alpine/combined: the same, plus the six commits discussed below.

Latest full runs, deliberately run side by side so every result has a control to compare against:

• alpine/baseline — https://github.com/alex-moch/zfs/actions/runs/33850693881
• alpine/combined — https://github.com/alex-moch/zfs/actions/runs/33851157795

The alpine/combined run is green on all six platforms, including alpine3-24; the alpine/baseline run is red on alpine3-24 alone. So the runner itself is in good shape. What follows is about how to land the pieces. Everything below is already implemented and pushed to my fork.

1. ksh93 on Alpine

Commit: https://github.com/alex-moch/zfs/commit/c5c511b1d642a92ea440cf6aa884e9e999332e3a

ksh93 is not available in the Alpine repositories. Currently the CI runner builds it from the upstream 1.0 branch. That branch is under active development, so any regression there hits the CI runner immediately. Options as I see them:

1. Pull a release tarball or a tag from upstream ksh93 and build that. Mirrors current behaviour, but still spends CI minutes on the build every run.
2. Install a prebuilt Alpine package. I have built one from the plain upstream v1.0.10 tag, no patches, and published it as a release. This is what alpine/combined currently does, and it removes the build entirely.

I want to be upfront about option 2 rather than slip it past review. I work in IT security and have watched supply-chain attacks land through exactly this shape of dependency, so I am raising it for discussion instead of arguing for it.

The mitigating facts, for what they are worth: this only ever runs inside the GitHub-hosted runner VM, the repository is public, and there are no secrets in that environment for a compromised binary to exfiltrate. The blast radius is a poisoned test result, not a credential leak.

2. is_kmemleak() reports a compiled-in kmemleak as a running one

Commit: https://github.com/alex-moch/zfs/commit/984dc00c7ee993bb62e7f2a4858b7dd11bf062c2

Commit 06d334b9e5b69a3336133a2aa02f69483264e2d9 dropped this test's expected-SKIP mask, so a skip that was already happening on Alpine now shows up as unexpected.

The cause is is_kmemleak(). It tests for /sys/kernel/debug/kmemleak, but the kernel creates that file before it checks whether the detector actually came up. Alpine's linux-stable kernel has kmemleak compiled in but disabled at boot, so the file is there and the detector is not.

A fix is in my branch. I just need to review it and get it PR-ready; as this is all very recent, I have not done that yet.

3. zfs_get_006_neg and option permutation — five options, none free

Commit: https://github.com/alex-moch/zfs/commit/f32ec0a82b370d54c3dcee841029c7dd2322d025

The test requires zfs get all -r to fail, and arranges that with export POSIXLY_CORRECT=1. glibc's getopt_long() honours that variable and stops permuting. musl's ignores it entirely, by design — permutation is decided solely by the first character of the optstring. None of the six getopt_long() call sites in zfs_main.c uses a leading +, so on musl zfs get all -r is accepted and the test fails.

Options as I see them:

1. + on all six optstrings. This is the change I have in my testing branch to confirm it is the root cause. It makes the test pass on both libcs. Drawback: it changes behaviour for everyone, not just Alpine. zfs list pool -r works today and would stop working, so any script that puts a flag after a positional argument breaks.

2. + on zfs_do_get only. The same change limited to the one call site a test actually covers, so far fewer commands change behaviour. Drawback: zfs get would then reject flags after positionals while zfs list, zfs send, zfs mount, zfs share, zfs program and zfs version still accept them, for no reason a user could guess.

3. Honour POSIXLY_CORRECT ourselves, supplying the + only when it is set:

   c复制

   #define    ZFS_OPTSTR(s)    (getenv("POSIXLY_CORRECT") != NULL ? "+" s : s)

   This is the only option that changes nothing user-visible and needs no edit to the test. Drawback: it hand-implements a libc convention in application code at six call sites purely to satisfy a test. And it changes nothing for real users: almost nobody sets POSIXLY_CORRECT, so zfs get all -r is still accepted in normal use.

4. Mask the test on musl in zts-report.py.in, so the failure is recorded as expected instead of breaking the run. No C code changes at all. It needs a new scoping axis, though: sys.platform is linux on Alpine, so the existing split cannot tell musl from glibc; platform.libc_ver() returns ('musl', '1') there and is stdlib only. Drawback: a masked entry looks the same in the report as a real defect. If zfs get's argument handling ever genuinely breaks on musl, this test would fail for that reason too and the mask would still report it as expected, so nothing about the actual difference gets fixed and the test stops being a useful check on Alpine.

5. Drop the argument-ordering cases from the test. Ten of the thirty bad_combine entries put a valid option after the property list. They come from the Solaris original, where getopt(3) never permuted, so they failed there naturally. On Linux they only fail because POSIXLY_CORRECT pushes glibc back into behaving the way Solaris did. Nothing in ZFS decides the outcome: build the same binary against a different C library and the verdict flips. That makes these ten cases a test of libc rather than of zfs get. The remaining twenty fail on their own merits on any libc — invalid options, invalid property lists, nonexistent datasets. On that reading the test is asserting something it was never meant to assert, and removing those cases is a fix rather than a workaround. Drawback: this assumes zfs get all -r is fine to accept. If it is meant to be rejected, then the test was checking something real, and deleting these cases throws away a genuine check while leaving zfs get accepting a command it should refuse.

One residual that none of these closes: zfs_main.c has 22 plain getopt() call sites alongside the six getopt_long() ones, and they diverge in the opposite direction — musl's getopt() never permutes, glibc's does. So zfs destroy tank/fs -r works on glibc and fails on musl today, and stays that way whatever is decided above.

None of these feels like the right answer to me. The first three change C code to satisfy a test without fixing anything that is actually broken for users, and option 1 — the one currently in my testing branch — changes behaviour that users may well depend on. That is why I would rather have direction here than pick one myself.

4. send-c_stream_size_estimate — needs review from someone who knows the internals

Commits: https://github.com/alex-moch/zfs/commit/78e3deaff43b8f98f74abd5db3046157da54c152, https://github.com/alex-moch/zfs/commit/2cd0b5f908b808884beaea8f322807023105812e, https://github.com/alex-moch/zfs/commit/8201d2ffce02c3a79d91ad55db3017bd8a336931

I did not write or verify any of this. The diagnosis and the commits are Claude's, and the reasoning is in the commit messages rather than repeated here. I do not know the send path well enough to confirm it, so I am leaving it as diagnostic data in the hope that someone who does will pick it up.

What I can say is only that the test fails on alpine/baseline and passes on alpine/combined. Whether the explanation behind that is correct is exactly the part I cannot judge.

Of the three commits, only the first one fixes the failing test. It changes how lzc_send_wrapper()'s relay handles the output file position during a dry run. The other two are separate bugs in the same function, found while investigating this one and not needed to make the Alpine run green: the second handles a destination opened with O_APPEND (zfs send >> file), which the relay could not write to at all, and the third stops a failing destination from killing the send with SIGPIPE instead of reporting the error.