Four kernel faults in ZFS across write, read, and buffer-teardown paths — count/pointer fields read as garbage (spa_taskq_dispatch x2, LZ4_uncompress infinite loop, memcg deref under dbuf_evict), ZFS 2.3.0 and 2.3.4

System information

Summary

Four kernel faults over eleven days, all in ZFS code paths, on two different OpenZFS versions and two different kernels. Two are fatal faults in spa_taskq_dispatch() on the write path; one is an unrecoverable infinite loop in the LZ4 decompressor on the read path; the fourth is a fatal fault during ARC buffer teardown (dbuf_evict → arc_buf_destroy → spl_slab_reclaim → vfree), where the mem_cgroup pointer derived from a page being freed held the value 0x80. In every case a field that should hold a small integer, count, or valid pointer instead holds garbage — twice demonstrably the same two struct fields.

All four were captured over netconsole. Nothing reached disk, because a panicking or soft-locked kernel cannot sync.

For completeness: this machine also had four earlier unclean resets (2026-07-25 to 2026-08-08) with no log trail at all, before netconsole was in place. We do not claim they are this bug — they are uncharacterised, and at least one earlier crash series on this box had a different, separately root-caused mechanism. Only the three captures below are offered as evidence.

Event 1 — 2026-08-13, OpenZFS 2.3.0-1, kernel 6.12.15 — page fault, write path

Uptime at fault: 432347 s (5d 0h 06m).

BUG: unable to handle page fault for address: 00000000c2c51028
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0
Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI
CPU: 15 UID: 0 PID: 439082 Comm: z_wr_int_0 Tainted: P  OE  6.12.15-production+truenas #1
Hardware name: Micro Computer (HK) Tech Limited MS-02-Ultra/ARWSA, BIOS 1.03 10/22/2025
RIP: 0010:spa_taskq_dispatch+0x94/0xb0 [zfs]
Code: 00 4c 89 e6 5d c1 e1 1b 41 5c 41 5d 41 5e e9 c3 3a e0 ff 4c 8b b3 80 01 00 00 e8 47
      76 ff ff 8b 8b 78 01 00 00 31 d2 48 f7 f1 <49> 8b 3c d6 eb be 48 8b 83 80 ...
RAX: 0000000000020a7b RBX: ffff98c082a394e0 RCX: 00000000c0aa3570
RDX: 0000000020435bcb RSI: 0000000000000053 RDI: ffffffffbe0303e0
RBP: ffff98de5d74df00 R08: ffff98de5d74df00 R09: 0000000000000000
R10: ffff98c019c153f0 R11: 0000000000550011 R12: ffffffffc0b1a1d0
R13: 0000000000000000 R14: ffffffffc0aa31d0 R15: ffff98de5d74b440
CR2: 00000000c2c51028
Call Trace:
 <TASK>
 zio_done+0x6e0/0xd90 [zfs]
 zio_execute+0x92/0xe0 [zfs]
 taskq_thread+0x253/0x4f0 [spl]
 kthread+0xcf/0x100
 ret_from_fork+0x31/0x50
 ret_from_fork_asm+0x1a/0x30
 </TASK>
Kernel panic - not syncing: Fatal exception

Event 2 — 2026-08-16, OpenZFS 2.3.4-1, kernel 6.12.99 — divide error, write path

Uptime at fault: 259426 s (3d 0h 04m). Same function, after upgrading 2.3.0 → 2.3.4.

Oops: divide error: 0000 [#1] PREEMPT SMP NOPTI
CPU: 11 UID: 0 PID: 2629239 Comm: z_wr_int_3 Tainted: P  OE  6.12.99-production+truenas #1
RIP: 0010:spa_taskq_dispatch+0x91/0xb0 [zfs]
Code: 38 04 00 00 4c 89 e6 5d c1 e1 1b 41 5c 41 5d 41 5e e9 23 5f 9f ff 4c 8b b3 80 01 00
      00 e8 d7 78 ff ff 8b 8b 78 01 00 00 31 d2 <48> f7 f1 49 8b 3c d6 eb be 48 ...
RAX: 0000ebf27b907816 RBX: ffff8b8de33c3ba0 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 00000000000000ee RDI: ffffffff97630400
RBP: ffff8b8fb37ee880 R08: ffff8b8fb37ee880 R09: 0000000000000001
R10: 0000000000000003 R11: 00000000000000ee R12: ffffffffc09e8700
R13: 0000000000000001 R14: 0000000000000000 R15: ffff8b8fb37ee880
Call Trace:
 <TASK>
 zio_vdev_io_start+0xdb/0x290 [zfs]
 zio_execute+0x92/0xe0 [zfs]
 vdev_queue_io_done+0x9a/0x160 [zfs]
 zio_vdev_io_done+0x110/0x1c0 [zfs]
 zio_execute+0x92/0xe0 [zfs]
 taskq_thread+0x253/0x4f0 [spl]
 kthread+0xcf/0x100
 ret_from_fork+0x31/0x50
 ret_from_fork_asm+0x1a/0x30
 </TASK>
Kernel panic - not syncing: Fatal exception

Analysis of events 1 and 2 — the same two struct fields, wrong both times

Decoding the Code: bytes, the faulting sequence is:

mov  r14, [rbx+0x180]     ; taskq array pointer   (believed stqs_taskq)
mov  ecx, [rbx+0x178]     ; taskq count           (believed stqs_count)
xor  edx, edx
div  rcx                  ; <-- event 2 faults here (#DE), RCX = 0
mov  rdi, [r14+rdx*8]     ; <-- event 1 faults here (#PF), bad index

• Event 1 survived the divide and faulted on the index. RCX = 0xc0aa3570, which is the low half of a module pointer: R14 = 0xffffffffc0aa31d0, only 0x3a0 away. A pointer-shaped value sitting in an integer count field.
• Event 2 faulted on the divide itself: RCX = 0 — and R14 = 0 as well.

⇒ Same struct, same field pair, garbage in both, across two OpenZFS versions and two kernels, in the write path both times.

Our hypothesis, offered as a lead rather than a finding: a taskq lifecycle race rather than stray corruption. A count of 0 together with a NULL pointer is the all-zeroes shape of a struct that has been freed, zeroed, or never populated — not the shape of random corruption. OpenZFS 2.2+ added dynamically-sized ZIO taskqs; a dispatch for a (type, priority) whose taskq set is torn down or not yet populated would produce exactly event 2, and event 1 looks like the same struct caught mid-teardown.

We have not confirmed that +0x178/+0x180 are stqs_count/stqs_taskq in the shipped zfs.ko — someone with the matching build should verify that before giving our reading any weight.

Event 3 — 2026-08-19, OpenZFS 2.3.4-1, kernel 6.12.99 — infinite loop, read path

Uptime at fault: 259623 s (3d 0h 07m). Not a panic — one CPU looped forever and the machine had to be power-cycled after 40 minutes.

watchdog: BUG: soft lockup - CPU#7 stuck for 157s! [IoThread:3375]
CPU: 7 UID: 0 PID: 3375 Comm: IoThread Tainted: P  OEL  6.12.99-production+truenas #1
Hardware name: Micro Computer (HK) Tech Limited MS-02-Ultra/ARWSA, BIOS 1.03 10/22/2025
RIP: 0010:LZ4_uncompress_unknownOutputSize+0x5be/0x860 [zfs]
Code: 59 02 00 00 48 83 fa 0f 0f 86 6c 02 00 00 48 8b 55 00 48 83 c3 20 48 83 c5 20 48 89
      53 e0 48 8b 55 e8 48 89 53 e8 48 8b 55 f0 <48> 89 53 f0 48 8b 55 f8 48 89 53 f8 48
      39 c3 72 d3 e9 b7 fa ff ff
RAX: ffffd0b1b9fff183 RBX: ffffd0b1b9fff00e RCX: ffffd0b1b9ffef6e
RDX: 61622f6e69622f3d RSI: ffffd0b1ba019000 RDI: 0000000000001bda
RBP: ffffd0b1b9ffd434 R08: ffffd0b1b9ff9000 R09: 0000000000000215
R10: ffffd0b1ba018fc0 R11: 000000000000000f R12: ffffd0b1b9fff183
R13: ffff8edf3d523a08 R14: ffffd0b1b9ff9000 R15: ffff8edf3d525630
Call Trace:
 <TASK>
 zfs_lz4_decompress_buf.constprop.0+0x2b/0x40 [zfs]
 zfs_lz4_decompress+0x4d/0x90 [zfs]
 ? abd_init_struct+0x40/0x50 [zfs]
 arc_buf_fill+0x26f/0x4c0 [zfs]
 arc_read+0x1502/0x1770 [zfs]
 ? __pfx_dbuf_read_done+0x10/0x10 [zfs]
 dbuf_read_impl.constprop.0+0x27f/0x380 [zfs]
 dbuf_read+0x2af/0x5d0 [zfs]
 dmu_buf_hold_array_by_dnode+0x118/0x5e0 [zfs]
 dmu_read_uio_dnode+0x74/0x150 [zfs]
 zfs_read+0x25e/0x530 [zfs]
 zpl_iter_read+0xb7/0x120 [zfs]
 vfs_read+0x244/0x370
 __x64_sys_pread64+0x98/0xd0
 do_syscall_64+0x87/0x1a0
 entry_SYSCALL_64_after_hwframe+0x76/0x7e
 </TASK>

Analysis of event 3

• The lockup was re-reported 11 times over 36 minutes (157 s → 2191 s stuck) and the register dump is byte-identical in every report — same RIP offset, same RAX, RBX, RCX. RBX is the loop's advancing cursor (cmp rbx,rax; jb) and it never advanced. This is a genuinely non-terminating loop, not slow progress.
• RDX holds 0x61622f6e69622f3d, i.e. the ASCII bytes =/bin/ab — plain text occupying a register the decompressor is treating as data.
• Userspace caller: middlewared performing a 4096-byte pread64. The machine hosts no VMs or containers (virt.instance.query → []); this was ordinary middleware file I/O, not a heavy workload.
• Collateral: txg_sync on both pools and a systemd task blocked for 120–483 s; every process touching the pool hung, including new SSH logins. Writes later resumed; the looping thread never did, and it could not be signalled — a graceful reboot issued from the web UI never started, and the box had to be hard power-cycled.

The point of interest: ZFS verifies the block checksum before handing the buffer to arc_buf_fill() for decompression. So either a buffer that passed its checksum decompressed as garbage, or arc_buf_fill() was handed the wrong buffer or length. Separately, LZ4_uncompress_unknownOutputSize() appears able to loop indefinitely on malformed input rather than returning an error — arguably a hardening bug in its own right, whatever delivered the bad input.

Event 4 — 2026-08-24, OpenZFS 2.3.4-1, kernel 6.12.99 — NULL-adjacent deref, ARC eviction/free path

Uptime at fault: 33976 s (9h 26m). Fault at 06:36:57 local; auto-reboot 10 s later.

BUG: kernel NULL pointer dereference, address: 00000000000005c8
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0
Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI
CPU: 22 UID: 0 PID: 506 Comm: dbuf_evict Tainted: P  OE  6.12.99-production+truenas #1
Hardware name: Micro Computer (HK) Tech Limited MS-02-Ultra/ARWSA, BIOS 1.03 10/22/2025
RIP: 0010:__mod_memcg_state+0x27/0x130
Code: 90 90 90 66 0f 1f 00 0f 1f 44 00 00 0f 1f 44 00 00 48 63 c6 55 0f b6 80 40 ae 26 8a
      53 3c ff 0f 84 e3 00 00 00 89 d3 48 89 fd <48> 8b 97 48 05 00 00 48 63 cb 65 48 01
      4c c2 18 83 fe 25 0f 84 bb
RSP: 0018:ffffceb64140bd78 EFLAGS: 00010013
RAX: 0000000000000022 RBX: 00000000ffffffff RCX: 0000000000000000
RDX: 00000000ffffffff RSI: 0000000000000032 RDI: 0000000000000080
RBP: 0000000000000080 R08: 0000000000000000 R09: 0000000000000147
R10: 00000000000007e0 R11: 00000000002a0029 R12: fffff0e0efe5f4c0
R13: ffff8b9c632a0490 R14: ffff8b9c632a04a8 R15: ffffffffc02d1340
CR2: 00000000000005c8 CR3: 0000001cbaf4a004 CR4: 0000000000f72ef0
Call Trace:
 <TASK>
 mod_memcg_state.constprop.0+0x1c/0x30
 vfree.part.0+0xa7/0x240
 spl_slab_reclaim+0xf3/0x140 [spl]
 arc_buf_destroy_impl+0x41/0x140 [zfs]
 arc_buf_destroy+0x69/0xe0 [zfs]
 dbuf_destroy+0x2e/0x2e0 [zfs]
 dbuf_evict_one+0x11f/0x160 [zfs]
 dbuf_evict_thread+0xa2/0x100 [zfs]
 thread_generic_wrapper+0x5a/0x70 [spl]
 kthread+0xcf/0x100
 ret_from_fork+0x31/0x50
 ret_from_fork_asm+0x1a/0x30
 </TASK>
Kernel panic - not syncing: Fatal exception

Analysis of event 4

• The faulting instruction is mov rdx, [rdi+0x548] with RDI = 0x80: the mem_cgroup pointer passed to __mod_memcg_state() held the value 0x80 (CR2 = 0x80 + 0x548 = 0x5c8). RSI = 0x32 (the stat item) and RDX = 0xffffffff (a −1 delta) look normal; only the memcg pointer is garbage.
• The memcg pointer on this path derives from the page being vfree()d by spl_slab_reclaim() during ARC dbuf eviction — i.e. the page's memcg_data (or the vm area's accounting) held a small flag-like integer where a pointer belongs. R12 = fffff0e0efe5f4c0 is consistent with a struct page in the vmemmap region.
• 0x80 is not a plausible bitflip of any valid kernel pointer (they differ in dozens of bits), matching the pattern of events 1–3: logically wrong values, not decayed ones.
• This is the first of the four on the free/teardown path; the thread is ZFS's own dbuf-cache eviction thread.
• Workload context, offered without causal claim: for the ~12 minutes before the fault, a remote backup client had been sequentially reading one dataset over an NFSv4 read-only mount at a modest ~11 MiB/s. Sustained reads do churn the dbuf cache (whence dbuf_evict activity), but the same nightly job performs the same reads without incident on other days.
• Unlike events 1–3, this fault arrived only 9h 26m after boot — the box was on a nightly-reboot regime by this date, which the fault therefore defeats.

What is common to all four

1. All four are a length, count, or pointer field holding a value it cannot legitimately hold — and in every case the garbage is logically shaped (a pointer half in a count, an all-zeroes struct, ASCII text as data, a flag-like small int as a pointer), not single-bit decay.
2. They span the write path, the read path, and the eviction/free path, and two OpenZFS versions (2.3.0, 2.3.4).
3. Events 1–3 arrived a whole number of days after boot, +0–7 minutes (5d, 3d, 3d). We noted that as an odd empirical pattern; event 4 (9h 26m) breaks it, so we no longer offer it as predictive — recorded only for completeness.
4. No pool damage: zpool status reports ONLINE, 0 0 0, "No known data errors" after each event, and scrubs repair 0 B.

What we have ruled out

• Transparent huge pages — an earlier, separately root-caused crash series on this same box was THP colliding with ARC. THP has read [never] throughout, verified after every upgrade, with compact_stall and compact_daemon_wake both 0.
• Power — the machine is fed through a metered smart plug logging every 6–10 s. Across one of these events voltage never left 241.6–245.3 V, and power rose (the boot signature) rather than dipping. Two identical machines on the same UPS output have zero unclean resets over the same period. Sub-second transients remain formally unexcluded.
• Disks — SMART clean on all four NVMe devices; zero lifetime minutes above warning temperature; no nvme error has ever appeared on the netconsole wire; both legs of a mirror are never independently at fault.
• Thermal — CPU ~56 °C, NVMe 50–53 °C, NIC 56 °C under sustained load, against limits of 95/105 °C.
• Memory leak or exhaustion — three days of continuous recorder history covering the run that ended in event 3: ARC steady within a ~0.5 GB band (121.4 → 119.1 GB), total memory usage flat at 94–96%. ARC is the elastic consumer here, so a leak elsewhere would have forced ARC eviction; it did not.

Known limitation, stated up front

The RAM is non-ECC (Error Correction Type: None), so we cannot categorically exclude a memory fault. What argues against it: memtest has been run repeatedly across this machine's history and has always passed; and events 1 and 2 put garbage into the same two struct fields of the same function on two different kernels and two different module builds — a pattern random bitflips do not reproduce. We would rather state this plainly than have it surface later.

Reproduction

No reliable reproducer. The faults occur under ordinary mixed NFS/SMB load, and there is no known trigger we can invoke on demand. The boot-relative cadence seen in events 1–3 did not survive event 4, so we currently have no predictive signal at all.

What we can supply on request

• Full unabridged netconsole captures for all four events, including complete module lists.
• The exact zfs.ko and kernel build in use, for stqs_count / stqs_taskq offset verification.
• zpool history -il, pool and dataset properties, and scrub results before and after each event.
• Forensics from a separate single-block corruption on this pool, dated by birth txg via zdb -ddddd, where both mirror legs held identical bad data from birth — i.e. the buffer went bad in memory between checksum and mirror split. We are not claiming it is the same bug; it is offered in case it proves relevant.