Permanent errors in metadata following 2.3.0 upgrade
Author: putnamCreated Feb 24, 2025Updated Sep 10, 2026
LabelsType: Defect
System information
| Type | Version/Name |
|---|---|
| Distribution Name | Debian |
| Distribution Version | trixie |
| Kernel Version | 6.12.12 |
| Architecture | amd64 |
| OpenZFS Version | zfs-2.3.0-1 zfs-kmod-2.3.0-1 |
Describe the problem you're observing
Now that 2.3.0 was merged into trixie, I just upgraded my system to 2.3.0 (from the 2.2.x series). Before rebooting, the pool was clean, having finished a scrub a couple of weeks ago. After reboot, I got an email from zed letting me know a resilver had occurred at boot and multiple permanent errors were present.
The pool is comprised of 6 vdevs, each 11 disks in raidz2. Compression and encryption are enabled. I am using a dedicated L2ARC device, a single SSD. secondarycache is set to metadata.
# zpool status tank -v
pool: tank
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
scan: resilvered 0B in 00:00:02 with 0 errors on Mon Feb 24 15:30:31 2025
< snip >
errors: Permanent errors have been detected in the following files:
<metadata>:<0x0>
<metadata>:<0x41827>
<metadata>:<0x34>
<metadata>:<0x3c>
<metadata>:<0xbe>
<metadata>:<0xc9>
Inspecting the "corrupt" metadata nodes:
# zdb -dd tank 0x0
Dataset mos [META], ID 0, cr_txg 4, 4.69G, 64926 objects
Object lvl iblk dblk dsize dnsize lsize %full type
0 3 128K 16K 255M 512 192M 16.51 DMU dnode
# zdb -dd tank 0x41827
Dataset mos [META], ID 0, cr_txg 4, 4.68G, 64840 objects
Object lvl iblk dblk dsize dnsize lsize %full type
268327 1 16K 128K 110K 512 128K 100.00 SPA space map
# zdb -dd tank 0x34
Dataset mos [META], ID 0, cr_txg 4, 4.68G, 64846 objects
Object lvl iblk dblk dsize dnsize lsize %full type
52 1 128K 1.50K 27.5K 512 1.50K 100.00 zap
# zdb -dd tank 0x3c
Dataset mos [META], ID 0, cr_txg 4, 4.68G, 64876 objects
Object lvl iblk dblk dsize dnsize lsize %full type
60 3 128K 128K 305M 512 710M 100.00 SPA history
# zdb -dd tank 0xbe
Dataset mos [META], ID 0, cr_txg 4, 4.68G, 64884 objects
Object lvl iblk dblk dsize dnsize lsize %full type
190 2 128K 16K 5.68M 512 4.88M 67.31 zap
# zdb -dd tank 0xc9
Dataset mos [META], ID 0, cr_txg 4, 4.68G, 64908 objects
Object lvl iblk dblk dsize dnsize lsize %full type
201 1 128K 128K 27.5K 512 128K 100.00 uint64
Exactly how concerned should I be here? I have been waiting for a long time for the 2.3.0 release and kind of worried I may have rushed into it today.
Source: openzfs/zfs