ZFS Non-boot zpool won't import on boot

jem · 2025-01-22T22:10:21+0000

I have built a 14.2-RELEASE system that boots from a UFS filesystem on an SSD, but also has four other hard disks attached containing a ZFS raidz pool.

I have the required variables set in the right files to start ZFS on boot:

Code:

# grep zfs /boot/loader.conf /etc/rc.conf
/boot/loader.conf:zfs_load="YES"
/etc/rc.conf:zfs_enable="YES"

However, after booting, the raidz pool hasn't imported and the datasets aren't mounted. I can manually run a zpool import and everything then works fine.

I found this comment on an earlier thread stating that a new /etc/rc.d/zpool script was added to resolve this, but it doesn't seem to be working for me. I wondered if the cachefile property on the pool being set to its default value of '-' might be the issue, so I tried changing it to cachefile=/etc/zfs/zpool.cache but it made no difference on the next boot and reverted to '-'.

Can anyone advise how I can fix this, besides adding a zpool import command to /etc/rc.local?

atax1a · 2025-01-22T23:09:56+0000

Try zpool import -o cachefile=/etc/zfs/zpool.cache poolname — i think it only updates the cache on import and export.

lowkeyloki · 2025-01-23T01:05:41+0000

doh! I can't read. Sorry.

cy@ · 2025-01-23T01:30:32+0000

jem said:
I have built a 14.2-RELEASE system that boots from a UFS filesystem on an SSD, but also has four other hard disks attached containing a ZFS raidz pool.

I have the required variables set in the right files to start ZFS on boot:

Code:

# grep zfs /boot/loader.conf /etc/rc.conf /boot/loader.conf:zfs_load="YES" /etc/rc.conf:zfs_enable="YES"

However, after booting, the raidz pool hasn't imported and the datasets aren't mounted. I can manually run a zpool import and everything then works fine.

I found this comment on an earlier thread stating that a new /etc/rc.d/zpool script was added to resolve this, but it doesn't seem to be working for me. I wondered if the cachefile property on the pool being set to its default value of '-' might be the issue, so I tried changing it to cachefile=/etc/zfs/zpool.cache but it made no difference on the next boot and reverted to '-'.

This was added when ZFS (Illumos upstream) was replaced by OpenZFS. OpenZFS doesn't do an implicit zpool import (in the kernel) like the old Illumos ZFS did.

jem said:
Can anyone advise how I can fix this, besides adding a zpool import command to /etc/rc.local?

Your /etc/zfs/zpool.cache isn't being updated at zpool import.

What kind of fs is rootfs?

jem · 2025-01-23T07:22:42+0000

cy@ said:
Your /etc/zfs/zpool.cache isn't being updated at zpool import.

I'm pretty certain it is. After my manual zpool import pool0, running zdb -C -U /etc/zfs/zpool.cache is returning correct information for my pool:

Code:

[root@filer2 ~]# zdb -C -U /etc/zfs/zpool.cache
pool0:
    version: 5000
    name: 'pool0'
    state: 0
    txg: 48987
    pool_guid: 4536089192970494829
    errata: 0
    hostid: 726799574
    hostname: 'filer2'
    com.delphix:has_per_vdev_zaps
    vdev_children: 1
    vdev_tree:
        type: 'root'
        id: 0
        guid: 4536089192970494829
        create_txg: 4
        com.klarasystems:vdev_zap_root: 129
        children[0]:
            type: 'raidz'
            id: 0
            guid: 3438085862275244924
            nparity: 1
            metaslab_array: 256
            metaslab_shift: 34
            ashift: 12
            asize: 8001576501248
            is_log: 0
            create_txg: 4
            com.delphix:vdev_zap_top: 130
            children[0]:
                type: 'disk'
                id: 0
                guid: 14336507559410368910
                path: '/dev/da2'
                devid: 'ata-SAMSUNG_HD204UI_S2H7J90B309265'
                phys_path: 'pci-0000:06:00.0-sas-phy13-lun-0'
                whole_disk: 1
                DTL: 779
                create_txg: 4
                com.delphix:vdev_zap_leaf: 131
            children[1]:
...
            children[2]:
...
            children[3]:
...

My root filesystem is on a simple 'freebsd-ufs' GPT partition:

Code:

[root@filer2 ~]# gpart show
=>      40  41942960  vtbd0  GPT  (20G)
        40      2008         - free -  (1.0M)
      2048    262144      1  efi  (128M)
    264192   4194304      2  freebsd-swap  (2.0G)
   4458496  37484504      3  freebsd-ufs  (18G)

[root@filer2 ~]# mount
/dev/vtbd0p3 on / (ufs, local, soft-updates)
devfs on /dev (devfs)
/dev/vtbd0p1 on /boot/efi (msdosfs, local)
tmpfs on /tmp (tmpfs, local)

vtbd0 is a virtio block device. The system is a VM running under qemu/kvm, and the four hard disks are passed through from the hypervisor as raw SCSI devices, but are detected fine during boot:

Code:

da0 at vtscsi0 bus 0 scbus2 target 0 lun 0
da0: <QEMU QEMU HARDDISK 2.5+> Fixed Direct Access SPC-3 SCSI device
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 1907729MB (3907029168 512 byte sectors)
da1 at vtscsi0 bus 0 scbus2 target 0 lun 1
da1: <QEMU QEMU HARDDISK 2.5+> Fixed Direct Access SPC-3 SCSI device
da1: 300.000MB/s transfers
da1: Command Queueing enabled
da1: 1907729MB (3907029168 512 byte sectors)
da2 at vtscsi0 bus 0 scbus2 target 0 lun 2
da2: <QEMU QEMU HARDDISK 2.5+> Fixed Direct Access SPC-3 SCSI device
da2: 300.000MB/s transfers
da2: Command Queueing enabled
da2: 1907729MB (3907029168 512 byte sectors)
da3 at vtscsi0 bus 0 scbus2 target 0 lun 3
da3: <QEMU QEMU HARDDISK 2.5+> Fixed Direct Access SPC-3 SCSI device
da3: 300.000MB/s transfers
da3: Command Queueing enabled
da3: 1907729MB (3907029168 512 byte sectors)

T-Daemon · 2025-01-23T07:29:18+0000

jem said:
The system is a VM running under qemu/kvm, and the four hard disks are passed through from the hypervisor as raw SCSI devices, but are detected fine during boot:

If the pool can be imported, then this shouldn't be a factor.

I couldn't reproducible the issue on a test system:
disk1 Root-on-UFS, raidz2, 4 disks storage pool.

It shouldn't matter, but try "cachefile=/boot/zfs/zpool.cache".

Apparently the cachefile property retrieve is inconsistent:

Code:

# zpool get cachefile tank
NAME  PROPERTY   VALUE      SOURCE
tank  cachefile  -          default

# zpool set cachefile=/etc/zfs/zpool.cache tank

# zpool get cachefile tank
NAME  PROPERTY   VALUE      SOURCE
tank  cachefile  -          default

# zpool set cachefile=/boot/zfs/zpool.cache tank

# zpool get cachefile tank
NAME  PROPERTY   VALUE                  SOURCE
tank  cachefile  /boot/zfs/zpool.cache  local

On system reboot the cachefile VALUE is gone, replaced by " - ". The pool is imported automatically, though.

If "cachefile=/boot/zfs/zpool.cache", a copy should be created in /etc/zfs as well.

jem · 2025-01-23T07:32:44+0000

OK, here's something strange. I rebooted the VM, logged in, and ran the following before doing anything else:

Code:

[root@filer2 ~]# zpool list
no pools available

[root@filer2 ~]# zpool import
   pool: pool0
     id: 4536089192970494829
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

    pool0       ONLINE
      raidz1-0  ONLINE
        da2     ONLINE
        da3     ONLINE
        da0     ONLINE
        da1     ONLINE

[root@filer2 ~]# ls -l /etc/zfs/zpool.cache
-rw-r--r--  1 root wheel 3288 Jan 22 22:21 /etc/zfs/zpool.cache

[root@filer2 ~]# zdb -C -U /etc/zfs/zpool.cache
pool0:
    version: 5000
    name: 'pool0'
    state: 0
    txg: 48987
    pool_guid: 4536089192970494829
    errata: 0
    hostid: 726799574
    hostname: 'filer2'
    com.delphix:has_per_vdev_zaps
    vdev_children: 1
    vdev_tree:
        type: 'root'
        id: 0
        guid: 4536089192970494829
        create_txg: 4
        com.klarasystems:vdev_zap_root: 129
        children[0]:
            type: 'raidz'
            id: 0
            guid: 3438085862275244924
            nparity: 1
            metaslab_array: 256
            metaslab_shift: 34
            ashift: 12
            asize: 8001576501248
            is_log: 0
            create_txg: 4
            com.delphix:vdev_zap_top: 130
            children[0]:
                type: 'disk'
                id: 0
                guid: 14336507559410368910
                path: '/dev/da2'
                devid: 'ata-SAMSUNG_HD204UI_S2H7J90B309265'
                phys_path: 'pci-0000:06:00.0-sas-phy13-lun-0'
                whole_disk: 1
                DTL: 779
                create_txg: 4
                com.delphix:vdev_zap_leaf: 131
            children[1]:
                type: 'disk'
                id: 1
                guid: 2246828322968303539
                path: '/dev/da3'
                devid: 'ata-SAMSUNG_HD204UI_S2H7J9CB304047'
                phys_path: 'pci-0000:06:00.0-sas-phy12-lun-0'
                whole_disk: 1
                DTL: 778
                create_txg: 4
                com.delphix:vdev_zap_leaf: 132
            children[2]:
                type: 'disk'
                id: 2
                guid: 8185657086808692798
                path: '/dev/da0'
                devid: 'ata-SAMSUNG_HD204UI_S2H7J90B308992'
                phys_path: 'pci-0000:06:00.0-sas-phy15-lun-0'
                whole_disk: 1
                DTL: 777
                create_txg: 4
                com.delphix:vdev_zap_leaf: 133
            children[3]:
                type: 'disk'
                id: 3
                guid: 4515887524442859984
                path: '/dev/da1'
                devid: 'ata-SAMSUNG_HD204UI_S2H7J90B309221'
                phys_path: 'pci-0000:06:00.0-sas-phy14-lun-0'
                whole_disk: 1
                DTL: 776
                create_txg: 4
                com.delphix:vdev_zap_leaf: 134
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2

[root@filer2 ~]# zpool import -c /etc/zfs/zpool.cache -a -N

[root@filer2 ~]# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
pool0  7.27T  64.5G  7.20T        -         -     0%     0%  1.00x    ONLINE  -

[root@filer2 ~]# zfs list
NAME                  USED  AVAIL  REFER  MOUNTPOINT
pool0                46.9G  5.11T   140K  /pool0
pool0/media          24.2G  5.11T   151K  /pool0/media
pool0/media/blu-ray  24.2G  5.11T  24.2G  /pool0/media/blu-ray
pool0/media/dvd       140K  5.11T   140K  /pool0/media/dvd
pool0/qemu-images    22.6G  5.11T  22.6G  /pool0/qemu-images

The zpool import -c /etc/zfs/zpool.cache -a -N command is exactly what /etc/rc.d/zpool runs, so I'm confused as to why it worked when I ran it manually, but not when the rc script ran it at boot.

Erichans · 2025-01-23T08:15:47+0000

Is there anything that indicates the rc.de setup executes as planned/fails in either the zfs history or dmesg?

jem · 2025-01-23T09:48:54+0000

Erichans said:
Is there anything that indicates the rc.de setup executes as planned/fails in either the zfs history or dmesg?

I added rc_debug="YES" to /etc/rc.conf and rebooted but there was nothing obvious in /var/log/messages afterwards.

However, manually running /etc/rc.d/zpool start after boot works and outputs a few DEBUG lines:

Code:

root@filer2 ~]# zpool list
no pools available

[root@filer2 ~]# /etc/rc.d/zpool start
/etc/rc.d/zpool: DEBUG: checkyesno: zfs_enable is set to YES.
/etc/rc.d/zpool: DEBUG: load_kld: zfs kernel module already loaded.
/etc/rc.d/zpool: DEBUG: run_rc_command: doit:  zpool_start

[root@filer2 ~]# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
pool0  7.27T  64.5G  7.20T        -         -     0%     0%  1.00x    ONLINE  -

These /etc/rc.d/zpool: DEBUG: ... lines were not present in the messages from the boot, suggesting that the zpool rc script isn't being run.

jem · 2025-01-23T10:23:35+0000

(deleted)

cy@ · 2025-01-23T17:50:13+0000

You're pretty certain it is? How do you know?

jem · 2025-01-23T18:49:10+0000

The timestamp of /etc/zfs/zpool.cache updates when I manually import the pool:

Code:

[root@filer2 ~]# zpool list
no pools available

[root@filer2 ~]# ls -l /etc/zfs/zpool.cache
-rw-r--r--  1 root wheel 3288 Jan 23 09:46 /etc/zfs/zpool.cache

[root@filer2 ~]# zpool import pool0

[root@filer2 ~]# ls -l /etc/zfs/zpool.cache
-rw-r--r--  1 root wheel 3288 Jan 23 18:42 /etc/zfs/zpool.cache

Also, the zpool.cache file is completedly removed when I export the pool, and reappears when I import the pool:

Code:

[root@filer2 ~]# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
pool0  7.27T  64.5G  7.20T        -         -     0%     0%  1.00x    ONLINE  -

[root@filer2 ~]# ls -l /etc/zfs/zpool.cache
-rw-r--r--  1 root wheel 3288 Jan 23 18:45 /etc/zfs/zpool.cache

[root@filer2 ~]# zpool export pool0

[root@filer2 ~]# ls -l /etc/zfs/zpool.cache
ls: /etc/zfs/zpool.cache: No such file or directory

[root@filer2 ~]# zpool import pool0

[root@filer2 ~]# ls -l /etc/zfs/zpool.cache
-rw-r--r--  1 root wheel 3288 Jan 23 18:47 /etc/zfs/zpool.cache

[root@filer2 ~]# zdb -CU /etc/zfs/zpool.cache | head -10
pool0:
    version: 5000
    name: 'pool0'
    state: 0
    txg: 55652
    pool_guid: 4536089192970494829
    errata: 0
    hostid: 726799574
    hostname: 'filer2'
    com.delphix:has_per_vdev_zaps