ZFS Cannot add cache device

X

Xavier

Hi,
My cache/log SSD recently died
I was able to remove both cache and log device, and replace log storage,:
Code: 
[root@numenor ~]# zpool status -v
  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 07:00:39 with 0 errors on Wed Apr  5 15:45:21 2023
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            ada1p3  ONLINE       0     0     0
            ada2p3  ONLINE       0     0     0
            ada3p3  ONLINE       0     0     0
        logs
          ada0p1    ONLINE       0     0     0

errors: No known data errors
but when I try to add cache :
Code: 
[root@numenor ~]# zpool add zroot cache ada0p2
cannot add to 'zroot': no such pool or dataset
However ada0p2 exists in /dev, and is gparted the same way as ada0p1
What is this error, Googling didn't help ?
Thanks, regards
Xavier
 
I wonder what it's really about, wild guess - removing logs vdev usually adds a "hole" one instead, and the add operation could be stumbling on it. Could you paste output of zdb -C, and check if truss'ing your zpool command would provide any hints?

P.S. which release is this on?
 
I ran into some weird error messages last time I tried to replace a data disk. It couldn't find the pool for some reason, similar type of error as this. "What do you mean, no such pool or dataset?!? It's right there!" Turns out I had to add a -o ashift=9. Check the zdb(8) output. Verify ashift.
 
Bingo ! There is a 'hole' device :
Code: 
zroot:
    version: 5000
    name: 'zroot'
    state: 0
    txg: 40592307
    pool_guid: 5451616106165845366
    errata: 0
    hostname: ''
    com.delphix:has_per_vdev_zaps
    hole_array[0]: 1
    vdev_children: 3
    vdev_tree:
        type: 'root'
        id: 0
        guid: 5451616106165845366
        create_txg: 4
        children[0]:
            type: 'raidz'
            id: 0
            guid: 5686254814491435991
            nparity: 1
            metaslab_array: 39
            metaslab_shift: 34
            ashift: 12
            asize: 2974822170624
            is_log: 0
            create_txg: 4
            com.delphix:vdev_zap_top: 35
            children[0]:
                type: 'disk'
                id: 0
                guid: 10672382406202185779
                path: '/dev/ada1p3'
                phys_path: 'id1,enc@n3061686369656d30/type@0/slot@1/elmdesc@Slot_00/p3'
                whole_disk: 1
                DTL: 13383
                create_txg: 4
                com.delphix:vdev_zap_leaf: 502
            children[1]:
                type: 'disk'
                id: 1
                guid: 14798645879798091999
                path: '/dev/ada2p3'
                phys_path: 'id1,enc@n3061686369656d30/type@0/slot@3/elmdesc@Slot_02/p3'
                whole_disk: 1
                DTL: 168
                create_txg: 4
                com.delphix:vdev_zap_leaf: 437
            children[2]:
                type: 'disk'
                id: 2
                guid: 13108278056532379818
                path: '/dev/ada3p3'
                phys_path: 'id1,enc@n3061686369656d30/type@0/slot@4/elmdesc@Slot_03/p3'
                whole_disk: 1
                DTL: 13380
                create_txg: 4
                com.delphix:vdev_zap_leaf: 689
        children[1]:
            type: 'hole'
            id: 1
            guid: 0
            whole_disk: 0
            metaslab_array: 0
            metaslab_shift: 0
            ashift: 0
            asize: 0
            is_log: 0
            is_hole: 1
        children[2]:
            type: 'disk'
            id: 2
            guid: 11998453435321605573
            path: '/dev/ada0p1'
            whole_disk: 1
            metaslab_array: 482
            metaslab_shift: 30
            ashift: 12
            asize: 137434234880
            is_log: 1
            create_txg: 40592264
            com.delphix:vdev_zap_leaf: 600
            com.delphix:vdev_zap_top: 601
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
 
That was only a guess, and I can't reproduce it locally. Also still wondering which release you are running, and would like to see the truss output -- what ioctl exactly is it failing on?
 
yuripv79 said:
That was only a guess, and I can't reproduce it locally. Also still wondering which release you are running, and would like to see the truss output -- what ioctl exactly is it failing on?
Click to expand...
I'm running 13-STABLE, the truss output is attached
Xavier
 

Attachments

OK, so it's really failing inside zfs module, and it looks like (guessing on your zdb output) that what SirDice suggested about ashift isn't your problem.

Could you please try something a bit more sophisticated: dtrace -n 'fbt:zfs::entry,fbt:zfs::return { trace(arg1); }' -c 'zpool add zroot cache ada0p2' saving that output somewhere, wonder where exactly it is failing.
 
OK, looking at dtrace output, this doesn't make much sense. What is your gpart show output? Are you sure ada0p2 isn't used by something else?
 
yuripv79 said:
OK, looking at dtrace output, this doesn't make much sense. What is your gpart show output? Are you sure ada0p2 isn't used by something else?
Click to expand...
SOLVED :
I noticed in top that I had 128GB of swap, which looked weird. The output of swapctl -l showed me, among other swap devices ada0p2 !
Seems like the replacement of the disk, or other unattended event remapped the devices. I edited /etc/fstab and reset the swap area to what it should be, and was able to add my cache device.
Thanks to all
Cheers
Xavier
 
You must log in or register to reply here.
Back
Top