ZFS System Unresponsive During ZFS Resilvering?

[dave]

Hello,

I have a physical machine that runs a ZFS pool of six drives. After one drive developed SMART warnings, I removed it, replaced it with a larger drive, and then ran the zpool replace command to resilver the pool.

As the resilvering started, I noted that it was going pretty slow. There was a notice about the configured vs native block size. I am not in a hurry and will be eventually replacing all the disks, so I let that go.

Then later, the machine became unresponsive. The CAPS-LOCK key turns the light on the keyboard on and off so I know it is responding at the BIOS level. Also, the machine responds to pings, but no network requests. I plugged in a monitor to the otherwise headless machine, but get no video signal.

So, what should I do? Most importantly, if I reset the machine, do I risk losing my ZFS pool?

 

[PMc]

Yes, machines might get stuck during scrub/resilver, if they are weak on CPU and the disks are fast (SSD). Obviousely this also depends on what else wants to run. We do not know what's wrong with your blocksize, but such things may also lead to ZFS reading a lot more data than actually would be needed.

Usually, it there is a power loss, on restart the pool will just continue the scrub/resilver as soon as ZFS gets online. Normally with non-booting ZFS that is, after leaving maintenance shell or when going multiuser - which may then lead to the system not coming further up. (I don't know what it does in case of root/boot ZFS, when ZFS gets online much earlier). It should not loose data, except another disk would fail.

 

[dave]

Thanks. I'm going to reset it and see where I'm at then. At least I will have a monitor/keyboard then!

 

[phoenix]

You can try the old CTRL+Alt+Del to issue a reboot. If that fails, you'll have to force power off.

If you have dedupe or compression enabled, then scrub/resilver will require a lot of RAM to run. Especially on older FreeBSD releases (pre-9 was especially bad). With dedupe enabled, you may need to reboot the system a few times to get it to complete the resilver. Yet another reason not to use dedupe unless you absolutely must (and even then, consider not using it).

Edit: not so much RAM itself as wired memory space in the KVA (kernel virtual address) space.

 

[dave]

OK, well, resetting the machine does not seem to have caused any issue with the resilver. It looks like it's about half done. I have stopped all unneeded services and will wait for it to finish. You can see the block size issue below. I have decided to just take the performance hit on that for the time being.

  pool: tank
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri May 24 11:51:38 2019
    4.57T scanned out of 9.31T at 887K/s, (scan is slow, no estimated time)
        780G resilvered, 49.14% done
config:

    NAME                       STATE     READ WRITE CKSUM
    tank                       DEGRADED     0     0     0
      raidz1-0                 DEGRADED     0     0     0
        label/zdisk1           ONLINE       0     0     0
        label/zdisk2           ONLINE       0     0     0
        label/zdisk3           ONLINE       0     0     0
        replacing-3            DEGRADED     0     0     0
          3548302548923285235  OFFLINE      0     0     0  was /dev/label/zdisk4/old
          label/zdisk4         ONLINE       0     0     0  block size: 512B configured, 4096B native
        label/zdisk5           ONLINE       0     0     0
        label/zdisk6           ONLINE       0     0     0
    cache
      label/zcache0            ONLINE       0     0     0

errors: No known data errors

  pool: upool0
 state: ONLINE
  scan: none requested
config:

    NAME             STATE     READ WRITE CKSUM
    upool0           ONLINE       0     0     0
      mirror-0       ONLINE       0     0     0
        label/zusb0  ONLINE       0     0     0
        label/zusb1  ONLINE       0     0     0

errors: No known data errors

 

[PMc]

You can see the block size issue below. I have decided to just take the performance hit on that for the time being.

Yeah - that's not the bottleneck that hit You. There must have been something else running low.

 

[dave]

It was RAM, no doubt. After restarting, I dialed back the workload, and the machine completed the resilvering without issue.

 

[dave]

I have one more question here. It's not really entirely related to the OP title, but I'll ask here anyway, since it's the same conversation...

Regarding the block issue, what is the outlook? If I continue replacing the disks with the new ones, will the pool ever be able to use the new block size, or will I eventually have to recreate it anyway?

If so, what if I were to use the (mixed) disks I have now to create a pool with the 4k block size? I currently have some bulk storage available that I could use temporarily while I rebuild the pool. Then I would be moving towards the native block size instead of away from it. Would that be better than my current situation?

 

[k.jacker]

If I continue replacing the disks with the new ones, will the pool ever be able to use the new block size

Sadly, no.

If so, what if I were to use the (mixed) disks I have now to create a pool with the 4k block size?

That's not a problem. You can use mixed disk with whatever blocksize they are/report in a 4k pool. With mixed disks, it's best to force the blocksize by sysctl vfs.zfs.min_auto_ashift=12 before creating the new pool. There is no need to set that permanently in /etc/sysctl.conf as the pool will remain to be a 4k pool after creation.

Anyway, to make all future pools 4k, it's a good idea to make it permanent. I do that, so I can't forget to set it manually.