ZFS zfs send hanging

[decuser]

So I took recursive snapshots:

sudo zfs snapshot -r prime@snap

Then I tried to send:

sudo zfs send -R prime@snap| pv | ssh wsenn@fenris "sudo zfs recv -Fd prime"

It went swimmingly until it hit 3.23 GiB, then the clock keeps ticking, but the network activity grinds to a halt. Help.

Here's what the progress looks like:

3.23GiB 0:02:17 [0.00 B/s] [   <=>                      ]

Where the 0:02:17 counts up by the second, the B/s never changes and the <=> stops moving.

If I do CTL-T a couple of times with a gap:

load: 0.55  cmd: ssh 4492 [select] 343.78r 8.81u 5.64s 0% 24696k
load: 0.36  cmd: ssh 4492 [select] 386.59r 8.81u 5.64s 0% 24696k

What else can I do to figure out what's hanging things up. I've done this a lot in the past (FreeBSD 13 and prior) and it worked fine. This is my first time trying it on 14.1, both odin and fenris are on 14.1p4.

-will

 

[gpw928]

Hmmm, maybe you are accumulating stuff in a buffer? Please show us the output of:

ssh wsenn@fenris pwd
ssh wsenn@fenris which zfs

 

[decuser]

For the time being, I removed pv from the pipeline and on fenris (the remote side), ran

while (true); do zpool iostat; sleep 5;done


seems to be making progress beyond 3.23GB. Any way to troubleshoot pv?

 

[decuser]

For the time being, I removed pv from the pipeline and on fenris (the remote side), ran

while (true); do zpool iostat; sleep 5;done


seems to be making progress beyond 3.23GB. Any way to troubleshoot pv?

It completed successfully. I’ll get the output from the commands and post them later when I get back to the machine .

 

[cracauer@]

misc/cstream can display throughput at specified intervals. I don't know much about pv.

Use it like this: `... | cstream -T10 -v2 | ...`

 

[decuser]

Hmmm, maybe you are accumulating stuff in a buffer? Please show us the output of:

ssh wsenn@fenris pwd
ssh wsenn@fenris which zfs

~ $ ssh wsenn@fenris pwd
/home/wsenn
~ $ ssh wsenn@fenris which zfs
/sbin/zfs

 

[Eric A. Borisch]

Make sure your pv is up-to-date. I tracked down a bug that was patched in https://codeberg.org/a-j-wood/pv/releases/tag/v1.8.12 which would frequently cause permanent hangs when used in a zfs send/recv pipe.

 

[decuser]

Make sure your pv is up-to-date. I tracked down a bug that was patched in https://codeberg.org/a-j-wood/pv/releases/tag/v1.8.12 which would frequently cause permanent hangs when used in a zfs send/recv pipe.

Interestingly, I'm up to date with what's in pkg repo, but it's still behind 1.8.12 (it's 1.8.10). I built from source with 1.8.12 and sure enough, it seems to work. I'll update when it completes, but it definitely got past the prior stopping point.

 

[Eric A. Borisch]

You must be on the default quarterly repo. According to freshports 1.8.14 is available (at least for amd64).

 

[decuser]

You must be on the default quarterly repo. According to freshports 1.8.14 is available (at least for amd64).

1.8.12 works, so I'm good until a normal update picks up something more current.

 

[T-Daemon]

If you don't mind a per-second report about how much data has been sent, you could enable verbose mode.

zfs-send(8)

      -v, --verbose
           Print verbose information about the stream package generated.  This
           information includes a per-second report of how much data has been
           sent.  The same report can be requested by sending SIGINFO or
           SIGUSR1, regardless of -v.