ZFS ZFS replication - interrupted and resumed - missing snapshots

[thilo]

I have a strange problem and I am not sure if it is a bug or my fault.
(I am using a stock 13.1-RELEASE)

When I resume an interrupted zfs transfer, it seems to be missing snapshots.

The below script reproduces the problem. It creates a filesystem with some snapshots and replicates it.
The result:

D="Backup/testset"
zfs destroy -r "$D"
zfs destroy -r "$D"_good
zfs destroy -r "$D"_bad
zfs create "$D"
zfs snapshot "$D@initial"
dd if=/dev/random bs=10m count=10 of=$D/file1
zfs snapshot "$D@add100m"
dd if=/dev/random bs=10m count=30 of=$D/file2
zfs snapshot "$D@add300m"
rm $D/file1
zfs snapshot "$D@del100m"
rm $D/file2
zfs snapshot "$D@del300m"
zfs snapshot "$D@final"

#Good transfer
zfs send -R "$D@final" | zfs recv -s "$D"_good

#Interrupted transfer (after 200MB)
zfs send -R "$D@final" | dd bs=10m count=200 | zfs recv -s "$D"_bad

#Resume transfer
TOKEN="$(zfs get -H receive_resume_token   "$D"_bad  | cut -f 3)"
zfs send -t "$TOKEN" | zfs recv -s "$D"_bad

# Good
zfs list -t snapshot -r "$D"_good
#Bad
zfs list -t snapshot -r "$D"_bad

And it produces:

NAME                          USED  AVAIL     REFER  MOUNTPOINT
Backup/testset_good@initial  85.2K      -      128K  -
Backup/testset_good@add100m  85.2K      -      100M  -
Backup/testset_good@add300m  85.2K      -      400M  -
Backup/testset_good@del100m  85.2K      -      300M  -
Backup/testset_good@del300m     0B      -      128K  -
Backup/testset_good@final       0B      -      128K  -
NAME                         USED  AVAIL     REFER  MOUNTPOINT
Backup/testset_bad@initial  85.2K      -      128K  -
Backup/testset_bad@add100m     0B      -      100M  -

does anyone know if I am missing something of if I should file a bug?

Thanks

 

[thilo]

A reply from Alan Somers:
It's not a bug. The problem is that you can't combine zfs send -R and also zfs recv -s . The former creates several independent send streams. But the latter generates a token that will only resume a single stream. And there's no way to fix the problem given ZFS's design. If the send gets interrupted, you need to go check on the destination side to see which file datasets are incomplete and resume all of them.