Mounting fails after upgrading 9.3R to 10.1R

[arapaima]

I did a binary upgrade with freebsd-update(8).

It went something like this:

# freebsd-update -r 10.1-RELEASE upgrade
# freebsd-update install

Installing updates...chflags: ///var/empty: Read-only file system

# shutdown -r now

I didn't really mind the error and just rebooted the system when freebsd-update finished.

I adjusted the zfs readonly property for /var/empty, and I guess it just resumed the previous installation? (since it didn't keep on with installing userland updates)

# zfs set readonly=off tank/var/empty
# freebsd-update install

Installing updates...
Kernel updates have been installed. Please reboot and run
"/usr/sbin/freebsd-update install" again to finish installing updates.

# shutdown -r now

Now FreeBSD is unable to mount the system.

Mounting failed from zfs:tank with error 2.

I'm not sure where to go from here and I have no clue on what the problem might be, so I would need some pointers.

I booted from a FreeBSD 10.1-RELEASE image and imported the pool by

# mkdir /tmp/mnt && zpool import -R /tmp/mnt -f tank

cannot share 'tank/user/local/storage': share (1M) failed

This didn't mount datasets with legacy mount points. I needed to mount these separately.
mount -t zfs tank /tmp/[...] .

My /boot/loader.conf (part of, ZFS related)

zfs_load="YES"
vfs.root.mountfrom="zfs:tank"
(...)

Is there a way of mounting all datasets under one single 'altroot'? (even those with legacy mount points).
This would give me an option to chroot <path> in to the filesystem and try freebsd-update rollback to restart the upgrade from the beginning.

Thanks!

 

[arapaima]

Now I'm able to mount all datasets under one single 'altroot' making it possible for me to chroot into the system.
zpool import couldn't handle legacy mount points, so I had to set one by zfs set mountpoint=/ tank.

I was also thinking maybe some features were missing on my pool making FreeBSD 10.1R unable to mount zfs:tank, so I upgraded it by zfs upgrade and zpool upgrade (from a FreeBSD 10.1R image).
This broked the boot process since the installed boot loader complained about missing feature flags.

I booted the FreeBSD 10.1-RELEASE image again to upgrade the bootloader
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada2

Maybe I'm dealing with a partial upgrade. I'm thinking that freebsd-upgrade should've upgraded the bootloader if needed?

Above just left me in the beginning again ("Mounting failed from zfs:tank with error 2"), so I've zfs rollback [...] all individual datasets from the snapshots (a state where the system was FreeBSD 9.3R).
Now I'm not able to boot at all:

ZFS: Unspoorted feature: com.delphix:embedded_data
ZFS: Unspoorted feature: com.delphix:embedded_data
ZFS: Unspoorted feature: com.delphix:embedded_data
(...)
ZFS: can't find pool by guid
ZFS: can't find pool by guid

can't load 'kernel'

I thought this was because of the 10.1R boot loader I installed earlier (the system should be FreeBSD 9.3R since I zfs rollback to a state before freebsd-upgrade.
Or maybe because of the kernel/ZFS implementation not supporting the 'embedded_data' feature.
I booted from a 9.3R image and replaced the bootloaders again. It didn't help. I'm also unable to import my pool from a 9.3R disk image (I need a 10.1R image for that)

So it seems like I'm in a FreeBSD 9.3R system were the kernel doesn't support a ZFS feature which is written to my ZFS pool. As far as I know there are no ways of downgrading zfs pools.

What options do I got, can I make a freebsd-upgrade on the pool and try to repair it?

As I see it now my only option is to connect an additional hard drive, copy the files and make a clean reinstallation.