ZFS Cannot boot after kernel upgrade to 11.2-RELEASE

W

wildtollwut

Hello,

after initiating the 11.2 upgrade on Sunday evening and finishing the kernel part of it,
the following reboot left me with a ZFS error:

Code: 
ZFS: i/o error - all block copies unavailable
ZFS: can't find pool by guid

The pool layout is simple, just a single disk, so no RAID.

I find it rather surprising that a kernel update can cause something like that but supposedly
it's not only the kernel but the whole boot directory being updated?

After not really finding a root cause by browsing the web and newsgroups I used a
live ISO to boot. Thus, I was able to mount my disk, copying back the old kernel but, of course,
to no avail.

The next thing I tried was upgrading the pool to the latest feature set
Code: 
zpool upgrade tank
but this did not change a thing. After that I tried rewriting the bootcode:
Code: 
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
This now leaves me with another error message
Code: 
ZFS: can't read MOS of pool tank
[...]
ZFS: I/O error - blocks larger than 16777216 are not supported
ZFS: can't find dataset 0
Default: tank/<0x0>

Still not know what the problem really is, I'm kind of at a loss here. The pool itself
looks healthy and all data is accessible. Is there some way to salvage the situation or
at least understand what's been going on? Can the zpool.cache file be corrupt/old somehow
and do I need to update it?
 
ZFS I/O errors generally means that the disk is bad. Or, easier to check, the cables are a bit dodgy. When you boot the live CD are you able to import the pool?
 
From what FreeBSD version did you upgrade?

Also: if you're in the boot menu and then press escape (to drop down to the boot prompt) what do you see if you issue the lsdev command? Does that actually recognize your ZFS pool?
 
I upgraded from 11.1-RELEASE and I'm able to import the pool as well as read all files from it (on a live system). The disk cannot have become bad from just kernel upgrading can it?

Since my latest change, I can't get to the boot prompt anymore, I believe I'm in some other kind of prompt that just allows me to enter something after
Code: 
Default: tank/<0x0>:
boot:
 
Can you post a gpart show? There are different ways of booting a system nowadays, so writing the gptzfsboot(8) may not be correct for your situation.
 
Interesting, it says corrupt here:
Code: 
gpart show ada0
=>         40   488397088  ada0  GPT    (233G)  [CORRUPT]
           40        1024     1  freebsd-boot    (512K)
         1064     8388608     2  freebsd-swap    (4.0G)
      8389672   480007456     3  freebsd-zfs    (229G)

gpart recover does not help though.
 
Yeah, that doesn't bode well for the disk. The gtpzfsboot(8) you used is the correct one for your situation at least. But I'm very much doubting the disk itself though, it might have some bad sectors in the wrong places.

I'd use the live CD and backup anything and everything important. Just to be on the safe side.
 
The disk is really new, like six weeks and SMART shows no abnormality. I highly doubt it's faulty. I find it much more likely that I messed up something, but I'm not sure how to find out what.
 
Even if you completely hosed the upgrade (which I've done many times) you would never get I/O errors or a corrupted GPT. So there's something dodgy going on.
 
Corrupt GPT is usually a sign that the disk geometry was detected incorrectly at the time of the first install and then a later version of the OS corrects the geometry error but this makes the existing partition table corrupt.

The other way the same can happen is if you mixed GPT partitioning with the GEOM raid classes like gmirror(8), there is still (I think) a conflict in the storage area of the GEOM mirror metadata and the backup GPT table at the end of the disk if GEOM mirror is used on whole disks.
 
Ok, so maybe the problem was my way of copying my disk layout from the old disk (the one before that) to this one. I still have a pool with the same name that I cannot destroy (it's faulty hence I cannot import and destroy it). Is it possible to copy all data from it to another disk, create the proper GPT partition scheme and write it back so it will work in the future? What are the proper steps for that and what about zpool.cache? Would zfs send help?
Or is there another chance to salvage what I have?
 
How did you copy the disk layout from your old disk?
I'm not sure how gpart backup/restore behaves when restoring to a disk of smaller size then the original, but if it behaves like dd, partitions that exceed the
size of the original disk, those partitions have to be deleted and recreated to fit.

Another possibility could be data corruption through trim, since from the size your disk looks like a SSD.
I have a Samsung SM841 that corrupts the partition table and destroys data when trim is enabled.
(not sure if that still applies, don't wan't to try ;) see below)
Code: 
cat /usr/src/UPDATING | grep -A25 -i 20160414
20160414:
        The CAM I/O scheduler has been committed to the kernel. There should be
        no user visible impact. This does enable NCQ Trim on ada SSDs. While the
        list of known rogues that claim support for this but actually corrupt
        data is believed to be complete, be on the lookout for data
        corruption. The known rogue list is believed to be complete:

                o Crucial MX100, M550 drives with MU01 firmware.
                o Micron M510 and M550 drives with MU01 firmware.
                o Micron M500 prior to MU07 firmware
                o Samsung 830, 840, and 850 all firmwares
                o FCCT M500 all firmwares

        Crucial has firmware http://www.crucial.com/usa/en/support-ssd-firmware
        with working NCQ TRIM. For Micron branded drives, see your sales rep for
        updated firmware. Black listed drives will work correctly because these
        drives work correctly so long as no NCQ TRIMs are sent to them. Given
        this list is the same as found in Linux, it's believed there are no
        other rogues in the market place. All other models from the above
        vendors work.

        To be safe, if you are at all concerned, you can quirk each of your
        drives to prevent NCQ from being sent by setting:
                kern.cam.ada.X.quirks="0x2"
        in loader.conf. If the drive requires the 4k sector quirk, set the
        quirks entry to 0x3.
 
So I found the problem. It's a race condition on boot with the zfs partitions. basically the system will try to mount the partitions in no specific order. This causes a problem for the ROOT partition needs to be mounted first then the rest can be mounted. I fixed this by booting to a USB drive and mounting the zfs zroot/default/ROOT partition first then zfs mount -a. After root boot the system came back without issues.
Steps:
1. boot off USB 11.2 disk
2. zpool import -R /mnt <zroot>
3. zfs mount <zroot>/default/ROOT
4. zfs mount -a
5. reboot back into the upgraded OS.
 
c-hair said:
So I found the problem. It's a race condition on boot with the zfs partitions. basically the system will try to mount the partitions in no specific order.
Click to expand...
That's nonsense, this could only happen if you completely mis-configured your setup somehow and even then I seriously doubt it. Mounting the root filesystem doesn't depend on any sequence at all; at first the bootloader is pointed to the filesystem in /boot/loader.conf through the vfs.root.mountfrom property. Then there's the pool itself which also denotes the root filesystem through the bootfs property.

See loader.conf(5) and zpool(8) respectfully, this is not a game of chance.

c-hair said:
I fixed this by booting to a USB drive and mounting the zfs zroot/default/ROOT partition first then
Click to expand...
The steps you described don't fix anything at all. If you had issues and they got resolved after this procedure then the cause of your problem hasn't been addressed yet. Considering the inconsistent nature (not working then working again) I'd start by running a scrub on your pool to verify that your disks aren't failing somehow, because that could explain a thing or two.

Sorry to say but you didn't fix anything at all here so I'd definitely be careful with that system.
 
I am having the same issue. The ZFS root file system is 8TB. I believe it is too large for the zfs root on ZFS. The boot issue happened after I filled the 8TB zfs with some data. Running the latest 12.0 release.
It is a RAID 6, 8TB. How would I install freebsd 12 on it? Can I use ZFS safely?


ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS of pool zroot
gptzfsboot: failed to mount default pool zroot

FreeBSD/x86 boot
ZFS: I/O error - blocks larger than 16777216 are not supported
ZFS: can't find dataset u
Default: zroot/<0x0>:
 
ahgu said:
I believe it is too large for the zfs root on ZFS
Click to expand...
ZFS can handle 16YiB (16 * 2^128) volumes and 16EB (16 * 2^64) max file sizes. So no, 8TB isn't too large for ZFS.

What's more likely though is that your controller isn't capable of dealing with 8TB drives.

ahgu said:
It is a RAID 6, 8TB. How would I install freebsd 12 on it? Can I use ZFS safely?
Click to expand...
Don't use ZFS on top of a hardware RAID.
 
ahgu, if you are using MSDOS partition table and you boot directly from your zfs partition (which I am not sure is possible, I think there must be a small boot partition with the zfs boot image),
then this combination is limited to 2TiB partition size for the modern x86/AMD64 system. If the system is older there may be different limitation.
If you are using a GPT partition table and/or a boot partition different than the zfs one - you should have no issues ;)
 
gnoma said:
ahgu, if you are using MSDOS partition table and you boot directly from your zfs partition (which I am not sure is possible, I think there must be a small boot partition with the zfs boot image),
Click to expand...
It's possible but rarely done on FreeBSD. On FreeBSD ZFS is typically used with GPT partitioning. Compare gptzfsboot(8) and zfsboot(8) for example.
 
SirDice said:
Can you post a gpart show? There are different ways of booting a system nowadays, so writing the gptzfsboot(8) may not be correct for your situation.
Click to expand...
please help me :( similar isssue :( plz help. :(
7003


I was installing the mininet port when i saw some sort of error occur (white letters instaead of the regualr gray [gray =OS, white = VirtualBox). It was stating things like timeout. Then it says disk removed. All of whch are impossible because the virtual disk on a drive that is inside my laptop.. That laptop dive is still functioning normal. Also, the VM became unresponsive, as well as virutal box, so i had to force quit virtual box.
 
You must log in or register to reply here.
Back
Top