Server doesn't boot after upgrading from 10.3 to 11.2

[Rob215x]

Hello,

I have a server used for web hosting that is located in another state so I only have access via SSH.
- I DO have root access.
- I am NOT an expert and I only know the basics so please bear with me.
- My system has been running great for a few years and I have upgraded FreeBSD successfully in the past.

Tonight I was following the guide at https://www.freebsd.org/releases/11.2R/installation.html and I completed:

# freebsd-update upgrade -r 11.2-RELEASE

# freebsd-update install

Everything looked good and it asked me to reboot. I did. Now the server won't come back up.

A tech at my hosting company sent me this screenshot:

They are not familiar with FreeBSD (It was a custom request I made when I originally bought the server) so I am kind of on my own at this point.

Any ideas what I should try?? I've been googling and searching this forum but couldn't find the same situation.

Thanks!!

 

[ShelLuser]

This is why I don't administrate remote servers without having access to the console itself.. In this day and age that should even be possible over the Internet.

Alas: /dev/ad4s1a seems to imply that you're using the older BSD partitioning setup and for some reason FreeBSD can't access that anymore. I'd reboot, and in the boot menu hit escape to drop down to the ok> prompt, then issue lsdev to see what the system detects. It might even be possible to run this command in the state the machine is in right now (assumption on my end).

There are two possible causes here: either something went wrong with your storage media which now prevents the system from booting, or something changed in the kernel which made it lose support for your setup (which seems unlikely to me).

So this could in theory also be caused by a hardware failure

 

[Rob215x]

UPDATE: (20 minutes later)

I had the tech at the data center send me a screenshot of the console output when he entered ? at the mountroot> prompt:

Next, he tried selecting the previous kernel from the boot menu and the server started up!!
- all of my web sites are running again!!

I logged into the server via SSH like I normally do:

FreeBSD 10.3-RELEASE-p11 (GENERIC) #0: Mon Oct 24 18:49:24 UTC 2016

Welcome to FreeBSD!

Release Notes, Errata: https://www.FreeBSD.org/releases/
Security Advisories:   https://www.FreeBSD.org/security/
FreeBSD Handbook:      https://www.FreeBSD.org/handbook/
FreeBSD FAQ:           https://www.FreeBSD.org/faq/
Questions List: https://lists.FreeBSD.org/mailman/listinfo/freebsd-questions/
FreeBSD Forums:        https://forums.FreeBSD.org/

And I did the following:

# df -a
Filesystem  1K-blocks     Used     Avail Capacity  Mounted on
/dev/ad4s1a   1012974   292100    639838    31%    /
devfs               1        1         0   100%    /dev
/dev/ad4s1e   1012974     4032    927906     0%    /tmp
/dev/ad4s1f 919657240 26649534 819435128     3%    /usr
/dev/ad4s1d  20266414 15082234   3562868    81%    /var

So, WHY are my original partitions listed as "ad4s1x" and when I tried to boot into the new kernel, all of the partitions are listed as "ada0s1x"???

What can I do next? I still need to upgrade my system to 11.2

thanks!!

 

[Glass]

You'll need someone to physically type in to the machine.

If they try:

ufs:/dev/ada0s1a rw

It may boot, they you'll need to update /etc/fstab to match.

 

[T-Daemon]

If I recall correctly the disk device node names have changed beginning with 11 to the previous 10.x Release. The 11.2 loader recognises the first disk as ada0 , whereas it tries to mount ad4s1a . Specify the root filesystem manually at the mountroot> prompt with the correct disk name (/dev/adaXs1a - replacing X with the corresponding disk number, if you have more than one disk). Afterwards change the entries in the loader variables and in /etc/fstab, or from the start - drop at the boot menu to single-user mode and make the changes from there Edit: On second thought better test the changes at the mountroot> prompt first, to see if it works, than to have modified the entries from single-user mode and the outcome is not what we want or worst.

 

[T-Daemon]

Why is the posting order not in sync? I posted after ShelLuser, now there are two other postings after him and my post is placed at the end?

Hi Rob215x,

So, WHY are my original partitions listed as "ad4s1x"

The loader variable vfs.root.mountfrom=ufs:/dev/ad4s1a is set in /boot/loader.conf

and when I tried to boot into the new kernel, all of the partitions are listed as "ada0s1x"???

New disk device node names in 11.x .

What can I do next? I still need to upgrade my system to 11.2

Look at Glass' post and mine, #5

 

[Rob215x]

Thanks everyone!

I will send this thread to the person who has access to the console, because I do not. I'm fairly certain they know what they're doing with Linux/Unix in general but they admitted they have little experience with FreeBSD. They are used to dealing with CentOS/WHM/CPanel.

So, for my own benefit and those who might find this thread later, how are other people doing upgrades without running into this issue?
- If I understand correctly, the disk node names were changed in release 11.x but I didn't see this in the documentation. Did I miss it? I always try to read documentation before upgrades so I read https://www.freebsd.org/releases/11.2R/relnotes.html
- Just now, I went back and read https://www.freebsd.org/releases/11.0R/relnotes.html and I didn't see anything about changing disk node names there either.
- In my googling, I only found one person with a similar problem but that person also moved the drive to a different interface so I'm not even sure its the same.
- So how is this not a common problem? Is it because I went from 10.3 to 11.2, skipping 10.4, 11.0, and 11.1? If so, I can't believe I'm the only one to do such a thing. IMO some additional documentation to help people avoid this issue would be great.
- And again, I'm sorry if I'm asking newb questions, but I'm always trying to learn.

To Review:
1. The person at the console will select the 11.2 kernel and at mountroot> will enter ufs:/dev/ada0s1a rw

2. Then the server should boot and I should have SSH access? Will I edit /etc/fstab myself or does it need to be edited by the person at the console? My current fstab:

# cat /etc/fstab
# Device        Mountpoint    FStype    Options        Dump    Pass#
/dev/ad4s1b        none        swap    sw            0        0
/dev/ad4s1a        /            ufs    rw                1        1
/dev/ad4s1e        /tmp        ufs    rw                2        2
/dev/ad4s1f        /usr        ufs    rw                2        2
/dev/ad4s1d        /var        ufs    rw                2        2
/dev/cd0        /cdrom        cd9660    ro,noauto    0        0

3. Do I need to edit /boot/loader.conf? Because currently all it has is:

# cat /boot/loader.conf
accf_http_load="YES"

Thanks again!!

 

[Rob215x]

UPDATE...
I had the tech with console access reboot the server and enter what was suggested. Here's the result:

Then, he was able to reboot the machine into the 10.3 kernel again and my web sites are back up.

Any ideas would be super helpful.

thanks,

Robert

 

[chrcol]

I admit I wasnt aware of this change and have successfully remote updated multiple machines, I got away with it because I am using disk labels and zfs.

But also the actual physical labels of the disks have always been ada0 anyway, so they would have stayed as ada0 on the update as well.

 

[T-Daemon]

First of all, before proceeding any further, have you a reliable backup of the server. If not you should make one ASAP.

- If I understand correctly, the disk node names were changed in release 11.x but I didn't see this in the documentation. Did I miss it? I always try to read documentation before upgrades so I read ...

I read about the disk device naming, but I didn’t remember exactly the circumstances. I made the statement in my post from memory, without reinvestigating first. “New disk device node names in 11.x” is not entirely correct. More accurate: "Legacy disk device naming dropped in 11.0 ".

Here the story: /dev/ad* naming originates from the ata(4) - generic ATA/ATAPI disk controller driver, first appeared in FreeBSD 4.0, turned into a CAM(4) interface module in FreeBSD 9.0, from (?) till 10.3 compatibility shims for legacy ATA device names used ( ada(4) - kern.cam.ada.legacy_aliases - creates symbolic links /dev/ada* -> /dev/ad* ) - removed in 11.0 (stable/11/UPDATING)

20151011:

Compatibility shims for legacy ATA device names have been removed.
It includes ATA_STATIC_ID kernel option, kern.cam.ada.legacy_aliases
and kern.geom.raid.legacy_aliases loader tunables, kern.devalias.*
environment variables, /dev/ad* and /dev/ar* symbolic links.

Your post #8

After mounting / , 11.2 is reading /etc/fstab , but the entries are not corresponding with the "List of GEOM managed disk devices:" (see the picture in your post #3)

After mountroot> ufs:/dev/ada0s1a the system is going to single user mode (picture post #8), stops at:

Enter full pathname of shell or RETURN for /bin/sh: - Hit ENTER here,

# mount -urw / - remount the root file system
( Edit: If you get after the command a message r/w mount denied, filesystem not clean - run fsck twice. )
.
Edit /etc/fstab (with ee(1) or vi(1)), replace all entries with ad4* with ada0*

/etc/fstab

# Device        Mountpoint    FStype    Options        Dump    Pass#
/dev/ada0s1b        none        swap    sw            0        0
/dev/ada0s1a        /            ufs    rw                1        1
/dev/ada0s1e        /tmp        ufs    rw                2        2
/dev/ada0s1f        /usr        ufs    rw                2        2
/dev/ada0s1d        /var        ufs    rw                2        2
/dev/cd0        /cdrom        cd9660    ro,noauto    0        0

# reboot - hopefully into 11.2,
( Edit: or # exit )
.
Once again Backup! , if haven't done yet.

 

[Rob215x]

Thanks for the reply. I will try this as soon as I can schedule it with my host.

First of all, before proceeding any further, have you a reliable backup of the server. If not you should make one ASAP.

- I back up my client web sites daily, including their mySQL databases.
- The mail server isn't used much, except for contact forms on the web sites. (clients host their email on Google or elsewhere)
- I have my httpd.conf and php.ini backed up (I recently upgraded Apache and PHP)

Am I missing anything?

 

[ShelLuser]

- I back up my client web sites daily, including their mySQL databases.
- The mail server isn't used much, except for contact forms on the web sites. (clients host their email on Google or elsewhere)
- I have my httpd.conf and php.ini backed up (I recently upgraded Apache and PHP)

Although that might be enough to eventually rebuild something it would take you an awful amount of time as well. At the very least make backups of /etc, /var and /usr/local/etc.

/etc should be obvious: this is where most of your configuration files reside. Although it may be easy to rebuild some, others can become a drag (think about /etc/fstab or /etc/make.conf). Same more or less applies to the local variant, better to be safe than sore.

/var is also an increasingly used place for configuration. When using Bind in a chrooted environment then this is where the main directory resides, programs like Unbound also dump their config files in here. And then there are things such as logfiles, port configuration files, your package database, etc.

However, you might want to try and exclude /var/cache and/or /var/tmp because that's mostly used for temporary data. Since you're using UFS I'd strongly suggest using dump, a mere # chflags nodump /var/cache /var/tmp should be enough to exclude these locations, then you can simply dump the entire filesystem: # dump 0uf - /var | gzip /root/var171218.ufs.gz, for example.

 

[T-Daemon]

Be advised, when the disk naming is changed in /etc/fstab , so the 11.2 kernel can mount the partitions and something goes wrong, and if you decide to boot the 10.3 kernel, it will not mount the partitions, because of the used legacy disk naming. The system will stuck at the mountroot> prompt. If you want to use the 10.3 system again, the entries in /etc/fstab need to be changed to the original settings. Sorry for not mentioning this earlier, it didn’t come to my mind then.

One more thought - If you wish not to worry about disk naming anymore, you could label the partitions. There is a fresh Thread 68692 about this subject and a Handbook chapter. In the Handbook chapter are two types of labelling mentioned - Labelling by name and by ufsid - unique file system id. When you look closer at the picture in post #3 under “List of GEOM managed disk devices” the ufsid’s of the partitions are listed there. But that’s for later, if taken in consideration. Upgrade first the system.

 

[Rob215x]

I just wanted to let everyone know that the tech at my hosting company was able to reboot the server into 11.2 by using the directions given above by T-Daemon above.

I was just able to SSH into my server, and was greeted by the FreeBSD 11.2 message. My current fstab now looks like:

# cat /etc/fstab
# Device            Mountpoint    FStype    Options       Dump    Pass#
/dev/ada0s1b        none          swap      sw        0    0
/dev/ada0s1a        /             ufs       rw             1    1
/dev/ada0s1e        /tmp          ufs       rw        2    2
/dev/ada0s1f        /usr          ufs       rw        2    2
/dev/ada0s1d        /var          ufs       rw        2    2
/dev/cd0            /cdrom        cd9660    ro,noauto    0    0

Thanks again!!

 

[T-Daemon]

Have you completed the upgrade to 11.2 Release. Since the process involves several freebsd-update install + reboot. The 11.2 kernel wasn’t able to mount root after the first reboot of upgrading. After the file system was mounted good, have you completed the upgrade? If not you might be in a system with 11.2 kernel, but incomplete system upgrade. Also all ports/packages need to be reinstalled (requirement after a major version upgrade).

 

[Rob215x]

After the file system was mounted good, have you completed the upgrade? If not you might be in a system with 11.2 kernel, but incomplete system upgrade. Also all ports/packages need to be reinstalled (requirement after a major version upgrade).

Thanks, and no I have not completed the upgrade. Since this is a live web server, and all of the web sites are currently working, I'm waiting until after the holidays to complete the upgrade. I'm very squeezed for time at the moment and will be visiting family. I want to complete the upgrade in the middle of the night (most of the web sites are local to the EST time zone and the server gets the least amount of traffic between 2am-5am) but I also want to make sure I can be up all night if I have trouble starting certain applications (I've have this happen in the past). I also want to make sure the same hosting company tech who's been helping me, is working that night, in case I need someone to access the console.

I'm looking forward to completing the upgrade but I'm trying to take the best path to keeping all of the web sites running during business hours until the holidays are over.

 

[dieselnutjob]

I experienced the exact same problem last week going from FreeBSD 10.4 to 11.0.
I had to manually mount my partitions from console and then edit /etc/fstab changing all the entries containing /dev/ad to /dev/ada

These are the logs from dmesg saying what device driver it is:-

ada0 at ata0 bus 0 scbus0 target 0 lun 0
ada0: <QEMU HARDDISK 0.12.1> ATA-7 device
ada0: Serial Number QM00001
ada0: 16.700MB/s transfers (WDMA2, PIO 8192bytes)
ada0: 20480MB (41943040 512 byte sectors)