FreeBSD 13.2-RELEASE -> 14.0-RELEASE Upgrade Stuck?

J

jbancino

I just recently did a clean install of FreeBSD 13.2-RELEASE on a production server a few weeks ago, and it has been running well so far. However, this morning, I started the upgrade to FreeBSD 14.0-RELEASE, and I am getting concerned that maybe it got stuck, because it is taking an abnormally long time to complete. This system is running on 2x4TB drives, which running ZFS in a mirror configuration on top of geli, which is encrypting the entire drives. This is exactly as the FreeBSD installer set up the disks; I did not do anything special to them besides add a few ZFS datasets and copy about ~1.2TB of data to them.

For context, here are the commands that I have run so far:

Code: 
$ freebsd-update -r 14.0-RELEASE upgrade
$ freebsd-update install
$ shutdown -r now
$ freebsd-update install    # (Stuck here)

The first round of freebsd-update install before the reboot went fine, but the second round after the reboot has been running for 8 hours and counting. The only output that has been produced so far is as follows:

Code: 
Creating snapshot of existing boot environment... done.
Installing updates...

The disks sound like they have been at 100% activity for the entire duration of the update and I have confirmed that they are with gstat. I feel like maybe that is a good sign, but I cannot figure out what freebsd-update could possibly be doing that would be so I/O intensive for so long, without reporting any status updates.

Is it normal for upgrades to run for this long? If not, how can I find out what the upgrade is doing and if something has gone wrong? If something has gone wrong, do I have any options besides rolling back to the last ZFS snapshot? Would that even work at this point, or would I need to re-install the system?

I have not interrupted the update; I am still waiting patiently for it to finish. Please let me know what additional information is required to troubleshoot this. Thank you.
 
Sorry, not a very useful answer but this sounds vaguely like an issue someone else raised in the last few weeks or so - ZFS and a lot of files was taking a long time. But I cannot remember enough of the details to help with search terms or any useful information (and whether I read about it here or on a FreeBSD mailing list.)
 
I think this isn't normal. Something bad happened. Have you read /var/log/message or dmesg -a in order to find an error?

Unless there is a serious hardware problem, you can use the automatic created boot environments to retrieve a somewhat stable system and maybe try again to upgrade base or come back to 13.2.
 
Did you run freebsd-update fetch install *before* the freebsd-update upgrade as stated in the Installation Instructions?

If not, you are missing some crucial bits needed for the upgrade:
Which leads to freebsd-update just stalling during the upgrade:
forums.freebsd.org

Late-breaking FreeBSD 14 breakage

From the FreeBSD Primary Release Engineering Team Lead at Some late-breaking FreeBSD 14 breakage
forums.freebsd.org forums.freebsd.org
Also documented here:
 
Update: I let the upgrade run overnight and it appears to have completed successfully. It was running for at least 16 hours (I don't know exactly because I left it unattended for a while). There were no errors reported on the console. I proceeded with the rest of the steps in the handbook, which were just these:

Code: 
$ pkg-static upgrade -f
$ freebsd-update install
$ shutdown -r now

And everything looks like it went fine, although the last freebsd-update install produced the following output:

Code: 
Creating snapshot of existing boot environment... done.
Installing updates...rmdir: ///usr/src/sys/contrib/openzfs/cmd/zvol_wait: Directory not empty
rmdir: ///usr/src/sys/contrib/openzfs/cmd/arc_summary: Directory not empty
 done.

This doesn't look fatal. But now I wonder... was freebsd-update building something from source? Could that be why it took so long?

It appears that I just needed to be patient. To address everyone's comments:

Emrion said:
Have you read /var/log/message or dmesg -a in order to find an error?
Click to expand...

Yes. Nothing related to the upgrade or a hardware failure showed up in either of those places, so it doesn't appear that there were any errors.

sko said:
Did you run freebsd-update fetch install *before* the freebsd-update upgrade as stated in the Installation Instructions?
Click to expand...

Yes. I frequently install patches with freebsd-update fetch and freebsd-update install, and I had done so just a few hours before starting the upgrade.

richardtoohey2 said:
ZFS and a lot of files was taking a long time
Click to expand...

Maybe this is it? My zroot ZFS pool, from which the FreeBSD system boots, is also used to store all of my data, which is quite a lot. The disks aren't particularly fast either, but I wouldn't call them slow. Perhaps that is related?

I don't want to mark this as solved yet because I'm still not exactly sure why the upgrade took this long and I would like some assurance that future upgrades won't take this long. If anyone has any technical knowledge of the FreeBSD update procedure and would like to share their insight as to what might cause an upgrade to appear stuck, I would greatly appreciate it. And if more information about my system is needed, I am happy to share it.

As far as I can tell, freebsd-update is just a shell script, so if I ever have time, I will do some digging into this for myself and report back here. I'm also interested in making the script more verbose. Installing updates... is less than helpful in scenarios like this; I'd rather have the script output specifically each step it is about to perform immediately before it performs it so I can get some more diagnostic information about where things are getting hung up.
 
jbancino said:
Maybe this is it? My zroot ZFS pool, from which the FreeBSD system boots, is also used to store all of my data, which is quite a lot. The disks aren't particularly fast either, but I wouldn't call them slow. Perhaps that is related?
Click to expand...
If that pool resides on spinning disks, this could be a factor. However, 16h doesn't sound "normal"...

Regarding those file/directory not found errors for /usr/src: TBH I've seen them on most systems where I have src installed. I suspect this is simply because I usually populate src via git (or svc before git) and only ever update that tree if I need it. So usually I forget to update that src tree before an upgrade, but I nuke /usr/src afterwards and do a fresh git clone...
 
I ran into the same issue on a root on ZFS system where it took hours for the the install to complete. /var/db/freebsd-update directory did have activity so I just let the update run and it finished overnight.
 
I have updated a root on ZFS today without any issues within an hour or so. The biggest part of the elapsed time has been spend for the download of the system and later of the packages. The system runs on a single SSD and the user data are a few 10GB only. The setup is rather simple. May be this is of some advantage.
 
jbancino said:
… at least 16 hours …
Click to expand...

Similarly (cross-reference):
  • {link removed}
jbancino said:
… As far as I can tell, freebsd-update is just a shell script, so if I ever have time, I will do some digging into this for myself and report back here. I'm also interested in making the script more verbose. …
Click to expand...

That's very kind, however please note that its days are numbered. This is not to discourage you; it's for you to prioritise, mid/long term.

Thanks


freebsd-update(8)-related reports in Bugzilla:
  • <{link removed}>
 
Last edited:
Same problem here. First part of the process went well, but after shutdown -r now and freebsd-update install things got slow... like suuuuper slow. I've been monitoring for a while and I can see things are being written, but with enormous pauses. It's been hours (at least 10h) and still didn't finish ?‍?
 
I have observed this behavior of the 2nd freebsd-update install hanging for ages on two different hosts now. As one of them was not exactly mission critical I decided to play around a little.

I intentionally performed a shutdown -r now several times while the 2nd freebsd-update install was hanging. I did this three times in a row to "ensure" that the behavior is consistent.
Then on the fourth run I first upgraded the bootloader, rebooted and suddenly the 2nd freebsd-update install completed within seconds as per expectations.

This host was running root-on-ZFS on a two-way mirror.
 
I experienced freebsd-update install being stuck when running it a first time before rebooting, and after rebooting as well. Running efibootmgr -v & cp /boot/loader.efi /boot/efi/efi/freebsd/ seems to have unblocked the situation for me.
 
I'm currently experiencing the same thing. I can see in top that freebsd-update is installing src files:

install -S -o 0 -g 0 -m 0755 5cdfb14ba4a732f19611842a52724913f9b02376c31bcd7805f7c0a7c525dc29 ///usr/src/tools/test/stress2/misc/geli.sh
Click to expand...

I'm upgrading from FreeBSD 13.4 and using zfs as the root filesystem. I'm just going to let it run, but there is something weird going on. It shouldn't take this long to copy these files.
 
Same here, like everyone is saying.

ZFS drive, after the download and reboot and then a freebsd-update install things got hella slow and unpredictable.

Would be nice if the devs put a loading bar so we would be able to known things are processing.

From FreeBSD 13.1 to 14 upgrade:

IMG_6064 Medium.jpeg
 
First_Law_of_Unix said:
Same here, like everyone is saying.

ZFS drive, after the download and reboot and then a freebsd-update install things got hella slow and unpredictable.

Would be nice if the devs put a loading bar so we would be able to known things are processing.

From FreeBSD 13.1 to 14 upgrade:

View attachment 19862
Click to expand...
and again: you should have upgraded first to 13.3-RELEASE to get the patches for freebsd-update. The "Not a directory" warnings are exactly what that fix was about, because some files became directories with 14.0-RELEASE


READ THE RELEASE-/INSTALL NOTES BEFORE A MAJOR RELEASE UPGRADE
 
sko said:
and again: you should have upgraded first to 13.3-RELEASE to get the patches for freebsd-update. The "Not a directory" warnings are exactly what that fix was about, because some files became directories with 14.0-RELEASE


READ THE RELEASE-/INSTALL NOTES BEFORE A MAJOR RELEASE UPGRADE
Click to expand...

I read somewhere that it’s not required. Can simply upgrade to major… guess that was false info.

So I used the boot environment to use the previous ZFS Snapshot and everything works again… not sure if I should upgrade to 13.3 and then to 14.1 now…
 
sko said:
you should have upgraded first to 13.3-RELEASE to get the patches for freebsd-update. The "Not a directory" warnings are exactly what that fix was about, because some files became directories with 14.0-RELEASE
Click to expand...
I think an update to the latest patch of 13.2-RELEASE would have sufficed in this case. There was excellent information about that in a message of a thread here.

Apart from the fact that BE-s saved your day, this blocking issue may have prevented you from your next reboot problems as you haven't mentioned any updates to the loader.
Read carefully:
 
You must log in or register to reply here.
Back
Top