freebsd-update has been running for 4 days

DaLynX

DaLynX

Hi,

I am looking for help trying to understand what is taking to so long.

I started a jail upgrade from 12.2 to 14.1 on Sunday. Previous ones have been long already, like one day. Okay. This one has been running for four days already. Here's the status:
Code: 
The following files will be updated as part of updating to
14.1-RELEASE-p2:
/.cshrc
/.profile
/COPYRIGHT
/bin/[
/bin/cat
/bin/chflags
/bin/chio
/bin/chmod                                                                         /bin/cp
/bin/csh
/bin/date
/bin/dd
/bin/df
/bin/domainname
/bin/echo
/bin/ed
/bin/expr                                                                          /bin/freebsd-version                                                               /bin/getfacl                                                                       /bin/hostname
To install the downloaded upgrades, run 'tmpcviffpdx [options] install'.           Installing updates...                                                              Kernel updates have been installed.  Please reboot and run                         'tmpcviffpdx [options] install' again to finish installing updates.
Installing updates...

It's been like this for at least three days, let's say, assuming the first one was used for the previous steps.

I can see it's running because install processes move on from file to file. But it's just painstakingly slow.

Code: 
root@vm:/home/lynx# ps auxww | grep install
root  51612   0.0  0.0  13456    8  0  IW+  -           0:00.00 /bin/sh /tmp/tmpcviffpdx -b /iocage/jails/accounting/root -d /iocage/jails/accounting/root/var/db/freebsd-update/ -f /iocage/jails/accounting/root/etc/freebsd-update.conf -r 14.1-RELEASE install
root  62514   0.0  0.0  13456  792  0  S+   Sun05       0:56.52 /bin/sh /tmp/tmpcviffpdx -b /iocage/jails/accounting/root -d /iocage/jails/accounting/root/var/db/freebsd-update/ -f /iocage/jails/accounting/root/etc/freebsd-update.conf -r 14.1-RELEASE install
root  89557   0.0  0.0  12864 2068  0  D+   14:43       0:00.00 install -S -o 0 -g 0 -m 0755 d75abedf73ee99dc733ff00b2680c0869cf954bc00aefd0d7bdc276b87f96bc2 /iocage/jails/accounting/root//usr/src/tools/test/stress2/misc/nfs4.sh
root  89559   0.0  0.1  12888 2140  1  S+   14:43       0:00.00 grep install

The server shows no CPU load, has over 20 GB free, no io (using top -m io -o total) now that I have solved a sendmail issue on another jail.

What could be causing this? Any ideas on how to find the issue? So hopefully I could at least try to solve it...
 
That install is in a 'D' state, meaning it's waiting on disk. Have you checked /var/log/messages? It might be stuck constantly retrying a write to a bad spot on the disk or have some other disk I/O related issues (often the case when the disk itself is near death).
 
Why would it move to the next item after some time then? Multiple retries ending up working?

I don't have messages from the last few hours but earlier in the day(/night) I have multiple entries like this, thanks for the hint:

Code: 
Jul 17 03:05:59 vm kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 41494, size: 40960

I will look that up.
 
DaLynX said:
Multiple retries ending up working?
Click to expand...
Or end up failing. It'll do 5 retries if I recall correctly, for every block it tries to write. Overall it can seemingly take forever to finish.

DaLynX said:
Code: 
Jul 17 03:05:59 vm kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 41494, size: 40960
Click to expand...
Swap is on disk I presume?

A hostname like 'vm' seems to indicate this FreeBSD install is a virtual machine? Maybe the issues are related to that. What kind of virtualization is it running on?
 
Yes swap partition:
Code: 
=>        34  1048575933  ada0  GPT  (500G)
          34           6        - free -  (3.0K)
          40        1024     1  freebsd-boot  (512K)
        1064         984        - free -  (492K)
        2048     4194304     2  freebsd-swap  (2.0G)
     4196352  1044377600     3  freebsd-zfs  (498G)
  1048573952        2015        - free -  (1.0M)

It's a vm on a proxmox (QEMU/KVM) linux host.
 
DaLynX said:
Yes swap partition:
Code: 
=>        34  1048575933  ada0  GPT  (500G)
          34           6        - free -  (3.0K)
          40        1024     1  freebsd-boot  (512K)
        1064         984        - free -  (492K)
        2048     4194304     2  freebsd-swap  (2.0G)
     4196352  1044377600     3  freebsd-zfs  (498G)
  1048573952        2015        - free -  (1.0M)

It's a vm on a proxmox (QEMU/KVM) linux host.
Click to expand...
I had raspberry pis on bad sd cards that performed faster. But this is the same single sata disk server am I correct? Did you already about getting additional disks (ssd) at OVH perhaps? unsure if they will diverge from their offered hardware though.
 
It is the same host although this VM has more resources (4GB RAM) and usually performs fine.

I noticed I'm installing / updating src although I don't use it. And the update seems to run faster if I stop other jails first, and after I removed some 500 000 files in a clientmqueue folder in one of them.

No swap error messages so far either so maybe there was just too much load?

In any case you're right it seems to argue in favour of a migration to beefier hardware. Or some rationalizing of the services we run?

Edit: the update actually finished soon after I stopped other jails.
 
Last edited:
If swap starts to fail you can also expect it to drag services along with it, it's not a FreeBSD issue. It's like expecting a car to work fine with a faulty fuel pump.
 
DaLynX said:
It is the same host although this VM has more resources (4GB RAM) and usually performs fine.

I noticed I'm installing / updating src although I don't use it. And the update seems to run faster if I stop other jails first, and after I removed some 500 000 files in a clientmqueue folder in one of them.

No swap error messages so far either so maybe there was just too much load?

In any case you're right it seems to argue in favour of a migration to beefier hardware. Or some rationalizing of the services we run?
Click to expand...
yes I'd opt for something with ssd, the SYS-2-SSD-32 seems like nice option, its bit more money per month though
 
Ah. That would be a bit tight in space. I was more thinking about the SYS-1-SAT-32 which has already double the RAM we have and comes with an array of 4×2TB HDD. So if I put all that in a zpool that would give me like 6TB usable + survival in case of loss of 1 drive, and the load would be split over several drives so it would probably perform much better.
 
I just did a source upgrade form 13.1 --> 13.2 --> 14.0 --> 14.1 on a box just fine, probably a bit overkill but it worked :-)
 
Actually just did 13.0 -> 13.3 -> 14.1 on some older laptop I'm preparing for my 6 year old to start discovering BSD on :)
Though 13.3 didn't boot after first freebsd-update install, because of parallel or serial port issue. And I forgot the bios password, so spend a good amount of time first finding a away to remove the bios password of a hp 6540b, and apparently there is a tool, HPBR to create some funky usb stick to do so. Now the laptop does whine about not having a product ID a.o. during boot but at least the bios password was removed and I could disable parallel port and serial so 13.3 booted properly and I could finish that and do upgrade to 14.1 after that. Now busy with the desktop, cant really force a 6 year old to start work from a tty console.
 
My testing (which needs more) with freebsd-update seems its normally fine to jump through multiple versions as long as you download a new freebsd-update tool and run it instead of using the one that is present on an older system. The only notes I have found that suggest updating to intermediate versions seems related to fixing bugs in freebsd-update by getting a new one through that full system update and then proceeding on. Another observation I thought I found was to update off of a version with known issue to the closest one to get there quickly with less changes and then proceed forward but that seemed more version specific for when that applies.
 
Mirror176 said:
… download a new freebsd-update tool and run it …
Click to expand...

I found a dead (end of life) 13.2-RELEASE-p4 (patch level 7) in a VirtualBox snapshot, used it for an upgrade to 14.1-RELEASE:
  • ignoring intermediate updates and upgrades
  • using the dead /usr/sbin/freebsd-update – without downloading anything newer.
First and last screenshots attached.

I assume that the rm-related lines during installation of world relate to FreeBSD-EN-23:12.freebsd-update.
 

Attachments

  • 2024-07-21 10-55 freebsd-update upgrade -r 14.1-RELEASE.png
    2024-07-21 10-55 freebsd-update upgrade -r 14.1-RELEASE.png
    45.1 KB · Views: 63
  • 2024-07-21 14-24 world updated, rm stuff.png
    2024-07-21 14-24 world updated, rm stuff.png
    48.2 KB · Views: 58
13.x to 14.x isn't a big leap. The notes for some of the other older jumps saying to update to a certain version first that I tracked any meaning for were just trying to get users to a version with a fixed freebsd-update and then have them proceed from there. My response then is, "why not just always get a newer freebsd-update and run that?" I haven't yet found issues using a much newer freebsd-update on the earlier systems that first supported freebsd-update.
 
I am missing something here. I have experienced what I consider to be abnormally long delays in completing a jail upgrade to 14.1. These jails were all running 13.2.

The discussion above refers to an updated freebsd-update. However, before upgrading the jails to 14.1 it was necessary to update the host to 14.1 and do a pkg upgrade. I understand that upgrading a jail to 14.1 on a host running 13.2 is not supported and likely would not work. How would one get any version of freebsd-update other than the most recent?
 
You must log in or register to reply here.
Back
Top