jails upgrading older jails to 14.2 is taking far too long

There is a bug of which I am aware that causes updates from earlier version of freebsd to 14.X to take extraordinary lengths of time to complete an update. I ran into this last year and I had to put this a aside due to other pressures. Now I need to update a bunch of jails to 14.2 from freebsd versions running from 12.3 to 13.x. I need a way to get the patch for this problem loaded before updating the jails via iocage. Is this possible or am I fated to go through a dozen or so 20 hour upgrades?
 
Are these thin or thick jails?

Alternatively to upgrading existing jails, you could just spawn new jails with the new version and migrate any data. Or if you did this the smart way the data would actually be stored outside of the jail (on it's own filesystem for example) then you just have to mount it.

That said, I generally don't have an issue with upgrading a jail, certainly not the kind you're seeing. It's just as fast (or slow depending on your viewpoint) as upgrading the host. But I don't use iocage, so perhaps the issue is with iocage's upgrade process?
 
No, the issue was identified long ago as a flaw in the update script. The same problem occurs with iocell. The fix, when updating directly, was to do a freebsd-update fetch install before updating. Which only works for systems running 13.x. Those on 12 are screwed.

I got caught with this one because I forgot what I ran into last year. This whole thing was supposed to take a weekend a year ago and it has just become a nightmare because of this bug.

Moving the data is fine. I can do that between jails. The issue is the applications. The reason we use fat jails is because some of our applications cannot be upgraded due to lost features in the later versions and getting older versions of packages installed can be problematic (if the packages can be obtained at all). Cannot carry forward 'technical debt' don't you know. (tech speak for 'I am too busy to bother with keeping things working for somebody else (patches welcome. . .)'.)
 
Iocage can take an extremely long time to upgrade because of /usr/src.

I remember a discussion somewhere on the TrueNAS forums saying that for some reason Iocage builds thick jails with /usr/src.

 
Iocage can take an extremely long time to upgrade because of /usr/src.

/usr/src updates always take extremely long when done via freebsd-update as it patches *every* *single* *file* *one* *at* *a* *time* in there. Especially on spinning rust which can only handle abysmally low IOPS, this will take forever.
The solution is simple: don't update /usr/src that way. Nuke it, update the system/jail and re-populate the sources via git or tarball (thats *if* you actually need the sources).

I have no idea if/how iocage can do it (best guess: it can't - they removed a lot of useful features from it when botching it together in python...), in iocell you can simply remove 'src.txz' from the 'ftpfiles' option, run the update and then re-populate /usr/src and add 'src.txz' to 'ftpfiles' (for the sake of correctness). For basejails this isn't necessary as they don't have those files in their dataset (unless you have done something weird to them).

In general: major release updates for basejails *can* also take a while (I often see 3-5 minutes on slower hosts until etcmerge starts). I usually see that the first jail takes rather long (with zfs doing a lot of work), the following ones are a lot quicker - I suspect this boils down to zfs having everything in ARC, so all zfs-related operations are faster by magnitudes. Merging etc can still take a while, depending on how much additional stuff there is installed in the jail.


regarding that truenas link: they are doing so much weird stuff with their system and deviate a lot from common practices (and sense) in many places, I'm not surprised they have weird behaviour on upgrades. Also they are lagging (hopelessly) behind, usually using EOL OS versions, so this abomination is in no way any source of viable information regarding FreeBSD...
 
/usr/src updates always take extremely long when done via freebsd-update as it patches *every* *single* *file* *at* *once* in there. Especially on spinning rust which can only handle abysmally low IOPS, this will take forever.
The solution is simple: don't update /usr/src that way. Nuke it, update the system/jail and re-populate the sources via git or tarball (thats *if* you actually need the sources).

I have no idea if/how iocage can do it (best guess: it can't - they removed a lot of useful features from it when botching it together in python...), in iocell you can simply remove 'src.txz' from the 'ftpfiles' option, run the update and then re-populate /usr/src and add 'src.txz' to 'ftpfiles' (for the sake of correctness). For basejails this isn't necessary as they don't have those files in their dataset (unless you have done something weird to them).

In general: major release updates for basejails *can* also take a while (I often see 3-5 minutes on slower hosts until etcmerge starts). I usually see that the first jail takes rather long (with zfs doing a lot of work), the following ones are a lot quicker - I suspect this boils down to zfs having everything in ARC, so all zfs-related operations are faster by magnitudes. Merging etc can still take a while, depending on how much additional stuff there is installed in the jail.


regarding that truenas link: they are doing so much weird stuff with their system and deviate a lot from common practices (and sense) in many places, I'm not surprised they have weird behaviour on upgrades. Also they are lagging (hopelessly) behind, usually using EOL OS versions, so this abomination is in no way any source of viable information regarding FreeBSD...
TrueNAS no longer does FreeBSD. The last version is/was 13.3. They have switched their focus to Linux with TrueNAS SCALE.
 
Back
Top