Solved pkg fails with mystery error after upgrade from 13.3 to 14.1

Nyantastic

Nyantastic

Today I upgraded from FreeBSD 13.3 to 14.1. The upgrade is finished:
Code: 
$ uname -a
FreeBSD mikan 14.1-RELEASE-p5 FreeBSD 14.1-RELEASE-p5 GENERIC i386

I did the required steps but now pkg seems to be corrupted.

Some of the software is not working, for example this:
Code: 
# service apache24 start
Performing sanity check on apache24 configuration:
ld-elf.so.1: Shared object "libssl.so.111" not found, required by "libdb-18.1.so"
Starting apache24.
ld-elf.so.1: Shared object "libssl.so.111" not found, required by "libdb-18.1.so"
/usr/local/etc/rc.d/apache24: WARNING: failed to start apache24
#

So I tried the advice found here:
forums.freebsd.org

Solved - pkg not working - ld-elf.so.1: Shared object "libssl.so.111" not found, required by "pkg"

#uname -a FreeBSD 12.0-BETA1 FreeBSD 12.0-BETA1 r339435 GENERIC amd64 I have got a size mismatch error with pkg and did a "IGNORE_OSVERSION=yes pkg-static update -f", which did not solve this error. pkg was not working: ld-elf.so.1: Shared object "libssl.so.111" not found, required by "pkg"...
forums.freebsd.org forums.freebsd.org

But running pkg upgrade
results in this "abort trap" error:
Code: 
# pkg upgrade
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
Updating database digests format: 100%
Checking for upgrades (824 candidates): 100%
Processing candidates (824 candidates): 100%
Checking integrity...Child process pid=1615 terminated abnormally: Abort trap

So I tried the advice in this discussion:
forums.freebsd.org

pkg-static upgrade -f pkg fails with Abort trap

Hi, after running FreeBSD-update install (to get to 12.1-RELEASE), then running pkg-static upgrade -f pkg fails.... Running "pkg bootstrap -f" recommended Updating FreeBSD repository catalogue... FreeBSD repository is up to date. All repositories are up to date. Checking for upgrades (488...
forums.freebsd.org forums.freebsd.org

Code: 
# pkg-static update -f
Updating FreeBSD repository catalogue...
Fetching meta.conf: 100%    178 B   0.2kB/s    00:01   
Fetching data.pkg: 100%    7 MiB   7.2MB/s    00:01   
Processing entries: 100%
FreeBSD repository update completed. 34414 packages processed.
All repositories are up to date.

But things are still not working.

I cannot use pkg bootstrap -f due to the following error:
Code: 
# pkg bootstrap -f
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:14:i386/latest, please wait...
Installing pkg-1.21.3...
package pkg is already installed, forced install
Extracting pkg-1.21.3: 100%
pkg-static: Package database is busy while closing!

pkg check -B --all pours out line upon line of errors about the missing libraries:
Code: 
(bind-tools-9.20.2) /usr/local/bin/nslookup - required shared library libssl.so.111 not found
(bind-tools-9.20.2) /usr/local/bin/nslookup - required shared library libcrypto.so.111 not found
(bind-tools-9.20.2) /usr/local/bin/nsupdate - required shared library libssl.so.111 not found
(bind-tools-9.20.2) /usr/local/bin/nsupdate - required shared library libcrypto.so.111 not found

At the end of the following, I also got the error "pkg: Package database is busy while closing!", which has also come up a few times.

I seem to have libcrypto.so.30 on my system in /lib.

I upgraded all the packages before the install of the new release of FreeBSD 14.1
 
Since I can't seem to do much with pkg at the moment, I tried to install /usr/ports/security/openssl111/ with make DISABLE_VULNERABILITIES=yes. Although it installed correctly, that got me /usr/local/lib/libcrypto.so.11 and I need libcrypto.so.111 apparently.
Code: 
# service apache24 start
Performing sanity check on apache24 configuration:
ld-elf.so.1: Shared object "libssl.so.111" not found, required by "libdb-18.1.so"
Starting apache24.
ld-elf.so.1: Shared object "libssl.so.111" not found, required by "libdb-18.1.so"
/usr/local/etc/rc.d/apache24: WARNING: failed to start apache24
 
RypPn said:
I seem to recall pkg-static upgrade -f, not update.
Click to expand...
The update fetches the catalogue of files whereas upgrade fetches the software packages themselves. There is more in man pkg-update and man pkg-upgrade. I am basically following the advice given in the threads linked. However, I'm not sure any pkg commands will help here, since the disappearance of libcrypto.so.111 seems to be related to the installation, since it was there in FreeBSD 13.3.
 
Have you ever built things from ports? Any chance this is a mix up of ports/packages?

FreeBSD 14.1 has OpenSSL 3.0 in the base, FreeBSD 13.x was OpenSSL 1.1.1.

You need to build/install everything with OpenSSL 3.0 but how you get there might depend what you've done in the past.
 
Charlie_ said:
libcrypto.so.111 is included in misc/compat13x.
Click to expand...
Thanks Charlie_. At the moment at least apache24 is running OK and pkg has stopped producing the strange errors. For the next person who comes along, the recovery looks like this (as root/superuser):
Code: 
cd /usr/ports/ports-mngt/portsnap
make install clean
portsnap fetch
portsnap update
cd /usr/ports/misc/compat13x/
make install clean
I'm now running pkg upgrade, and it seems to be installing all the things for freebsd-14 libraries. I'm not really sure what happened since I thought I had followed the instructions correctly, but it's probably an error on my part.

Edit: It seems I spoke too soon:
Code: 
pkg upgrade

The operation will free 387 MiB.
21 KiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching portsnap-1.1_3.pkg: 100%   21 KiB  21.8kB/s    00:01    
Checking integrity...Child process pid=31574 terminated abnormally: Abort trap
# pkg-static bootstrap -f
pkg(8) is already installed. Forcing reinstallation through pkg(7).
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:14:i386/latest, please wait...
Installing pkg-1.21.3...
package pkg is already installed, forced install
Extracting pkg-1.21.3: 100%
pkg-static: Package database is busy while closing!
#
 
For a major upgrade (version X.n to X+1.n) it should have been install kernel, reboot, install userland (this would have handled some of OpenSSL), then re-install (or rebuild if you use ports) ports/packages.

But for some reason you've not got any/all of the OpenSSL-related updates.
 
richardtoohey2 said:
For a major upgrade (version X.n to X+1.n) it should have been install kernel, reboot, install userland (this would have handled some of OpenSSL), then re-install (or rebuild if you use ports) ports/packages.

But for some reason you've not got any/all of the OpenSSL-related updates.
Click to expand...
Thank you for your input. I believe I did all of these stages. I have done upgrades of FreeBSD several times without any issues, including an amd64-based system last week which I upgraded from 13.3 to 14.1. However, on this system, after I installed the compat13x, the "old" software started working again, but pkg is still producing the unusual errors.

The "busy while closing" error has exactly one occurrence in the whole internet:

"pkg-static: Package database is busy while closing!" at DuckDuckGo

DuckDuckGo. Privacy, Simplified.
duckduckgo.com duckduckgo.com

The "child process abort trap" error is also fairly obscure:

 
You didn't complete the upgrade of your userland application after the change of the major version of the system.
First you have to install "pkg" using "pkg bootstrap -f" which will force the upgrade of the pkg to the newer version regardless of it's existence.
Then you can perform upgrade of all installed pkgs.

If your database of the pkg is corrupted you can write down the prime-origin list of the pkgs then you can restore it's backup:
Code: 
Restore a backup    database:
         % rm /var/db/pkg/local.sqlite
         % xzcat /var/backups/pkg.sql.xz | pkg shell
 
VladiBG said:
You didn't complete the upgrade of your userland application after the change of the major version of the system.
First you have to install "pkg" using "pkg bootstrap -f" which will force the upgrade of the pkg to the newer version regardless of it's existence.
Click to expand...

Thank you for your input. I did try that one several times, as well as trying with pkg-static. Perhaps we should note that it actually says that is "recommended" rather than being a necessary part of the install. It fetched about 800 upgrades but somehow the installation didn't proceed correctly. Next time I do this I'll memoize the commands as I do it.

VladiBG said:
Then you can perform upgrade of all installed pkgs.

If your database of the pkg is corrupted you can write down the prime-origin list of the pkgs then you can restore it's backup:
Code: 
Restore a backup    database:
         % rm /var/db/pkg/local.sqlite
         % xzcat /var/backups/pkg.sql.xz | pkg shell
Click to expand...

Thank you again. As I received your message I was in the middle of typing out that that was what I was planning next, since as far as I can make out from the error reports reached by the Google searches, the database is corrupted. The above code is from the end of man pkg in case anyone wants to refer to it. I'm running pkg right now and it is getting past "Checking integrity" but still not completed.
 
Unfortunately this didn't work:
Code: 
     Restore a backup database:
           % rm /var/db/pkg/local.sqlite
           % xzcat /var/backups/pkg.sql.xz | pkg shell
The output is
Code: 
# pkg -d upgrade
...
Checking integrity...Abort trap (core dumped)
I get back the "Child process terminated abnormally" if I leave out the -d option.

So unfortunately this problem isn't resolved yet.
 
thindil said:
Maybe you are a victim of this bug: PR 280345. At least, for me, disabling legacy providers in OpenSSL fixed the "Child process terminated abnormally" error.
Click to expand...
I don't know, but I don't have the word legacy in this configuration file:
Code: 
[ben@mikan] {14:59 52} ~ 506 $ grep legacy /etc/ssl/openssl.cnf 
[ben@mikan] {14:59 54} ~ 507 $
 
What's the output of:
Code: 
pkg shell
.databases
PRAGMA integrity_check;
.quit

Can you run
pkg prime-origins or pkg info ?
 
VladiBG said:
What's the output of:
Code: 
pkg shell
.databases
PRAGMA integriy_check;
.quit
Click to expand...
Nothing appears:
Code: 
# pkg shell
SQLite version 3.45.2 2024-03-12 11:06:23
Enter ".help" for usage hints.
sqlite> .databases
main: /var/db/pkg/local.sqlite r/w
sqlite> PRAGMA integrity_check;
sqlite> .quit
#
 
It's the best if you write down the output of pkg prime-origins those are the pkg that you installed and in a case you need to delete all pkg and reinstall them you will need this list.

So far everything looks good. What's happen when you run pkg update
 
VladiBG said:
It's the best if you write down the output of pkg prime-origins those are the pkg that you installed and in a case you need to delete all pkg and reinstall them you will need this list.
Click to expand...

Thank you, someone else also suggested that above.

Everything is working, so I don't need to delete any of the installed packages, as far as I know. The only problem is with pkg. As far as ripping everything out and replacing it goes, since this is an i386 system it only has about a year or so before the OS has to be completely replaced anyway, since FreeBSD 15 won't support i386.
VladiBG said:
So far everything looks good. What's happen when you run pkg update
Click to expand...

Code: 
# pkg update
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
 
Ok, and when you try to upgrade the installed pkgs with pkg upgrade ?
If there's error post the entire output with debug option pkg -d upgrade
 
VladiBG said:
Ok, and when you try to upgrade the installed pkgs with pkg upgrade ?
If there's error post the entire output with debug option pkg -d upgrade
Click to expand...
The output is very large, so I've attached it rather than quoting. The commands were as follows.
Code: 
# pkg -d upgrade > upgrade-out.txt 2>&1 
Abort trap (core dumped)
#
 

Attachments

I tried compiling a pkg with debugging support:
Code: 
# cd /usr/ports/ports-mgmt/pkg
# DEBUG_FLAGS=-g CFLAGS=-pipe make
# cd work/pkg-1.21.3/src/
using the advice found here: https://stackoverflow.com/questions...sd-utility-from-source-with-debugging-symbols

Somehow or another the debugging didn't seem to work perfectly. I ran "pkg upgrade" inside the debugger, the same crash occurred, and then I ran gdb again on the core file.
Code: 
# gdb pkg-static pkg-static.core
The output looks like this:
Code: 
Reading symbols from pkg-static...
[New LWP 100073]
Core was generated by `/usr/ports/ports-mgmt/pkg/work/pkg-1.21.3/src/pkg-static upgrade'.
Program terminated with signal SIGABRT, Aborted.
Sent by thr_kill() from pid 72395 and user 0.
#0  thr_kill () at thr_kill.S:4

warning: 4    thr_kill.S: No such file or directory
(gdb) bt
#0  thr_kill () at thr_kill.S:4
#1  0x00c867aa in __raise (s=6) at /usr/src/lib/libc/gen/raise.c:50
#2  0x00cf842d in abort () at /usr/src/lib/libc/stdlib/abort.c:64
#3  0x00655dd0 in xcalloc ()
#4  0x00659c0d in pkg_addfile_attr ()
#5  0x00659a40 in pkg_addfile ()
#6  0x006d804d in pkgdb_load_files ()
#7  0x006d6702 in pkgdb_ensure_loaded_sqlite ()
#8  0x006d679b in pkgdb_ensure_loaded ()
#9  0x008b56a0 in pkg_conflicts_append_chain ()
#10 0x008ac4cf in pkg_jobs_check_conflicts ()
#11 0x008ab59f in pkg_jobs_solve ()
#12 0x00629d11 in exec_upgrade ()
#13 0x0061f535 in main ()
Somehow or another the variables and code lines don't appear in this debugger so I couldn't get any further. pkg_addfile_attr is in the libpkg subdirectory in the file pkg.c.
 
Are you running out of memory?
this is i386 why don't you use amd64?
did you try to force upgrade of all pkg with pkg upgrade -f
 
You must log in or register to reply here.
Back
Top