Solved pkg(8) and fetch(1) errors with let's encrypt certs after 12.2-RELEASE

[dch]

This caused me a fair bit of grief last week, so hopefully this saves somebody else some.

The issue is that after running freebsd-update on an existing system, to get to 12.2-RELEASE, that running pkg (or any fetch-related command), errors similar to the following turn up:

Certificate verification failed for /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority
34375790592:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915

The issue is that the local certificate stores are missing or invalid:

# ls -AFGhl /etc/ssl/cert.pem /usr/local/etc/ssl/cert.pem /usr/local/openssl/cert.pem
lrwxr-xr-x  1 root  wheel    43B Oct 18 13:04 /etc/ssl/cert.pem@ -> ../../usr/local/share/certs/ca-root-nss.crt
-rwxr-xr-x  1 root  wheel    33B Oct  3 16:06 /usr/local/etc/ssl/cert.pem*
-rwxr-xr-x  1 root  wheel    30B Oct  3 16:06 /usr/local/openssl/cert.pem*

I don't yet know how this happened, but here is the simple fix:

# pkg delete -yf ca_root_nss
# rm -rf /usr/local/etc/ssl/cert.pem /usr/local/openssl/cert.pem
# pkg add /var/cache/pkg/ca_root_nss-3.58.txz
# certctl rehash
# ls -AFGhl /usr/local/etc/ssl/cert.pem /usr/local/openssl/cert.pem
-rw-r--r--  1 root  wheel    Oct 18 13:04 /usr/local/etc/ssl/cert.pem
-rw-r--r--  1 root  wheel   768K Oct 18 13:04 /usr/local/openssl/cert.pem

I don't know if certlctl rehash is needed, but its a cool new feature of 12.2-RELEASE so we should show it off

How I diagnosed this error

what's frustrating is that using curl(1) has no issues whatsoever:

# curl -4vsSLo /dev/null https://pkg.example.org/meta.conf
*   Trying 95.216.67.172:443...
* Connected to pkg.example.org (95.216.67.172) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /usr/local/share/certs/ca-root-nss.crt ******* curl uses the ca_root_nss package certs file directly ************
*  CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2612 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
# fetch -o /dev/null https://pkg.example.org/meta.conf
Certificate verification failed for /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
34370654208:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:

We can run pkg with -ddd flags to see more info, and we see it using a different cert store. Let's reproduce that in fetch directly as its less noisy:

# fetch -o /dev/null https://pkg.example.org/meta.conf -vv
scheme:   "https"
user:     "pkg"
host:     "pkg.example.org"
port:     "0"
document: "/meta.conf"
---> pkg.example.org:443
resolving server address: pkg.example.org:443
SSL options: 82004854
Peer verification enabled
Using CA cert file: /usr/local/etc/ssl/cert.pem  **************** fetch uses the user-modifiable cert store *********************
Certificate verification failed for /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
34370654208:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
fetch: https://pkg.example.org/meta.conf: Authentication error

It took me a couple of careful re-re-readings to notice this key difference!

However, simply re-installing security/ca_root_nss over the top doesn't change the existing certs, as these are potentially user-modifiable, so the port only installs them if they're missing, the first time around:

Scanning /usr/local/etc/ssl/certs for certificates...
# pkg install -r pkg -f ca_root_nss
Updating pkg repository catalogue...
pkg repository is up to date.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):
Installed packages to be REINSTALLED:
        ca_root_nss-3.58 [pkg]
Number of packages to be reinstalled: 1
...
# pkg list ca_root_nss
/etc/ssl/cert.pem
/usr/local/etc/ssl/cert.pem.sample ******* a sample file, only overwritten if cert.pm is missing ********
/usr/local/openssl/cert.pem.sample ******* a sample file, only overwritten if cert.pm is missing ********
/usr/local/share/certs/ca-root-nss.crt ***** always updated as part of deployed port **********
/usr/local/share/licenses/ca_root_nss-3.58/LICENSE
/usr/local/share/licenses/ca_root_nss-3.58/MPL20
/usr/local/share/licenses/ca_root_nss-3.58/catalog.mk

# ls -AFGhl /etc/ssl/cert.pem /usr/local/etc/ssl/cert.pem /usr/local/openssl/cert.pem
lrwxr-xr-x  1 root  wheel    43B Oct 18 13:04 /etc/ssl/cert.pem@ -> ../../usr/local/share/certs/ca-root-nss.crt
-rwxr-xr-x  1 root  wheel    33B Oct  3 16:06 /usr/local/etc/ssl/cert.pem*
-rwxr-xr-x  1 root  wheel    30B Oct  3 16:06 /usr/local/openssl/cert.pem*

So, to fix it we need to purge the damaged files, and allow the port to re-install itself to clean up. Finally, I think this is a bug in curl; it should default to the user-modifiable cert store, and not directly to the ca_root_nss installed one.