My exactly same problem was finally solved after weeks of research. Although this may seem trivial to experts, I believe what I have learned could be quite useful to someone like me. So I share what I found here.
Short answer:
The first configuration posted by jdn06 is correct, as long as the root CAs with hashed names were put in the CACERT_PATH directory.
The second config points the CACERT to the cert.pem from the ca_root_mss port, which may not be what we want because this is only good for verifing the CAs from the peer. It does not contain the intermediate CA (the chain.pem in Let's Encrypt's case) so the verification of the local side CA will fail.
Long answer:
Sendmail STARTTLS does two things: encryption and cert verification.
For the messages to be encrypted, the SERVER_CERT and SERVER_KEY are necessary. The SERVER_CERT contains the public key and will be delivered to the peer for them to encrypt/decrypt messages. Our server uses the SERVER_KEY which contains the private key to do the encryption and decryption. Basically that's it.
For the cert verification to work, we need to know how a cert gets verified first.
The following is a functional equivalent by using openssl to do the verification: (Using Let's Encrypt certs as example here)
openssl verify -CAfile /usr/local/etc/ssl/cert.pem -untrusted chain.pem cert.pem
cert.pem: OK
-CAfile: the trusted root certs you have. It's the one installed by the ca_root_nss port in this example.
-untrusted: the intermediate CA certs which sign your end point cert. In Let's Encrypt's case, this is the chain.pem obtained from the Let's Encrypt.
Last parameter: the end point cert you want to verify. cert.pem from Let's Encrypt in this example.
The cert contains an issuer information, which points to the CA who signs this cert. This means an end point cert can be traced back to an intermediate CA, and further up, until the root CA cert is reached, which is the cert that got signed by itself. The search ends here. If the whole chain can be traced successfully to a root cert, and that root cert happens to be in your trusted root CA list, then the verification is OK. The last parameter cert.pem is signed by Let's Encrypt's chain.pem, and chain.pem is signed by DST Root CA X3 which is one root CA listed in ca_root_nss, so the verification succeeds.
If the verification fails at its first stage, which means the intermediate cert signing the end point cert can not be found, you get a error #20 "unable to get local issuer certificate". If it fails at other step, it gives an error #2 "unable to get issuer certificate".
Sendmail does the same thing as the previous openssl example as shown in the following chart.
Root CA certs Intermediate CA cert End cert
openssl verify -CAfile /usr/local/etc/ssl/cert.pem -untrusted chain.pem cert.pem
sendmail STARTTLS: CACERT_PATH CACERT SERVER_CERT
Yes, you read it correctly. The CACERT_PATH and CACERT mean completely different things if you have an intermediate CA sitting in between. They are not redundant settings as you might think after reading many documents on the web. The CACERT_PATH is often not the directory of your CACERT.
Let's go with the easiest first: the SERVER_CERT.
Unlike most http server, sendmail seems to only take one cert from the SERVER_CERT. Thus pointing it to fullchain.pem, which contains your site cert followed by the Let's Encrypt cert, does not give you what you might think. Pointing it to the cert.pem from Let's Encrypt is a reasonable choice.
CACERT is the cert which signs your SERVER_CERT. More, it will also get delivered to the peer when sendmail is doing the SSL negotiation. Besides using it to verify your SERVER_CERT, the peer will check its issuer, to make sure it is signed by one of their trusted root CAs. So pointing it to the cert.pem from the ca_root_nss port is never a good idea. Firstly, it's big. The ca_root_nss contains 157 root certs as of this writing, which is unnecessarily big for negotiation. Worse, it does not contain the Let's Encrypt cert because the latter is not a root CA, thus it can't get your SERVER_CERT verified. You will get a error #20 "unable to get local issuer certificate" because the first step to find the issuer of your SERVER_CERT will fail in the ca_root_nss. You would want to use the chain.pem from Let's Encrypt as your CACERT because that's what signs your cert.pem.
When I say CACERT_PATH and CACERT are different, I am not 100% correct because sendmail also use CACERT in its root CA search. That's why the cert verification for the peer succeeded when the ca_root_nss is used as CACERT. But you need CACERT to point to your intermediate cert, so the root CAs part is taken care of by the CACERT_PATH.
You might remember that all documents tell you to do the following:
ln -s CA.pem `openssl x509 -hash -noout < CA.pem`.0
You may ask the same question as I did: why is that necessary? Why not just point a setting to your root CA file, such as the cert.pem from ca_root_nss. This is because your sendmail may receive E-mails from all over the world and each of the peer may have a different root CA. Doing a linear search over your hundreds of trusted root CAs is obvious not efficient. Therefore sendmail use CACERT_PATH, which points to a directory full of trusted root CAs, with each of the name hashed. When sendmail gets the intermediate cert from the peer while doing the SSL negotiation, it reads its issuer (its root CA) info, hashes it, and then gets the corresponding root CA by directly reading the hashed name file from the CACERT_PATH directory without searching.
Since you use Let's Encrypt cert rather than a self-signed cert, you probably want to receive E-mails from any potential senders in the world. Thus your CACERT_PATH directory is supposed to contain many hashed names, not just the one for verifying your own cert. I stopped my research here because the following solution already worked for me. Probably there is another simpler way to do the same. I write the following script to separate the ca_root_nss cert.pem into individual files each containing only one cert.
Code:
#!/bin/sh
# Separate the root cert into files each with only one cert and name hashed
# Required: ca_root_nss
RCert=/usr/local/etc/ssl/cert.pem
DESTDIR=/usr/local/etc/ssl/ROOT
mkdir -p $DESTDIR
cd $DESTDIR
rm -f *
cat $RCert | sed -E '/^(Certificate:|SHA1 Fingerprint|#| |$)/d' | split -p '-----BEGIN CERTIFICATE-----'
for a in `ls $DESTDIR`
do
mv $a `openssl x509 -hash -noout < $a`.0
done
The first part removes the non-encoding part of the cert.pem and then splits it into 157 separate cert files. The second part renames each of the files to a hashed name.
This is what I use in
/etc/mail/<site name>.mc
Code:
define(`CERT_DIR', `/usr/local/etc/letsencrypt/live/<my site FQDN>')dnl
define(`confSERVER_CERT', `CERT_DIR/cert.pem')dnl
define(`confSERVER_KEY', `CERT_DIR/privkey.pem')dnl
define(`confCLIENT_CERT', `CERT_DIR/cert.pem')dnl
define(`confCLIENT_KEY', `CERT_DIR/privkey.pem')dnl
define(`confCACERT', `CERT_DIR/chain.pem')dnl
define(`confCACERT_PATH', `/usr/local/etc/ssl/ROOT')dnl
define(`confLOG_LEVEL', `14')
The only functional difference with jdn06's first version is that my CACERT_PATH points to a directory where my script extracts the ca_root_nss to.
The LOG_LEVEL 14 reveals more details in the log. With the aforementioned configuration, the log for receiving an E-mail from gmail looks like the following. (I also have SMTP AUTH configured, which is not in the scope of this reply. It can be done easily by following the online FreeBSD Handbook. )
Code:
Aug 25 04:19:47 server sm-mta[56394]: NOQUEUE: connect from mail-wm0-f43.google.com [74.125.82.43]
Aug 25 04:19:47 server sm-mta[56394]: AUTH: available mech=SCRAM-SHA-1 DIGEST-MD5 OTP CRAM-MD5 NTLM LOGIN PLAIN ANONYMOUS, allowed mech=GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN
Aug 25 04:19:47 server sm-mta[56394]: v7P4JlRJ056394: Milter: no active filter
Aug 25 04:19:47 server sm-mta[56394]: STARTTLS: x509 cert verify: depth=0 /CN=<server.my-domain>, state=0, reason=unable to get certificate CRL
Aug 25 04:19:47 server sm-mta[56394]: STARTTLS: x509 cert verify: depth=1 /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3, state=0, reason=unable to get certificate CRL
Aug 25 04:19:47 server sm-mta[56394]: STARTTLS: x509 cert verify: depth=2 /O=Digital Signature Trust Co./CN=DST Root CA X3, state=0, reason=unable to get certificate CRL
Aug 25 04:19:49 server sm-mta[56394]: STARTTLS: x509 cert verify: depth=0 /C=US/ST=California/L=Mountain View/O=Google Inc/CN=smtp.gmail.com, state=0, reason=unable to get certificate CRL
Aug 25 04:19:49 server sm-mta[56394]: STARTTLS: x509 cert verify: depth=1 /C=US/O=Google Inc/CN=Google Internet Authority G2, state=0, reason=unable to get certificate CRL
Aug 25 04:19:49 server sm-mta[56394]: STARTTLS: x509 cert verify: depth=2 /C=US/O=GeoTrust Inc./CN=GeoTrust Global CA, state=0, reason=unable to get certificate CRL
Aug 25 04:19:49 server sm-mta[56394]: STARTTLS=server, relay=mail-wm0-f43.google.com [74.125.82.43], version=TLSv1.2, [COLOR=#ff0000]verify=OK[/COLOR], cipher=AES128-GCM-SHA256, bits=128/128
Aug 25 04:19:49 server sm-mta[56394]: STARTTLS=server, cert-subject=/C=US/ST=California/L=Mountain+20View/O=Google+20Inc/CN=smtp.gmail.com, cert-issuer=/C=US/O=Google+20Inc/CN=Google+20Internet+20Authority+20G2, verifymsg=ok
Aug 25 04:19:49 server sm-mta[56394]: AUTH: available mech=SCRAM-SHA-1 DIGEST-MD5 EXTERNAL OTP CRAM-MD5 NTLM LOGIN PLAIN ANONYMOUS, allowed mech=GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN
Aug 25 04:19:50 server sm-mta[56394]: v7P4JlRK056394: from=<my-email@gmail.com>, size=2239, class=0, nrcpts=1, msgid=<CAKXD16RWSrR62uCMdHQ-MrMThZhHhKk9uVESYsJfCM5VL4HB0g@mail.gmail.com>, proto=ESMTPS, daemon=IPv4, relay=mail-wm0-f43.google.com [74.125.82.43]
When receiving E-mails, the sendmail STARTTLS is in its server role. The server side cert was verified first, then the client cert (Gmail) verified. Starting from the end cert (depth=0), then the intermediate cert (depth=1) and finally the root cert (depth=2). Both were successful.
When sending E-mails, the sendmail STARTTLS is in its client role. The log looks like the following.
Code:
Aug 25 04:19:51 server sm-mta[56395]: v7P4JlRK056394: SMTP outgoing connect on <server.my-domain>
Aug 25 04:19:51 server sm-mta[56395]: STARTTLS=client, init=1
Aug 25 04:19:51 server sm-mta[56395]: STARTTLS=client, start=ok
Aug 25 04:19:52 server sm-mta[56395]: STARTTLS: x509 cert verify: depth=0 /C=US/ST=California/L=Sunnyvale/O=Yahoo Inc./OU=Information Technology/CN=*.am0.yahoodns.net, state=0, reason=unable to get certificate CRL
Aug 25 04:19:52 server sm-mta[56395]: STARTTLS: x509 cert verify: depth=1 /C=US/O=Symantec Corporation/OU=Symantec Trust Network/CN=Symantec Class 3 Secure Server CA - G4, state=0, reason=unable to get certificate CRL
Aug 25 04:19:52 server sm-mta[56395]: STARTTLS: x509 cert verify: depth=2 /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5, state=0, reason=unable to get certificate CRL
Aug 25 04:19:52 server sm-mta[56395]: STARTTLS=client, relay=mx-tw.mail.gm0.yahoodns.net., version=TLSv1.2, verify=OK, cipher=ECDHE-RSA-AES128-GCM-SHA256, bits=128/128
Aug 25 04:19:52 server sm-mta[56395]: STARTTLS=client, cert-subject=/C=US/ST=California/L=Sunnyvale/O=Yahoo+20Inc./OU=Information+20Technology/CN=*.am0.yahoodns.net, cert-issuer=/C=US/O=Symantec+20Corporation/OU=Symantec+20Trust+20Network/CN=Symantec+20Class+203+20Secure+20Server+20CA+20-+20G4, verifymsg=ok
Aug 25 04:19:54 server sm-mta[56395]: v7P4JlRK056394: to=my-addr@yahoo.com.tw, delay=00:00:04, xdelay=00:00:04, mailer=esmtp, pri=32544, relay=mx-tw.mail.gm0.yahoodns.net. [203.188.197.111], dsn=2.0.0, stat=Sent (ok dirdel)
Aug 25 04:19:54 server sm-mta[56395]: v7P4JlRK056394: done; delay=00:00:04, ntries=1
Aug 25 04:19:54 server sm-mta[56395]: STARTTLS=client, SSL_shutdown failed: -1
Only the server side cert is verified when my sendmail is the client. The verification was successful.
The "unable to get certificate CRL" is not considered an error. Including an exhaustive list of cert revocation list is not scalable anymore and is gradually obsolete in favor of OCSP.
