Some time ago, I had a similar issue with delayed ssh login after upgrading to FreeBSD 11.2.
Initially I solved this by switching on the server side to
security/openssh-portable, without exactly knowing why this was the fix. Later, I found out, that the port created a whole new set of public-key files into
/usr/local/etc/ssh witch got the actual (correct)
root@server.domain added to the key, while the original key files in
/etc/ssh got an old no more working domain added. Incidently I changed the base domains in the LAN shortly before. Anyway, the actual fix was to refresh the key files. However, some of the machines are multihomed, and on those I removed the
root@server.domain hint altogether, in order to prevent any future distortions.