Raspberry Pi 4B - reboot process stops during Upgrade from FreeBSD 14.0-RELEASE-p8 to 14.1-RELEASE-p2 (with mountroot: mmcsd0 error 19). Please help.

blackfoxx

blackfoxx

Hi there :)

My name is Stan from Germany, I'm 45 y.o. and experienced/advanced FreeBSD admin/user since 2019.
(Previously admin/user of several Linux-Distros, until "systemd" occured.)
So I'm familiar with UNIX-like systems...
I'm working with FreeBSD workstations and servers since Version 12.
As a side project, I'm running a private FreeBSD webserver with Apache, sqlite, PHP, calDAV and webDAV.
This system is installed on a "SanDisk Extreme PRO" SDHC-Card, which runs within a "Raspberry Pi 4B (8GB)" at my home.
Since the initial setup in 2019 it runs just fine, without any issues or errors.
The upgrades from Version 12 to 13 and 13 to 14 went through without any problems too.
So I was very happy with my setup for about 5 years... Until last Sunday (2024-06-30).

During my regular maintenance, I decided to upgrade it from 14.0 to 14.1, as the release was already some weeks ago.
But before doing so, I updated it to the latest (14.0-RELEASE-p8) version, followed by pkg update/upgrade,
to ensure the latest versions of all installed packages too.
This pre-process went through without any problems (as usual since 2019!).

Now we're coming to the upgrade process from 14.0-RELEASE-p8 to 14.1-RELEASE-p2,
which resulted in an error (for the first time, in my case)...

Protocol:
Code: 
blackfoxx@BERRYFOXX:~ $ sudo freebsd-update upgrade -r 14.1-RELEASE
Password:
Looking up update.FreeBSD.org mirrors... 3 mirrors found.
Fetching metadata signature for 14.0-RELEASE from update2.freebsd.org... done.
Fetching metadata index... done.
Fetching 1 metadata patches. done.
Applying metadata patches... done.
Inspecting system... done.

The following components of FreeBSD seem to be installed:
kernel/generic world/base

The following components of FreeBSD do not seem to be installed:
kernel/generic-dbg world/base-dbg

Does this look reasonable (y/n)? y

Fetching metadata signature for 14.1-RELEASE from update2.freebsd.org... done.
Fetching metadata index... done.
Fetching 1 metadata patches. done.
Applying metadata patches... done.
Fetching 1 metadata files... done.
Inspecting system... done.
Fetching files from 14.0-RELEASE for merging... done.
Preparing to download files... done.
Fetching 3271 patches... (...) done.
Applying patches... done.
Fetching 444 files... (...) done.
Attempting to automatically merge changes in files... done.
The following file could not be merged automatically: /etc/ntp/leap-seconds
Press Enter to edit this file in /usr/bin/vi and resolve the conflicts
manually...
(manually merged "leap-seconds" and "sshd_config" files, according to guidelines by the FreeBSD-Forum)
The following files will be removed as part of updating to 14.1-RELEASE-p2:
(tl;dr)
The following files will be added as part of updating to 14.1-RELEASE-p2:
(tl;dr)
The following files will be updated as part of updating to 14.1-RELEASE-p2:
(tl;dr)
To install the downloaded upgrades, run "/usr/sbin/freebsd-update install".

blackfoxx@BERRYFOXX:~ $ sudo /usr/sbin/freebsd-update install
Installing updates...
Kernel updates have been installed. Please reboot and run
"/usr/sbin/freebsd-update install" again to finish installing updates.

blackfoxx@BERRYFOXX:~ $ sudo reboot

Then the following sh*t happens:
Code: 
> Raspberry Pi 4B (8GB) Firmware boots up successfully...
> FreeBSD 14.1 Kernel + Modules are loading successfully...
> FreeBSD Loader takes over and starts to boot...
>> After a few seconds it stops booting with the message: "mmcsd0: Timeout"
>> followed by:
>>> mountroot: waiting for device /dev/mmcsd0s2a...
>>> Mounting from ufs:/dev/mmcsd0s2a failed with error 19.
>>> ? - List valid disk boot devices
>>>> mountroot> ?
>>>> mountroot: mmcsd0
>>>>> mountroot> ufs:/dev/mmcsd0s2a
>>>>> mountroot: waiting for device /dev/mmcsd0s2a...
>>>>> Mounting from ufs:/dev/mmcsd0s2a failed with error 19.
-END OF STORY- (so far)
- From now on every attempt to (re)boot fails again at this same stage.
- My "SanDisk Extreme PRO" SDHC-Card is still fine, without any (physical) errors.
- I checked it 3 times and in every possible way with my FreeBSD 14.0 Workstation.
- fsck and fsck_ffs detected NO errors or issues > file system is clean!
- I can access the card via my "Transcend" Cardreader and read all files without any issues.
- Another SD-Card, with the original "Raspberry Pi OS", boots up normally.
- If I write my previous made Backup-Image (with the FreeBSD 14.0 Installation) back to the same SD-Card, it boots up normally as it always did before.
- So any Hardware issues (regarding the RPI 4B or the SD-Card) are absolutely excluded.

Filesystem-Layout of mmcsd0:

FS_Server_2024-07-01_18-04-08.png


- I already spent 12+ hours searching the web for "Mounting from ... failed with error 19" issues and tried a dozen "solutions".
- I tried the default versions of "/boot/loader.conf", "/etc/fstab", "/etc/rc.conf" and "/etc/sysctl.conf".
- I tried the latest version of "u-boot.bin"
- I tried the latest Firmware for my RPI model, but with no luck at all...
- I wasn't able to find out what this "error 19" in fact means.

Now I reached the end of my knowledge (and patience) and hope that some (or at least one) of you experts here will be able to help me getting my server up and running again!
(But please avoid suggestions like "You have to install/setup from scratch...", as it will be the absolute last solution I wanna do.)

Thanks in advance!
 
Last edited by a moderator:
blackfoxx said:
please avoid suggestions like "You have to install/setup from scratch..."
Click to expand...
But maybe it would be good to try - on another SD card if you have one - just to make sure that FreeBSD 14.1 will definitely work on this Pi?

So at least you'll know it should definitely work (or not) before you carry on trying to fix the upgrade.
 
This was discussed last month in the mailing lists (see link below). It's likely you have a Raspberry Pi 4B with a revision ending in B0T as opposed to the newer revision ending in C0T. I use a Raspberry Pi 4B as a networking device (since the genet driver performs far better than on Linux) and had the exact same problem when upgrading from 13.2 to 13.3. Then I inserted a new memory card with 14.0 on it and upgraded to 14.1. Same result, so it affects both 13.3 and 14.1. For now I'm back to running 14.0 and it's working fine.

https://lists.freebsd.org/archives/freebsd-arm/2024-May/003924.html

I briefly checked through the bug reports but it seems none of those involved in the linked conversation have submitted one.
 
Thank you Professor_Fate !
This Information helped me (at least) to stop trying to fix here "something" which is out of my influence.
So I'm gonna stay with (fine working) 14.0 too, until a working solution is (hopefully) found...
 
Hi Blackfoxx,

Here is the bug report on the issue but doesn't look like much has been done with it: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=280131

From my understanding reading through it this problem only seems to happen on the 2GB RPI4s and possibly the 1GB versions as well. Mine is a 2GB. For now I'll keep it running 14.0 and I'll try upgrading to 14.2 when that comes out. Hopefully it will be fixed in that version. I always keep an offline tar file consisting of of all the config files I edit on a system. That way if the system fails to boot after upgrade I can easily image the card back to 14.0, pkg install my programs, untar my config files, and then be back and running in a matter of minutes.
 
It looks like it is going to be wait and hope (I don't think there will be that many FreeBSD developers able to look at this and fix it? Probably countable on one hand?)

Or try another Pi or two until you find one that works. Or back to Linux for Pi-usage.

I don't think there are going to be any good solutions.

Hopefully I'll be proved wrong, but might be worthwhile coming up with a plan B.
 
Ah, okay. Thanks for your answers so far...
I was wondering a bit, as my RPI 4B model is a 4GB one. But it definitely has this problem too.
My thoughts were similar to Professor_Fate's.
As my webserver doesn't contain any sensible stuff, it doesn't really matter to let it run with 14.0 for another while.
Next attempt to upgrade would be 14.2. If it fails again for the same reasons, I put my RPI 4b into my "currently unusable electronics box" and will continue with an 1 y.o. amd64 Intel NUC (which I can buy from a friend for 50€) and a fresh FreeBSD setup from scratch.
After running the RPI 4b with FreeBSD for about 4 years now, without any issues(!), I wouldn't mind to invest some time and 50 bucks to start over again.
Maybe the amd64 platform is also less "problematic" than the arm64 one...
As I have another 2 Intel amd64 workstations running with FreeBSD, where upgrading (even to new major versions - from 12 to 13 to 14) never resulted in a mess or disaster, my impression is that amd64 is more "robust".
Back then, when I started "experimenting" with the RPI, I had no expectations. It was more kinda fun. And after 1 or 2 years in 24/7 service I already wondered that it still runs and works like a charm. So it felt like a "bonus" to me.
But maybe now it's time to move forward again and try something new...
 
For many years we used a desktop computer (with dual port intel PRO1000) as a multi vlan routing/firewall device. It simply ran PF, Unbound, and ISC-DHCP-Server. After it died I realized it was overkill for what it was doing anyway. Replaced with the Raspberry Pi 4 and the nice thing is the config files needed just a couple little changes. Even a Raspberry Pi for this use case just works with no headaches.

Going back to the first link I provided above, it seems to hint that this problem was not happening in CURRENT so it's possible it's already fixed and will trickle down to 14.2-RELEASE or maybe 15.0-RELEASE.
 
blackfoxx said:
What to do when 14.0 reaches EOL at 2024-09-30?
Click to expand...
Professor_Fate said:
Going back to the first link I provided above, it seems to hint that this problem was not happening in CURRENT so it's possible it's already fixed and will trickle down to 14.2-RELEASE or maybe 15.0-RELEASE.
Click to expand...
If I had an RPI with this issue, I would check out 14.1-STABLE. If it shows the same issue, move on to 15.0-CURRENT.

If 15.0-CURRENT boots ok, build an CURRENT image for production use, excluding debugging features and use it until 15.0-STABLE and ultimately 15.0-RELEASE.
 
Thanks for your hints and suggestions! I will give them a try a.s.a.p.
Of course I wouldn't like to replace an actually working device with an kinda overkill one. That's not my philosophy.
For the moment I just hope that 14.2 or 15.0 will work again like 14.0.
We'll see...
 
richardtoohey2 said:
It looks like it is going to be wait and hope (I don't think there will be that many FreeBSD developers able to look at this and fix it? Probably countable on one hand?)

Or try another Pi or two until you find one that works. Or back to Linux for Pi-usage.

I don't think there are going to be any good solutions.

Hopefully I'll be proved wrong, but might be worthwhile coming up with a plan B.
Click to expand...
Plan B? You could use a high quality S10 USB flash drive (32GB are cheap now, hard to find 8GB and 16GB) and install your FreeBSD image there. Change the EEProm value to 0x0F14 or 0xF214 with RaspiConfig utility to boot from the USB Flash drive or SSD (0x4) before trying to boot from a MicroSD card. Use the UGREEN case with the M.2 NVME stick enclosed inside and hook the USB 3.0 cable to the Blue USB 3.0 ports on the Raspberry Pi 4B.

Has anybody compiled the raspiconfig source code to work directly as a FreeBSD application?

https://ghostbsd-arm64.blogspot.com/2022/09/freebsd-140-compiling-kernel-for.html FreeBSD 14.x compiling kernel on Raspberry Pi.
https://ghostbsd-arm64.blogspot.com/2024/01/hdmi-audio-sound-patches-into-ghostbsd.html
Adding VCHIQ HDMI Audio patches to FreeBSD or to GhostBSD Kernel source code to play audio files on the Television speakers. This is good for watching youtube videos on the HDMI Television. Will these patches get incorporated into the build image for Raspberry Pi 3/4 ??
Where to look at built binary FreeBSD images for the Raspberry Pi 3 and 4 hardware. Will there be a binary build image for the Raspberry Pi 5?

https://download.freebsd.org/releases/arm64/aarch64/ISO-IMAGES/14.1/ RPI3/4

Will the extra uart serial port hardware on the Raspberry Pi 4B have devices drivers built into the binary image file to interface on Serial Port 2 to a GPS ascii uart interface to make a local network time server? Just asking. The development tree does use 2 uart serial interface /dev/tty0 and /dev/tty1 but not tty2, tty3, tty4, tty5

On the 4 hard do test with a USB flash drive or move up to a USB SSD like the Ugreen case with the M.2 NVME stick 250, 500GB or 1 Terrabyte. Makes a great Poudriere Package builder, with NGINX webserver to serve your freshly built Arm64 Packages.
https://ghostbsd-arm64.blogspot.com/2023/10/poudriere-setup-to-compile-from.html

Wish you well on your use case for your Raspberry Pi 4B, 3B, 400 hardware. It works great as a low cost educational ARM64 (aarch64) Do-It-Yourself (DIY) workstation.
https://ghostbsd-arm64.blogspot.com/2023/12/ghostbsd-arm64-future-road-map-ideas-to.html

Anybody work out PXE booting from a local TFTP server your Raspberry Pi hardware and wrote some documentation steps to follow?
 
You must log in or register to reply here.
Back
Top