Solved Unexpected ubldr behavior on ClearFog base

pygr · Aug 4, 2018

Hi,

The issue: u-boot can find, load and transfer control to ubldr, but ubldr instantly fails with return code 0x1badab1.

Details: These instructions were followed to build and install FreeBSD. U-boot was downloaded and installed using instructions on that same web page. Booting that build from an SD card works just fine. Transferring that image to an M.2 SATA SSD and using dd to install the u-boot for SATA results in an installation that doesn't work so well. With appropriate choice of u-boot commands, ubldr can be found and loaded, but that results in failure of ubldr:

Code:

U-Boot SPL 2018.01-02337-gdc4398fdb8 (May 14 2018 - 18:46:33)
High speed PHY - Version: 2.0
Detected Device ID 6828
board SerDes lanes topology details:
| Lane #  | Speed |  Type       |
--------------------------------
|   0    |  3   |  SATA0       |
|   1    |  0   |  SGMII1      |
|   2    |  5   |  PCIe1       |
|   3    |  5   |  USB3 HOST1  |
|   4    |  5   |  USB3 HOST0  |
|   5    |  0   |  SGMII2      |
--------------------------------
PCIe, Idx 1: detected no link
High speed PHY - Ended Successfully
DDR3 Training Sequence - Ver TIP-1.29.0
DDR3 Training Sequence - Switching XBAR Window to FastPath Window
DDR3 Training Sequence - Ended Successfully
Trying to boot from SATA
AHCI 0001.0000 32 slots 2 ports 6 Gbps 0x3 impl SATA mode
Found 1 device(s).


U-Boot 2018.01-02337-gdc4398fdb8 (May 14 2018 - 18:46:33 +0200), Build: jenkins-u-boot-clearfog-model=base,variant=sata-10

SoC:   MV88F6828-A0 at 1600 MHz
DRAM:  1 GiB (800 MHz, 32-bit, ECC not enabled)
MMC:   mv_sdh: 0
AHCI 0001.0000 32 slots 2 ports 6 Gbps 0x3 impl SATA mode
Found 1 device(s).
Model: SolidRun Clearfog
Board: SolidRun ClearFog Base
SCSI:  AHCI 0001.0000 32 slots 2 ports 6 Gbps 0x3 impl SATA mode
Net:   eth1: ethernet@70000, eth2: ethernet@30000, eth3: ethernet@34000
Hit any key to stop autoboot:  0
=> scsi scan
scanning bus for devices...
  Device 0: (0:0) Vendor: ATA Prod.: TS64GMTS400 Rev: O091
            Type: Hard Disk
            Capacity: 61057.3 MB = 59.6 GB (125045424 x 512)
Found 1 device(s).
=> load scsi 0:1 0x1000000 ubldr
reading ubldr
296569 bytes read in 11 ms (25.7 MiB/s)
=> bootelf 0x1000000
CACHE: Misaligned operation at range [01000098, 01030128]
CACHE: Misaligned operation at range [01030128, 01030c2b]
CACHE: Misaligned operation at range [01030c2c, 01032634]
CACHE: Misaligned operation at range [01032634, 01034eaf]
CACHE: Misaligned operation at range [01034eb0, 010365c0]
CACHE: Misaligned operation at range [010365c0, 0103661c]
CACHE: Misaligned operation at range [0103661c, 01036694]
CACHE: Misaligned operation at range [01036694, 01036764]
CACHE: Misaligned operation at range [01036764, 01036770]
## Starting application at 0x01000098 ...
## Application terminated, rc = 0x1badab1

I suppose that return code comes from ubldr, but I can't locate it anywhere in the ubldr source code.

The strange thing (for me anyway) is that if those ubldr bits are written to the appropriate place on the SD card, there are no problems with booting from that storage medium:

Code:

U-Boot SPL 2018.01-02337-gdc4398fdb8 (May 14 2018 - 18:46:33)
High speed PHY - Version: 2.0
Detected Device ID 6828
board SerDes lanes topology details:
| Lane #  | Speed |  Type       |
--------------------------------
|   0    |  3   |  SATA0       |
|   1    |  0   |  SGMII1      |
|   2    |  5   |  PCIe1       |
|   3    |  5   |  USB3 HOST1  |
|   4    |  5   |  PCIe2       |
|   5    |  0   |  SGMII2      |
--------------------------------
PCIe, Idx 1: detected no link
PCIe, Idx 2: detected no link
High speed PHY - Ended Successfully
DDR3 Training Sequence - Ver TIP-1.29.0
DDR3 Training Sequence - Switching XBAR Window to FastPath Window
DDR3 Training Sequence - Ended Successfully
Trying to boot from MMC1


U-Boot 2017.07-05756-gc166d3d160 (Jul 10 2017 - 21:15:35 +0200)

SoC:   MV88F6828-A0 at 1600 MHz
I2C:   ready
DRAM:  1 GiB (800 MHz, ECC not enabled)
MMC:   mv_sdh: 0
Model: SolidRun Clearfog A1
Board: SolidRun ClearFog
Net: 
Error: ethernet@30000 address not set.
No ethernet found.
Hit any key to stop autoboot:  0
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
reading /boot.scr
205 bytes read in 10 ms (19.5 KiB/s)
## Executing script at 00200000
reading ubldr
1103491 bytes read in 65 ms (16.2 MiB/s)
CACHE: Misaligned operation at range [01000098, 01038ed8]
CACHE: Misaligned operation at range [01038ed8, 0103909c]
CACHE: Misaligned operation at range [0103909c, 010390a8]
CACHE: Misaligned operation at range [010390b0, 0103c2eb]
CACHE: Misaligned operation at range [0103c2ec, 0103e0dc]
CACHE: Misaligned operation at range [0103e0dc, 01042905]
CACHE: Misaligned operation at range [01042910, 010443e4]
CACHE: Misaligned operation at range [010443e4, 01044440]
CACHE: Misaligned operation at range [01044440, 01044a3c]
CACHE: Misaligned operation at range [01044a40, 010487b0]
## Starting application at 0x01000098 ...
Consoles: U-Boot console
Compatible U-Boot API signature found @0x3fbcf898

FreeBSD/armv6 U-Boot loader, Revision 1.2
(Sat Jul 28 11:05:33 PDT 2018 root@ghrBSD)

DRAM: 1024MB
Number of U-Boot devices: 1
U-Boot env: loaderdev not set, will probe all devices.
Found U-Boot device: disk
  Probing all disk devices...
  Checking unit=0 slice=<auto> partition=<auto>... good.
Booting from disk0s2a:
/boot/kernel/kernel data=0x781d80+0x52280 syms=[0x4+0xa7b00+0x4+0xa5c1a]

Hit [Enter] to boot immediately, or any other key for command prompt.


Type '?' for a list of commands, 'help' for more detailed help.
loader>

Is there something in the source code that ties proper functioning of ubldr to the SD card? Is there some way the build can be configured so that ubldr works well with the SATA interface?

Edit: I believe I've found the source code that generates the 0x1badab1: freebsd-master/stand/uboot/common/main.c. Apparently the error occurs because ubldr can't find the API magic signature. Now I get to figure out what that means and how to fix it. It's still a mystery why the error occurs for ubldr on the SATA drive and not for the same bits on the SD card.

balanga · Aug 4, 2018

Interesting project.... Are you building a router?

pygr · Aug 5, 2018

I'm attempting to make a firewall appliance that is more secure than my current cheap modem/gateway. Initially, I was intending to try out Opnsense or perhaps Pfsense, but I'm now thinking those are over the top for my needs (no services to the outside world, not even VPN).

balanga · Aug 5, 2018

Maybe take a look at OpenWrt... It's very configurable.

pygr · Aug 6, 2018

balanga

balanga said:
Maybe take a look at OpenWrt... It's very configurable.

Maybe I'll try that when I give up on FreeBSD for ClearFog.

I have discovered the thread https://forums.freebsd.org/threads/interfacing-freebsd-with-u-boot.61424/ you were involved in about a year ago and I think it gives enough details to solve the ubldr problem. Hopefully, the main issue is exposing the U-Boot API.

balanga · Aug 6, 2018

I'm hoping to get back to that project before long. I almost had it working, but after a setenv my GoFlex Home unit seemed to die and didn't show any signs of life. One thing that keeps me from trying to start it again is the lack of a kwboot for FreeBSD... I hate having to boot Linux just to achieve simple tasks which FreeBSD should be perfectly capable of doing.

pygr · Aug 9, 2018

I've been making some progress. More or less following the instructions here, I built U-Boot from source. The toolchain used was the current one from Linaro. The modifications to configs/clearfog_defconfig were

Code:

CONFIG_MVEBU_SPL_BOOT_DEVICE_SATA=y

and

Code:

CONFIG_API=y

. I'm aware that Crochet suggests a different way to expose the U-Boot API, but what I did seems to work.

With the fresh build of U-Boot and a fresh build of FreeBSD, the ClearFog boot got up to where control was passed to the kernel:

Code:

U-Boot SPL 2018.01-02337-gdc4398fdb8-dirty (Aug 07 2018 - 11:30:34)
High speed PHY - Version: 2.0
Detected Device ID 6828
board SerDes lanes topology details:
 | Lane #  | Speed |  Type       |
 --------------------------------
 |   0    |  3   |  SATA0       |
 |   1    |  0   |  SGMII1      |
 |   2    |  5   |  PCIe1       |
 |   3    |  5   |  USB3 HOST1  |
 |   4    |  5   |  USB3 HOST0  |
 |   5    |  0   |  SGMII2      |
 --------------------------------
PCIe, Idx 1: detected no link
High speed PHY - Ended Successfully
DDR3 Training Sequence - Ver TIP-1.29.0
DDR3 Training Sequence - Switching XBAR Window to FastPath Window
DDR3 Training Sequence - Ended Successfully
Trying to boot from SATA
AHCI 0001.0000 32 slots 2 ports 6 Gbps 0x3 impl SATA mode
Found 1 device(s).


U-Boot 2018.01-02337-gdc4398fdb8-dirty (Aug 07 2018 - 11:30:34 -0700)

SoC:   MV88F6828-A0 at 1600 MHz
DRAM:  1 GiB (800 MHz, 32-bit, ECC not enabled)
MMC:   mv_sdh: 0
AHCI 0001.0000 32 slots 2 ports 6 Gbps 0x3 impl SATA mode
Found 1 device(s).
*** Warning - bad CRC, using default environment

Model: SolidRun Clearfog
Board: SolidRun ClearFog Base
SCSI:  AHCI 0001.0000 32 slots 2 ports 6 Gbps 0x3 impl SATA mode
Net:   
Warning: ethernet@70000 (eth1) using random MAC address - d2:63:b4:cc:44:71
eth1: ethernet@70000
Warning: ethernet@30000 (eth2) using incremented MAC address - d2:63:b4:cc:44:72
, eth2: ethernet@30000
Warning: ethernet@34000 (eth3) using incremented MAC address - d2:63:b4:cc:44:73
, eth3: ethernet@34000
Hit any key to stop autoboot:  0
=> scsi scan
scanning bus for devices...
  Device 0: (0:0) Vendor: ATA Prod.: TS64GMTS400 Rev: O091
            Type: Hard Disk
            Capacity: 61057.3 MB = 59.6 GB (125045424 x 512)
Found 1 device(s).
=> go ${kernel_addr_r}
## Starting application at 0x00800000 ...
Consoles: U-Boot console 
Compatible U-Boot API signature found @0x3fb8f218

FreeBSD/armv6 U-Boot loader, Revision 1.2
(Wed Aug  8 12:14:08 PDT 2018 root@ghrBSD)

DRAM: 1024MB
Card did not respond to voltage select!
mmc_init: -95, time 19
Number of U-Boot devices: 1
U-Boot env: loaderdev not set, will probe all devices.
Found U-Boot device: disk
  Probing all disk devices...
  Checking unit=0 slice=<auto> partition=<auto>... good.
Booting from disk0s3a:
/boot/kernel/kernel data=0x82c5e4+0xaba1c syms=[0x4+0x8cd30+0x4+0xe2c1f]
/boot/dtb/armada-388-clearfog.dtb size=0x675c
Loading /boot/defaults/loader.conf
/
Hit [Enter] to boot immediately, or any other key for command prompt.
Booting [/boot/kernel/kernel]...               
Using DTB from loaded file '/boot/dtb/armada-388-clearfog.dtb'.
Kernel entry at 0xc00100...
Kernel args: (null)

at which point things stop and become unresponsive. So it looks like the dtb and kernel are loaded OK, but there is a problem after control is passed to the kernel :-(

ShelLuser · Aug 9, 2018

If you want to fully focus on security then I can't help wonder if OpenBSD wouldn't be a better choice here.

pygr · Aug 9, 2018

ShelLuser said:
If you want to fully focus on security then I can't help wonder if OpenBSD wouldn't be a better choice here.

Perhaps. How hard is it to build OpenBSD for Marvell Armada 388?

ShelLuser · Aug 9, 2018

pygr said:
Perhaps. How hard is it to build OpenBSD for Marvell Armada 388?

No clue, sorry. That's probably best asked on their forums.

pygr · Oct 17, 2019

I know this is an old thread, but there is something to add to the topic, so I thought I might share that.

Apparently the serial console went non responsive (no input or output) because the serial console device

Code:

marvell,armada-38x-uart

described in the dts file didn't correspond to a driver. That can be fixed in the dts file by replacing

Code:

marvell,armada-38x-uart

where it shows up with

Code:

snps,dw-apb-uart

. Or, if you prefer, modifying uart-dev-snps.c, before building the kernel.

That fix works with FreeBSD 12.X, but not 11.2. I'm still trying to figure that one out because OPNsense currently runs on 11.2, but not 12.X.

pygr