aarch64: Trouble with CN99xx AHCI and FastLinQ QL41000 Controllers

tuaris

tuaris

I'm currently playing around with an ThunderX2 server from Gigabyte (R281-T91). For the moment it's a machine I purchased for learning more about ARM64, experimentation, and testing (I've until now been limited to just low powered SBC's). Eventually I want to use it as a build box for ports and releases, then further down the line use it for multi tenant web/jail hosting.

I'm able to boot 13.0-RC3, but I am unable to install it. 12.2-RELEASE and snapshots just kernel panic at boot when trying to load the SATA driver. 14.0 snapshots have the same results as 13.0.

The hardware isn't so new based on my understanding, and the drivers for the two devices I am having trouble with look like they are in FreeBSD. While these are not issues that would prevent me from using the machine, it would be preferable to not have to add extra hardware.

First here's a link to the full output of dmesg.boot and the output of pciconf. I've pasted the parts relevant to the SATA and network interface controllers below.

For the CN99xx [ThunderX2] Integrated AHCI/SATA 3 Host Controller, it seems to attempt to load a driver but it fails?

Code: 
ahci0: <AHCI SATA controller> mem 0x42220000-0x4222ffff at device 16.0 numa-domain 0 on pci0
pcib0: Failed to translate resource 0-ffffffffffffffff type 3 for ahci0
ahci0: 0x10000 bytes of rid 0x18 res 3 failed (0, 0xffffffffffffffff).
device_attach: ahci0 attach returned 6
ahci0: <AHCI SATA controller> mem 0x42200000-0x4220ffff at device 16.1 numa-domain 0 on pci0
pcib0: Failed to translate resource 0-ffffffffffffffff type 3 for ahci0
ahci0: 0x10000 bytes of rid 0x18 res 3 failed (0, 0xffffffffffffffff).
device_attach: ahci0 attach returned 6

Code: 
none0@pci0:0:16:0:    class=0x010601 rev=0x00 hdr=0x00 vendor=0x14e4 device=0x9027 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Broadcom Inc. and subsidiaries'
    device     = 'CN99xx [ThunderX2] Integrated AHCI/SATA 3 Host Controller'
    class      = mass storage
    subclass   = SATA
    cap 11[80] = MSI-X supports 4 messages
                 Table in map 0x18[0x0], PBA in map 0x18[0x8000]
    cap 09[90] = vendor (length 112)
none1@pci0:0:16:1:    class=0x010601 rev=0x00 hdr=0x00 vendor=0x14e4 device=0x9027 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Broadcom Inc. and subsidiaries'
    device     = 'CN99xx [ThunderX2] Integrated AHCI/SATA 3 Host Controller'
    class      = mass storage
    subclass   = SATA
    cap 11[80] = MSI-X supports 4 messages
                 Table in map 0x18[0x0], PBA in map 0x18[0x8000]
    cap 09[90] = vendor (length 112)

I'm not sure what's going on with the SATA controller. Without it I don't have anywhere to install the OS.

For the FastLinQ QL41000 Series 10/25/40/50GbE Controller, it doesn't even have the driver.

Code: 
pcib11: <PCI-PCI bridge> at device 11.0 numa-domain 0 on pci0
pci11: <PCI bus> numa-domain 0 on pcib11
pci11: <network, ethernet> at device 0.0 (no driver attached)
pci11: <network, ethernet> at device 0.1 (no driver attached)

Code: 
none2@pci0:11:0:0:    class=0x020000 rev=0x02 hdr=0x00 vendor=0x1077 device=0x8070 subvendor=0x1458 subdevice=0x0011
    vendor     = 'QLogic Corp.'
    device     = 'FastLinQ QL41000 Series 10/25/40/50GbE Controller'
    class      = network
    subclass   = ethernet
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 8 messages, 64 bit, vector masks
    cap 10[70] = PCI-Express 2 endpoint max data 128(512) FLR NS
                 max read 512
                 link x4(x8) speed 8.0(8.0) ClockPM disabled
    cap 11[b0] = MSI-X supports 129 messages
                 Table in map 0x20[0x0], PBA in map 0x20[0x1000]
    cap 03[d0] = VPD
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
    ecap 0002[148] = VC 1 max VC0
    ecap 0003[168] = Serial 1 0000000000000000
    ecap 0004[178] = Power Budgeting 1
    ecap 000e[188] = ARI 1
    ecap 0019[198] = PCIe Sec 1 lane errors 0
    ecap 0017[1f8] = TPH Requester 1
    ecap 0018[284] = LTR 1
    ecap 000b[28c] = Vendor [1] ID 0002 Rev 3 Length 256
    ecap 000b[38c] = Vendor [1] ID 0001 Rev 1 Length 56
    ecap 001f[3c4] = Precision Time Measurement 1
    ecap 000b[3d0] = Vendor [1] ID 0003 Rev 1 Length 84
    ecap 0015[424] = Resizable BAR 1
none3@pci0:11:0:1:    class=0x020000 rev=0x02 hdr=0x00 vendor=0x1077 device=0x8070 subvendor=0x1458 subdevice=0x0011
    vendor     = 'QLogic Corp.'
    device     = 'FastLinQ QL41000 Series 10/25/40/50GbE Controller'
    class      = network
    subclass   = ethernet
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 8 messages, 64 bit, vector masks
    cap 10[70] = PCI-Express 2 endpoint max data 128(512) FLR NS
                 max read 512
                 link x4(x8) speed 8.0(8.0) ClockPM disabled
    cap 11[b0] = MSI-X supports 129 messages
                 Table in map 0x20[0x0], PBA in map 0x20[0x1000]
    cap 03[d0] = VPD
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
    ecap 0003[168] = Serial 1 0000000000000000
    ecap 0004[178] = Power Budgeting 1
    ecap 000e[188] = ARI 1
    ecap 0017[1f8] = TPH Requester 1
    ecap 000b[28c] = Vendor [1] ID 0002 Rev 3 Length 256
    ecap 000b[38c] = Vendor [1] ID 0001 Rev 1 Length 56
    ecap 000b[3d0] = Vendor [1] ID 0003 Rev 1 Length 84
    ecap 0015[424] = Resizable BAR 1

According to qlnxe(), it should be possible to load a driver with kldload (or loader.conf). When I try a kldload if_qlnxe I get a file not found error. The driver doesn't seem to be preset on the boot only or installation ISO images for 13.0/14.0.

Are these specific devices not available on aarch64 or do I need to build my own custom ISO images?
 
To load the qlnxe(4) driver you must load the firmware first in /boot/loader.conf
Code: 
if_qlnxe_load="YES"

Have you done that?
forums.freebsd.org

QLogic FastLinQ 41112 Dual Por t 10GbE SFP+ Adapter

Hi, Our new storage server come with this network card: QLogic FastLinQ 41112 Dual Por t 10GbE SFP+ Adapter, PCIe Low Profile FreeBSD detect this hardware: # pciconf -lv none90@pci0:59:0:0: class=0x020000 card=0x00021077 chip=0x80701077 rev=0x02 hdr=0x00 vendor = 'QLogic Corp.'...
forums.freebsd.org forums.freebsd.org
 
Phishfry said:
To load the qlnxe(4) driver you must load the firmware first in /boot/loader.conf
Code: 
if_qlnxe_load="YES"

Have you done that?
forums.freebsd.org

QLogic FastLinQ 41112 Dual Por t 10GbE SFP+ Adapter

Hi, Our new storage server come with this network card: QLogic FastLinQ 41112 Dual Por t 10GbE SFP+ Adapter, PCIe Low Profile FreeBSD detect this hardware: # pciconf -lv none90@pci0:59:0:0: class=0x020000 card=0x00021077 chip=0x80701077 rev=0x02 hdr=0x00 vendor = 'QLogic Corp.'...
forums.freebsd.org forums.freebsd.org
Click to expand...
Looks like the driver isn't even available as module for that to work.

I managed to get FreeBSD installed to a logical drive connected the onboard SAS RAID controller and the systems can boot from that. It's not ideal since this is suposedly my storage array, but at least it gives me more of a chance to try and debug stuff. The first thing I tried was to build my own custom kernel, but it seems even that's not possible. I added "device qlnxe" to my custom configuration file, /usr/src/sys/arm64/conf/THUNDERX2. Running "make buildkernel KERNCONF=THUNDERX2" results in:

Code: 
config: Error: device "qlnxe" is unknown
 
If you look at the source code you will see there is a parent device.
Perhaps you need to include that in your kernconf
/usr/src/sys/dev/qlnx/
 
I think the issue with the SATA controller might be a bug, maybe I should report it?
I'll give it one more go with RC4 before I do that.
 
tuaris said:
I think the issue with the SATA controller might be a bug, maybe I should report it?
I'll give it one more go with RC4 before I do that.
Click to expand...
T-Daemon said:
There won't be a qlnxe(4) arm64 support added in 13.0-RC4. In the release engineering process [1] all changes to the base system are committed to main (14.0-CURRENT in this case) first, then merged to the stable (13) branch.

13.0-RC4 has been released yesterday on March 29th. There is no qlnxe(4) addition to arm64 in main nor in stable/13.

[1] https://docs.freebsd.org/en/articles/freebsd-releng/#introduction
Click to expand...

Damn, I maneged to overlook you are talking about the SATA controller. Sorry tuaris.
 
T-Daemon said:
Damn, I maneged to overlook you are talking about the SATA controller. Sorry tuaris.
Click to expand...
No problem.

I went ahead and reported the bug. I also managed to find a very inexpensive Broadcom NetXtreme II BCM57711 from eBay that seems to be supported by bge(), and it's available on arm64. Hopefully someone is working on getting the qlnxe() driver into ARM64 CURRENT? Or should I open a bug report for that too?
 
tuaris said:
Hopefully someone is working on getting the qlnxe() driver into ARM64 CURRENT? Or should I open a bug report for that too?
Click to expand...
It's unlikely arm64 or other architectures are going to get support. Inspecting the README* in the source package provided by the manufacturer, the driver is compatible only with 64-bit Intel EM64T and AMD64.

If the manufacturer doesn't provide support for arm64 there is nothing the FreeBSD team can do.

README_FOR_L2_AND_SRIOV.txt
Code: 
                     README File                                                           
           Cavium 45000 and 41000 series                                                   
25/40/100 Gigabit Ethernet Adapter Driver for FreeBSD 10.x/11.x                           
                                                                                          
           Cavium, Inc., All rights reserved.

.....
2. OS Support                                                                             
                                                                                          
The Qlogic 45xxx 25/40/100Gigabit Ethernet/CNA driver is compatible with the               
following OS platforms:                                                                   
* FreeBSD 10.x/11.x(64-bit) [Intel EM64T, AMD64]

README_FOR_RDMA.txt
Code: 
                        README File                                                        
                Cavium 45000 and 41000 series                                              
25/40 Gigabit Ethernet Adapter RDMA iWARP/RoCE(v1) Driver for FreeBSD 10.4 and 11.1        
                                                                                           
        Copyright (c) 2017-2019 Cavium Inc.,  All rights reserved
.....
2. OS Support                                                                             
                                                                                          
   The Qlogic 41000/45000 25/40 Gigabit Ethernet iWARP RDMA driver is compatible           
   with the following OS platforms:                                                       
    * FreeBSD 11.1(64-bit) [Intel EM64T, AMD64]
 
You must log in or register to reply here.
Back
Top