Dell T640: install & boot from NVME

[dmitry n medvedev]

good morning, nice FreeBSD community.

The Context:

1. Dell T640:
   1. CPU: 2x Silver
   2. RAM: 4x SK Hynix 64GB DDR4 2666 MHz HMAA8GL7CPR4N-VK TF AC
   3. HBA330:
      1. 8x HGST 800GB SSD SAS 12Gbps HUSMM1680ASS201
      2. slot: 7
   4. NVME-to-PCIE adapter: GLOTRENDS PA21 2 Ports M.2 NVMe to PCIe 4.0 X8 Adapter without PCIe Bifurcation Function
      1. 2x Crucial P5 Plus M.2 500GB
      2. slot 3
   5. GPU: NVidia Quadro P2200
      1. slot: 1
   6. iDRAC:
      1. Boot Mode: UEFI
      2. Firmware Version: 7.00.00.171
      3. Integrated Devices:
         1. Slot Bifurcation: Auto Discovery Of Bifurcation
      4. BIOS: 2.21.0

The Problem:
I can install FreeBSD onto mirrored M.2. The iDRAC creates a "boot record" FreeBSD. The records is grayed out and hence "unavailable". Please have a loot at the attached photo.

The Question: what am I doing wrong?

Thank you for all ideas!

best regards,
Dmitry

 

[rmartin]

Good day Dmitry,

Still these days, and depending on the machine and purpose, I still consider if I really (really) need UEFI.
Personally, and in most cases, a BIOS/CSM boot "works for me".

I would tick the following options:

• Disable secure boot, enable CSM
• Which partition scheme did you choose for those NVMe drives? How did you mirror them?.
  • During the installation, I usually would use Auto (ZFS), mirror both disks, choose a GPT (BIOS+UEFI) partition scheme, and check if I need to increase the swap size.
• Not directly related, but disable the X2Apic mode under the processor settings. This AFAIK to date will prevent you also from booting.

 

[sko]

Last time I had to deal with dell servers and the same problem came up, the answer turned out to be very simple: It's not the fault of the OS/Bootloader, it's simply (yet another) artificial limitation by DELL. Booting from NVMe as well as bifurcation was (and maybe still is?) reserved for the top-end variants - not by hardware limitations, but by firmware.

The offered "solution" back then was to boot another bootloader (e.g. rEFInd) from USB and let this boot the actual OS. Our solution was to dump those 2 remaining DELL servers and replace them with Supermicro...


Regarding BIOS/CSM boot: there's really no reason to still use that. It's much more fragile and inflexible than UEFI and nowadays causes more problems than it might solve...
UEFI implementations at least on servers have been stable, 'correct'/standards conformant and 'just working'™ for a long time now. It's usually only gaming hardware that always has to add something "special" which causes problems by violating the standards. But this phenomenon has also been true for classic BIOS on those platforms, so it's not really related to UEFI...

 

[dmitry n medvedev]

while being in the process of applying changes suggested by rmartin , I must admit that I have failed to boot even from the Dell BOSS-S1 card an hour ago. The same problem - BIOS cannot find the device to boot from.

PS: will report back the progress

 

[rmartin]

while being in the process of applying changes suggested by rmartin , I must admit that I have failed to boot even from the Dell BOSS-S1 card an hour ago. The same problem - BIOS cannot find the device to boot from.

PS: will report back the progress

Some quick thoughts

My bad if my initial response gave the impression that UEFI installs are not working properly or are error prone. It was just a personal passing comment from someone that regrettably deals with a lot of old and legacy hardware.

From what I understand, your T640 is a 14th generation model, which should be officially capable of booting from a PCIe NVMe drive in UEFI mode. Earlier generations did not support this functionality.

About the BOSS-S1 card, I found some information here that might be of interest.
Also, maybe unrelated, but you might want to consider flashing that H330 firmware to an HBA330.

At the end of the day, you could follow sko suggestion to use an additional USB/vFlash bootloader, or maybe less ideally, simply carry out a zroot installation on those 8x SSD SAS.

 

[sko]

simply carry out a zroot installation on those 8x SSD SAS.

putting a uefi (fat32) partition + bootloader on one/some of them should also work. the root pool can still reside on nvme.
i.e. install normally on nvme; before rebooting create yet another uefi partition on one/some of the SAS SSDs and copy the loader there (or just dd the partition from one of the nvmes)