Other Server (U.2) NVMe hot-swap support status?

[spork]

This is honestly just a revisit of a post from 2019 (see below). I've been asking around on IRC, our Supermicro vendor, etc. and just not coming up with any info on how well hot swap works in practice with NVMe drives on a backplane in a server running FreeBSD. I'm not sure if some of the big guys out there just don't speak publicly about their hardware choices in this day and age or if people still find that SATA/SAS are sufficient. My understanding is that NVMe on laptops and desktops where the "drive" is just inserted in a slot on the mainboard works fine, but also nobody really has reason to be hot-swapping those.

And a recap, still same set of questions:

I'm about to buy a few more servers and it seems like NVMe is the way things are going, and the pricing is decent enough. The "U.2" standard seems here to stay and there are plenty of servers with NVMe "hot swap" bays for "U.2" format drives.
...
Anyone with practical experience who can weigh in on whether hot swap exists and more importantly, how well it works?

Of note that was 12.2, we'd be talking 14.latest here.

Original thread, where the consensus was "doesn't work" and/or "if it does, too soon to use it":

Other Thread 'NVMe hot swap working?'

Jan 7, 2019

I'm about to buy a few more servers and it seems like NVMe is the way things are going, and the pricing is decent enough. The "U.2" standard seems here to stay and there are plenty of servers with NVMe "hot swap" bays for "U.2" format drives.

I've done a fair amount of googling and looking through mailing list archives and I'm not really seeing anything definitive on whether or not FreeBSD supports hot swap on NVMe drives or not. I imagine it's a tricky mess - it's not like pulling a drive off the SATA or SAS bus, you're basically removing a PCIe device without powering the host off...

• spork
• Replies: 14
• Forum: Storage

 

[cracauer@]

Good question. I just missed a deal on a hotswap frame, so I'm still not able to test.

Probably also depends on the specific equipment.

 

[PMc]

I'm not sure if some of the big guys out there just don't speak publicly about their hardware choices in this day and age

That happens. I don't really know why. I would occasionally like to have a talk about some topic from the pro realm, but to no avail.

And a recap, still same set of questions: Of note that was 12.2, we'd be talking 14.latest here. Original thread, where the consensus was "doesn't work" and/or "if it does, too soon to use it"

Nice question. I think the matter is, does the hardware do ist correctly? I have no hardware to test such, but on the software side, if I get it correctly, the U.2 is just an omnibus, and the actual connection is PCI. The question then is, can we hotswap PCI? Or does the following look any different with U.2?

# devinfo
    pcib0
      pci0
        hostb0
        pcib1
          pci1
            nvme0
# devctl detach nvme0
# devinfo
    pcib0
      pci0
        hostb0
        pcib1
          pci1
#

Apr 20 00:09:05 <kern.crit> disp kernel: [427617] nda0: <KBG50ZNV512G KIOXIA HP01AN00 234PH47KQMLK>
Apr 20 00:09:05 <kern.crit> disp kernel: [427617] (nda0:nvme0:0:0:1): Periph destroyed
Apr 20 00:09:05 <kern.crit> disp kernel: [427617] nvme0: detached

I don't see reason why it should not be possible to rip the device out of the system at this point.
But then, how do we get it back? A few unsuccesful ventures, and then:

# devctl reset -d  pci1
# devctl
    pcib0
      pci0
        hostb0
        pcib1
          pci1
            nvme0
#

Apr 20 00:13:54 <kern.crit> disp kernel: [427906] nvme0: <Generic NVMe Device> mem 0xf7d00000-0xf7d03fff irq 16 at device 0.0 on pci1
Apr 20 00:13:54 <kern.crit> disp kernel: [427906] nvme0: failed to alloc HMB
Apr 20 00:13:54 <kern.crit> disp syslogd: last message repeated 3 times
Apr 20 00:13:54 <kern.crit> disp kernel: [427906] nvme0: Allocated 40MB host memory buffer
Apr 20 00:13:54 <kern.crit> disp kernel: [427906] nda0 at nvme0 bus 0 scbus3 target 0 lun 1
Apr 20 00:13:54 <kern.crit> disp kernel: [427906] nda0: <KBG50ZNV512G KIOXIA HP01AN00 234PH47KQMLK>

Seems to work:

# dd if=/dev/nda0 bs=4096
^C
1613+0 records in
1613+0 records out
6606848 bytes transferred in 0.829604 secs (7963861 bytes/sec)

So, what else do we need?