USB3 PCIe Cards with FreeBSD low speed

sol-invictus · Oct 14, 2024

Hello,

we wanted to connect a USB3 PCI Express card to our aging HP Proliant DL380 Gen8 (which has PCIe 3.0 slots with x16, x8 and a PCI-e 2.0 slot x4).

The USB3 cards are working but at a very low speed. When I test the speed by connecting a USB3 HDD the speed is about 5 to 7 MB/s with FreeBSD 13.3, 13.4, 14.0 and 14.1. With Debian 12 on the same Hardware I get rates from 110 to 130MB/s.

Hardware Specs HP Proliant DL380 Gen8: https://support.hpe.com/hpesc/publi...UID-E03B5B98-4537-4F81-8EE0-7F5A6517ADBD.html

There are different USB 3.x PCI-Express Cards in use.

Here is the output of

Code:

 [user]@[server][~]# pciconf -lbccv

ehci0@pci0:0:26:0:    class=0x0c0320 rev=0x05 hdr=0x00 vendor=0x8086 device=0x1d2d subvendor=0x103c subdevice=0x18a9
    vendor     = 'Intel Corporation'
    device     = 'C600/X79 series chipset USB2 Enhanced Host Controller'
    class      = serial bus
    subclass   = USB
    bar   [10] = type Memory, range 32, base 0xf6c60000, size 1024, enabled
    cap 01[50] = powerspec 2  supports D0 D3  current D0
    cap 0a[58] = EHCI Debug Port at offset 0xa0 in map 0x14
    cap 13[98] = PCI Advanced Features: FLR TP

ehci1@pci0:0:29:0:    class=0x0c0320 rev=0x05 hdr=0x00 vendor=0x8086 device=0x1d26 subvendor=0x103c subdevice=0x18a9
    vendor     = 'Intel Corporation'
    device     = 'C600/X79 series chipset USB2 Enhanced Host Controller'
    class      = serial bus
    subclass   = USB
    bar   [10] = type Memory, range 32, base 0xf6c50000, size 1024, enabled
    cap 01[50] = powerspec 2  supports D0 D3  current D0
    cap 0a[58] = EHCI Debug Port at offset 0xa0 in map 0x14
    cap 13[98] = PCI Advanced Features: FLR TP

xhci0@pci0:7:0:0:    class=0x0c0330 rev=0x00 hdr=0x00 vendor=0x1b21 device=0x1242 subvendor=0x1b21 subdevice=0x1242
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM1142 USB 3.1 Host Controller'
    class      = serial bus
    subclass   = USB
    bar   [10] = type Memory, range 64, base 0xf7ef0000, size 32768, enabled
    cap 05[50] = MSI supports 8 messages, 64 bit
    cap 11[68] = MSI-X supports 8 messages, enabled
                 Table in map 0x10[0x2000], PBA in map 0x10[0x2080]
    cap 01[78] = powerspec 3  supports D0 D3  current D0
    cap 10[80] = PCI-Express 2 endpoint max data 256(512) RO NS
                 max read 4096
                 link x2(x2) speed 5.0(5.0) ASPM disabled(L1)
    ecap 0002[100] = VC 1 max VC0
    ecap 0001[200] = AER 1 0 fatal 0 non-fatal 0 corrected
    ecap 0019[280] = PCIe Sec 1 lane errors 0
    ecap 0018[300] = LTR 1

xhci1@pci0:4:0:0:    class=0x0c0330 rev=0x01 hdr=0x00 vendor=0x1106 device=0x3483 subvendor=0x1106 subdevice=0x3483
    vendor     = 'VIA Technologies, Inc.'
    device     = 'VL805/806 xHCI USB 3.0 Controller'
    class      = serial bus
    subclass   = USB
    bar   [10] = type Memory, range 64, base 0xf7df0000, size 4096, enabled
    cap 01[80] = powerspec 3  supports D0 D3  current D0
    cap 05[90] = MSI supports 4 messages, 64 bit enabled with 1 message
    cap 10[c4] = PCI-Express 2 endpoint max data 256(256) RO NS
                 max read 4096
                 link x1(x1) speed 5.0(5.0) ASPM disabled(L0s/L1) ClockPM disabled
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected

xhci2@pci0:10:0:0:    class=0x0c0330 rev=0x00 hdr=0x00 vendor=0x1b21 device=0x1242 subvendor=0x1b21 subdevice=0x1242
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM1142 USB 3.1 Host Controller'
    class      = serial bus
    subclass   = USB
    bar   [10] = type Memory, range 64, base 0xf7ff0000, size 32768, enabled
    cap 05[50] = MSI supports 8 messages, 64 bit
    cap 11[68] = MSI-X supports 8 messages, enabled
                 Table in map 0x10[0x2000], PBA in map 0x10[0x2080]
    cap 01[78] = powerspec 3  supports D0 D3  current D0
    cap 10[80] = PCI-Express 2 endpoint max data 128(512) RO NS
                 max read 4096
                 link x2(x2) speed 5.0(5.0) ASPM disabled(L1)
    ecap 0002[100] = VC 1 max VC0
    ecap 0001[200] = AER 1 0 fatal 0 non-fatal 0 corrected
    ecap 0019[280] = PCIe Sec 1 lane errors 0
    ecap 0018[300] = LTR 1

uhci0@pci0:1:0:4:    class=0x0c0300 rev=0x02 hdr=0x00 vendor=0x103c device=0x3300 subvendor=0x103c subdevice=0x3381
    vendor     = 'Hewlett-Packard Company'
    device     = 'Integrated Lights-Out Standard Virtual USB Controller'
    class      = serial bus
    subclass   = USB
    bar   [20] = type I/O Port, range 32, base 0x3c00, size 32, enabled
    cap 05[70] = MSI supports 1 message, 64 bit
    cap 10[80] = PCI-Express 1 legacy endpoint max data 128(128)
                 max read 128
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
    cap 01[f0] = powerspec 3  supports D0 D3  current D0

xhci3@pci0:42:0:0:    class=0x0c0330 rev=0x03 hdr=0x00 vendor=0x1912 device=0x0014 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Renesas Technology Corp.'
    device     = 'uPD720201 USB 3.0 Host Controller'
    class      = serial bus
    subclass   = USB
    bar   [10] = type Memory, range 64, base 0xfbff0000, size 8192, enabled
    cap 01[50] = powerspec 3  supports D0 D3  current D0
    cap 05[70] = MSI supports 8 messages, 64 bit
    cap 11[90] = MSI-X supports 8 messages, enabled
                 Table in map 0x10[0x1000], PBA in map 0x10[0x1080]
    cap 10[a0] = PCI-Express 2 endpoint max data 128(128) NS
                 max read 4096
                 link x1(x1) speed 5.0(5.0) ASPM disabled(L0s/L1) ClockPM disabled
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
    ecap 0018[150] = LTR 1

xhci4@pci0:38:0:0:    class=0x0c0330 rev=0x02 hdr=0x00 vendor=0x1912 device=0x0015 subvendor=0x1912 subdevice=0x0015
    vendor     = 'Renesas Technology Corp.'
    device     = 'uPD720202 USB 3.0 Host Controller'
    class      = serial bus
    subclass   = USB
    bar   [10] = type Memory, range 64, base 0xfbbf0000, size 8192, enabled
    cap 01[50] = powerspec 3  supports D0 D3  current D0
    cap 05[70] = MSI supports 8 messages, 64 bit
    cap 11[90] = MSI-X supports 8 messages, enabled
                 Table in map 0x10[0x1000], PBA in map 0x10[0x1080]
    cap 10[a0] = PCI-Express 2 endpoint max data 128(128) NS
                 max read 4096
                 link x1(x1) speed 5.0(5.0) ASPM disabled(L0s/L1) ClockPM disabled
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
    ecap 0003[140] = Serial 1 1300000092431408
    ecap 0018[150] = LTR 1

xhci5@pci0:39:0:0:    class=0x0c0330 rev=0x02 hdr=0x00 vendor=0x1912 device=0x0015 subvendor=0x1912 subdevice=0x0015
    vendor     = 'Renesas Technology Corp.'
    device     = 'uPD720202 USB 3.0 Host Controller'
    class      = serial bus
    subclass   = USB
    bar   [10] = type Memory, range 64, base 0xfbcf0000, size 8192, enabled
    cap 01[50] = powerspec 3  supports D0 D3  current D0
    cap 05[70] = MSI supports 8 messages, 64 bit
    cap 11[90] = MSI-X supports 8 messages, enabled
                 Table in map 0x10[0x1000], PBA in map 0x10[0x1080]
    cap 10[a0] = PCI-Express 2 endpoint max data 128(128) NS
                 max read 4096
                 link x1(x1) speed 5.0(5.0) ASPM disabled(L0s/L1) ClockPM disabled
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
    ecap 0003[140] = Serial 1 1300000092431408
    ecap 0018[150] = LTR 1

xhci6@pci0:40:0:0:    class=0x0c0330 rev=0x02 hdr=0x00 vendor=0x1912 device=0x0015 subvendor=0x1912 subdevice=0x0015
    vendor     = 'Renesas Technology Corp.'
    device     = 'uPD720202 USB 3.0 Host Controller'
    class      = serial bus
    subclass   = USB
    bar   [10] = type Memory, range 64, base 0xfbdf0000, size 8192, enabled
    cap 01[50] = powerspec 3  supports D0 D3  current D0
    cap 05[70] = MSI supports 8 messages, 64 bit
    cap 11[90] = MSI-X supports 8 messages, enabled
                 Table in map 0x10[0x1000], PBA in map 0x10[0x1080]
    cap 10[a0] = PCI-Express 2 endpoint max data 128(128) NS
                 max read 4096
                 link x1(x1) speed 5.0(5.0) ASPM disabled(L0s/L1) ClockPM disabled
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
    ecap 0003[140] = Serial 1 1300000092431408
    ecap 0018[150] = LTR 1

xhci7@pci0:41:0:0:    class=0x0c0330 rev=0x02 hdr=0x00 vendor=0x1912 device=0x0015 subvendor=0x1912 subdevice=0x0015
    vendor     = 'Renesas Technology Corp.'
    device     = 'uPD720202 USB 3.0 Host Controller'
    class      = serial bus
    subclass   = USB
    bar   [10] = type Memory, range 64, base 0xfbef0000, size 8192, enabled
    cap 01[50] = powerspec 3  supports D0 D3  current D0
    cap 05[70] = MSI supports 8 messages, 64 bit
    cap 11[90] = MSI-X supports 8 messages, enabled
                 Table in map 0x10[0x1000], PBA in map 0x10[0x1080]
    cap 10[a0] = PCI-Express 2 endpoint max data 128(128) NS
                 max read 4096
                 link x1(x1) speed 5.0(5.0) ASPM disabled(L0s/L1) ClockPM disabled
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
    ecap 0003[140] = Serial 1 1300000092431408
    ecap 0018[150] = LTR 1

xhci8@pci0:33:0:0:    class=0x0c0330 rev=0x00 hdr=0x00 vendor=0x1b21 device=0x2142 subvendor=0x1b21 subdevice=0x2142
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM2142/ASM3142 USB 3.1 Host Controller'
    class      = serial bus
    subclass   = USB
    bar   [10] = type Memory, range 64, base 0xfbaf0000, size 32768, enabled
    cap 05[50] = MSI supports 8 messages, 64 bit
    cap 11[68] = MSI-X supports 8 messages, enabled
                 Table in map 0x10[0x2000], PBA in map 0x10[0x2080]
    cap 01[78] = powerspec 3  supports D0 D3  current D0
    cap 10[80] = PCI-Express 2 legacy endpoint max data 256(512) RO NS
                 max read 4096
                 link x2(x2) speed 8.0(8.0) ASPM disabled(L0s/L1)
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
    ecap 0019[200] = PCIe Sec 1 lane errors 0
    ecap 0018[300] = LTR 1
    ecap 001e[400] = L1 PM Substates 1

So I don`t know if it is the right thing to test the speed but this is how I tried:

Code:

 [user]@[server][~]# mount -t ext2fs /dev/da2p1 /media/
[user]@[server][~]# dd if=/dev/urandom of=/media/test.img bs=4M count=1000 status=progress && sync
1000+0 records in
1000+0 records out
4194304000 bytes transferred in 613.674362 secs (6834739 bytes/sec)

camcontrol devlist:

Code:

<HP RAID 1(1+0) OK>                at scbus0 target 0 lun 0 (pass0,da0)
<SanDisk SanDisk 3.2 Gen1 DL17>    at scbus5 target 0 lun 0 (da1,pass1)
<WD Elements 25A3 1030>            at scbus6 target 0 lun 0 (da2,pass2)

So what I'm doing wrong here?

With Debian 12 the same hardware it is working fast and as it should, with FreeBSD (boot from USB Stick in Single Mode) I get the slow speed via USB3.

drr · Oct 14, 2024

Could it be the ext2fs file system?

USB I/O very slow

Hi, I downloaded an 2,7 GB ISO image and copied to a pen drive formatted as ext2. The copying progress started with a write speed of 3 MB/s and went slower and slower and ended with a write speed of 627 KB/s. The copy process took me 47 minutes and I decided to test the write speed of Debian...

forums.freebsd.org

covacat · Oct 14, 2024

just dd from and to raw disk without a fs
would be more telling

CeXP1917 · Oct 14, 2024

ext2 and NTFS in FreeBSD slower because of fusefs.

If tests with dd, UFS and ZFS show low speed, then target for investigation is hardware.

sol-invictus · Oct 15, 2024

WOW! I tried copying from /dev/urandom to a partition without a filesystem and it reached 160MB/s

sol-invictus · Oct 15, 2024

Now I'm using a different USB-HDD. I made a UFS partition and the speed even increased! So now I get 260MB/s

Could it be that there is such a big difference between ext4 and UFS? I wanted to have the possibility to work with the USB-HDD on FreeBSD and Linux - there UFS seems not to be supported that well regarding read and write:

Solved - r/w access to UFS from Linux

Apparently openSUSE is the only Linux distribution which provides r/w access to UFS. Can anyone confirm this?

forums.freebsd.org

drr · Oct 16, 2024

sol-invictus said:
I wanted to have the possibility to work with the USB-HDD on FreeBSD and Linux

I have used ntfs on drives shared between FreeBSD, linux and windows; not sure if it would suit your needs. ntfs also uses fusefs, but I have not noticed slow transfer speeds.

sol-invictus · Oct 16, 2024

Ok, thanks, I tried now with NTFS also. This was not that easy because I used a FreeBSD booted from USB Stick. Tried there to first get usb thetering to work (kldload if_urndis) but I failed with dhclient (read only ISO). The following worked on our TrueNAS:

Code:

kldload fusefs
ntfs-3g -o rw /dev/da2p10 /media

There I got a speed of 52MB/s with the NTFS filesystem - so this tells me that ext4 is still around 1/10th of the speed of NTFS.

drr · Oct 16, 2024

drr said:
I have used ntfs on drives shared between FreeBSD, linux and windows

I use zfs on my FreeBSD-only drives. I have managed to mount one of these drives on a linux distro previously; i think the linux module/package was 'zfsutils-linux'.