Hello all,
Apologies for the long post. But I wanted to provide enough detail on the problem I'm facing as well as everything I've tried so far to resolve it.
The Problem
I used to have a desktop built around a Gigabyte Z170X-SOC FORCE motherboard and a Core i7-6700K processor. Back when I built it, FreeBSD did not yet support Skylake graphics. So, I used an nVidia Quadro K1200 graphics card connected to three monitors. I started off with FreeBSD 11.0 and upgraded it over the years to 11.1, 11.2, and, finally, 12.0.
Things were working fine until a few weeks ago when the Gigabyte motherboard died. So I built a new desktop centered around a Supermicro X11DAC motherboard and two Xeon Gold 5222 processors. I kept the graphics card and disks from the old desktop.
FreeBSD boots up on the new desktop but the Xorg server doesn't detect the nVidia card.
System Info
Hardware
As mentioned above, the desktop has a Supermicro X11DAC motherboard, which features an on-board VGA controller provided by its ASPEED AST2500 BMC.
The UEFI has a setting, viz., Advanced -> PCIe/PCI/PnP Configuration -> VGA Priority, to select the primary graphics device for system boot. I have set this to "Offboard." AFAICT there is no setting to completely disable the on-board VGA.
FreeBSD
Here is the output of
The kernel definitely detects both graphics cards as can be seen from these
If it helps, the full
Here are the kernel modules I have loaded:
PCI Devices
Here are the relevant lines from
If it helps, the full output of
I also ran
Xorg
On my old desktop, I had configured X to use the proprietary nVidia driver in /usr/local/etc/X11/xorg.conf.d/device.conf:
Here are the contents of /var/log/Xorg.0.log when I run
Attempted Solutions
Sanity Check With VESA Driver
To confirm that the X11DAC isn't some sort of server-only motherboard that won't support graphics at all (illogical and unlikely but worth confirming nevertheless), I replaced the
However, it was low resolution and one monitor only (connected, of course, to the on-board VGA port). This makes sense as the on-board VGA is meant to be used for remote management via IPMI. Obviously, that is not a viable solution for everyday use as a development workstation.
Updating Xorg Config
As a first stab at the problem, I added
Next, I added a
Thinking that, perhaps, with GPU detection disabled, I needed to provide an explicit
Finally, I ran
Ubuntu 19.04 Installer
My next line of attack was to see if Linux would work. So, I downloaded the Ubuntu 19.04 installer ISO, wrote it to a USB stick, and booted up the computer with it. The Ubuntu installer did detect both the on-board VGA as well as the nVidia card. It configured and launched the Xorg server with the
All three monitors came alive. However, there was excessive and frequent flashing and flickering. Presumably, this can be solved with the proprietary nVidia driver.
The important point is that Ubuntu's Xorg server detected both graphics devices:
Upgrading x11-servers/xorg-server to Version 1.20.4
Since Ubuntu was running version 1.20.4 of the Xorg server while FreeBSD 12 is on 1.18.4, I thought upgrading it might help. For that, I came across PR 196678; its fourth attachment is a patch set for building version 1.20.4 of the x11-servers/xorg-server port.
I applied the above patch set to my local ports tree and then ran
I also copied nvidia_drv.so and libglx.so from their usual locations under /usr/local to the corresponding locations where I'd unpacked the package. And I reverted device.conf to its original state (i.e., without my
All ready, I ran this newly built version 1.20.4 of Xorg and, just like 1.18.4, it too failed to detect the nVidia card. Here is the log file.
Project Trident Installer
As one last-ditch effort, since it is a desktop-oriented FreeBSD derivative, thinking perhaps they'd patched the Xorg server to get it to work right, I downloaded the latest STABLE release of the Project Trident installer ISO and tried it out (the latest CURRENT release doesn't boot on my machine).
But their GUI installer simply uses the VESA driver. So, I escaped to a shell and tried
Next Steps
It looks to me like the Xorg server on FreeBSD doesn't probe the PCI bus properly. I had a quick look at its code, but, naturally, couldn't really tell much. Is anyone on the forum familiar with this code? I'd be willing to patch and test.
Or maybe there's no need to futz around with the X server code. Maybe I'm simply missing some kernel or other config needed to get things working. Has anyone else faced a similar problem (especially someone using a Supermicro motherboard as a workstation rather than a headless, remotely managed server)?
Well, I'm out of ideas. Any further pointers on how to proceed would be deeply appreciated. TIA.
Apologies for the long post. But I wanted to provide enough detail on the problem I'm facing as well as everything I've tried so far to resolve it.
The Problem
I used to have a desktop built around a Gigabyte Z170X-SOC FORCE motherboard and a Core i7-6700K processor. Back when I built it, FreeBSD did not yet support Skylake graphics. So, I used an nVidia Quadro K1200 graphics card connected to three monitors. I started off with FreeBSD 11.0 and upgraded it over the years to 11.1, 11.2, and, finally, 12.0.
Things were working fine until a few weeks ago when the Gigabyte motherboard died. So I built a new desktop centered around a Supermicro X11DAC motherboard and two Xeon Gold 5222 processors. I kept the graphics card and disks from the old desktop.
FreeBSD boots up on the new desktop but the Xorg server doesn't detect the nVidia card.
System Info
Hardware
As mentioned above, the desktop has a Supermicro X11DAC motherboard, which features an on-board VGA controller provided by its ASPEED AST2500 BMC.
The UEFI has a setting, viz., Advanced -> PCIe/PCI/PnP Configuration -> VGA Priority, to select the primary graphics device for system boot. I have set this to "Offboard." AFAICT there is no setting to completely disable the on-board VGA.
FreeBSD
Here is the output of
freebsd-version -kru:
Code:
12.0-RELEASE-p7
12.0-RELEASE-p7
12.0-RELEASE-p7The kernel definitely detects both graphics cards as can be seen from these
dmesg lines:
Code:
vgapci0: <VGA-compatible display> port 0x2000-0x207f mem 0x9c000000-0x9cffffff,0x9d000000-0x9d01ffff irq 17 at device 0.0 numa-domain 0 on pci3
vgapci1: <VGA-compatible display> port 0xf000-0xf07f mem 0xfa000000-0xfaffffff,0x39ffe0000000-0x39ffefffffff,0x39fff0000000-0x39fff1ffffff irq 96 at device 0.0 numa-domain 1 on pci19
vgapci1: Boot video device
nvidia1: <Quadro K1200> numa-domain 1 on vgapci1
vgapci1: child nvidia1 requested pci_enable_io
vgapci1: child nvidia1 requested pci_enable_io
nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 390.87 Tue Aug 21 15:53:31 PDT 2018If it helps, the full
dmesg log is available here.Here are the kernel modules I have loaded:
Code:
Id Refs Address Size Name
1 64 0xffffffff80200000 243d260 kernel
2 1 0xffffffff8263f000 3a9a10 zfs.ko
3 2 0xffffffff829e9000 a4f0 opensolaris.ko
4 1 0xffffffff829f7000 c7e0 aesni.ko
5 1 0xffffffff82a04000 1ead0 geom_eli.ko
6 1 0xffffffff83112000 1a20 fdescfs.ko
7 1 0xffffffff83114000 f4ef0 nvidia-modeset.ko
8 1 0xffffffff83209000 c42ae8 nvidia.ko
9 2 0xffffffff83e4c000 39970 linux.ko
10 3 0xffffffff83e86000 2e28 linux_common.ko
11 3 0xffffffff83e89000 53140 vboxdrv.ko
12 1 0xffffffff83edd000 7078 ioat.ko
13 1 0xffffffff83ee5000 1800 uhid.ko
14 1 0xffffffff83ee7000 23a8 ums.ko
15 2 0xffffffff83eea000 2ce0 vboxnetflt.ko
16 2 0xffffffff83eed000 a020 netgraph.ko
17 1 0xffffffff83ef8000 1710 ng_ether.ko
18 1 0xffffffff83efa000 3f30 vboxnetadp.ko
19 1 0xffffffff83efe000 acf mac_ntpd.ko
20 1 0xffffffff83eff000 16a78 ext2fs.ko
21 1 0xffffffff83f16000 7bd0 ipmi.ko
22 1 0xffffffff83f1e000 b10 smbus.ko
23 1 0xffffffff83f1f000 b98 coretemp.koPCI Devices
Here are the relevant lines from
pciconf -lvbe:
Code:
vgapci0@pci0:3:0:0: class=0x030000 card=0x20001a03 chip=0x20001a03 rev=0x41 hdr=0x00
vendor = 'ASPEED Technology, Inc.'
device = 'ASPEED Graphics Family'
class = display
subclass = VGA
bar [10] = type Memory, range 32, base 0x9c000000, size 16777216, enabled
bar [14] = type Memory, range 32, base 0x9d000000, size 131072, enabled
bar [18] = type I/O Port, range 32, base 0x2000, size 128, enabled
vgapci1@pci0:216:0:0: class=0x030000 card=0x114010de chip=0x13bc10de rev=0xa2 hdr=0x00
vendor = 'NVIDIA Corporation'
device = 'GM107GL [Quadro K1200]'
class = display
subclass = VGA
bar [10] = type Memory, range 32, base 0xfa000000, size 16777216, enabled
bar [14] = type Prefetchable Memory, range 64, base 0x39ffe0000000, size 268435456, enabled
bar [1c] = type Prefetchable Memory, range 64, base 0x39fff0000000, size 33554432, enabled
bar [24] = type I/O Port, range 32, base 0xf000, size 128, enabled
PCI-e errors = Correctable Error Detected
Unsupported Request Detected
Corrected = Advisory Non-Fatal ErrorIf it helps, the full output of
pciconf -lvbe is available here.I also ran
nvidia-smi -a. It reports the card's bus ID as 00000000:D8:00.0. Incidentally, 0xD8 is 216 in decimal. The full output of nvidia-smi -a is here.Xorg
On my old desktop, I had configured X to use the proprietary nVidia driver in /usr/local/etc/X11/xorg.conf.d/device.conf:
Code:
Section "Device"
Identifier "K1200"
Driver "nvidia"
EndSectionHere are the contents of /var/log/Xorg.0.log when I run
startx with the above config on the new desktop. The reason Xorg fails is because it only detects one graphics device, i.e., the on-board VGA:
Code:
[ 1806.956] (--) PCI:*(0:3:0:0) 1a03:2000:1a03:2000 rev 65, Mem @ 0x9c000000/16777216, 0x9d000000/131072, I/O @ 0x00002000/128, BIOS @ 0x????????/65536Attempted Solutions
Sanity Check With VESA Driver
To confirm that the X11DAC isn't some sort of server-only motherboard that won't support graphics at all (illogical and unlikely but worth confirming nevertheless), I replaced the
nvidia in device.conf with vesa. When I ran startx after this change, I did indeed get my regular GUI desktop.However, it was low resolution and one monitor only (connected, of course, to the on-board VGA port). This makes sense as the on-board VGA is meant to be used for remote management via IPMI. Obviously, that is not a viable solution for everyday use as a development workstation.
Updating Xorg Config
As a first stab at the problem, I added
BusID "PCI:216:0:0" to device.conf. No effect; Xorg still only detected the on-board VGA and complained about not finding any devices. The Xorg log is here.Next, I added a
ServerFlags section and explicitly disabled AutoAddGPU in it to see if the server would pick the card with the above PCI bus ID. No luck.Thinking that, perhaps, with GPU detection disabled, I needed to provide an explicit
ServerLayout and other sections, I proceeded to create a complete Xorg config with everything spelled out about devices, monitors, etc. All to no avail.Finally, I ran
nvidia-xconfig, hoping that it would do some magic hardware probing and generate a usable config. Sadly, it is devoid of magic.Ubuntu 19.04 Installer
My next line of attack was to see if Linux would work. So, I downloaded the Ubuntu 19.04 installer ISO, wrote it to a USB stick, and booted up the computer with it. The Ubuntu installer did detect both the on-board VGA as well as the nVidia card. It configured and launched the Xorg server with the
nouveau driver. Here is the Xorg log file.All three monitors came alive. However, there was excessive and frequent flashing and flickering. Presumably, this can be solved with the proprietary nVidia driver.
The important point is that Ubuntu's Xorg server detected both graphics devices:
Code:
[ 28.976] (--) PCI: (3@0:0:0) 1a03:2000:1a03:2000 rev 65, Mem @ 0x9c000000/16777216, 0x9d000000/131072, I/O @ 0x00002000/128
[ 28.976] (--) PCI:*(216@0:0:0) 10de:13bc:10de:1140 rev 162, Mem @ 0xfa000000/16777216, 0x39ffe0000000/268435456, 0x39fff0000000/33554432, I/O @ 0x0000f000/128, BIOS @ 0x????????/131072Upgrading x11-servers/xorg-server to Version 1.20.4
Since Ubuntu was running version 1.20.4 of the Xorg server while FreeBSD 12 is on 1.18.4, I thought upgrading it might help. For that, I came across PR 196678; its fourth attachment is a patch set for building version 1.20.4 of the x11-servers/xorg-server port.
I applied the above patch set to my local ports tree and then ran
make package in /usr/ports/x11-servers/xorg-server. Instead of installing this newly built package, I simply unpacked its tarball in my home directory. I then made the executable SUID root:
Code:
sudo chown root:wheel /path/to/bin/Xorg
sudo chmod 4755 /path/to/bin/XorgI also copied nvidia_drv.so and libglx.so from their usual locations under /usr/local to the corresponding locations where I'd unpacked the package. And I reverted device.conf to its original state (i.e., without my
ServerFlags, ServerLayout, etc. experiments) but with the BusID still in place.All ready, I ran this newly built version 1.20.4 of Xorg and, just like 1.18.4, it too failed to detect the nVidia card. Here is the log file.
Project Trident Installer
As one last-ditch effort, since it is a desktop-oriented FreeBSD derivative, thinking perhaps they'd patched the Xorg server to get it to work right, I downloaded the latest STABLE release of the Project Trident installer ISO and tried it out (the latest CURRENT release doesn't boot on my machine).
But their GUI installer simply uses the VESA driver. So, I escaped to a shell and tried
startx from there. But, as before, /var/log/Xorg.0.log showed detection of only the on-board VGA.Next Steps
It looks to me like the Xorg server on FreeBSD doesn't probe the PCI bus properly. I had a quick look at its code, but, naturally, couldn't really tell much. Is anyone on the forum familiar with this code? I'd be willing to patch and test.
Or maybe there's no need to futz around with the X server code. Maybe I'm simply missing some kernel or other config needed to get things working. Has anyone else faced a similar problem (especially someone using a Supermicro motherboard as a workstation rather than a headless, remotely managed server)?
Well, I'm out of ideas. Any further pointers on how to proceed would be deeply appreciated. TIA.
