vmware Hot-plugging vmx3 interface won't work

P

Pouaite

Hello,

I have a VMWare virtualization environment and a guest FreeBSD version that are both deprecated (ESXi 6.5 and FreeBSD 11, unfortunately -CURRENT which doesn't help). I am in the process to move to Proxmox, and upgrade to FreeBSD 14 -RELEASE), but on one site I have an issue which cannot wait for upgrade, and where I need to avoid rebooting.

When I try to add a virtual Interface (VMX3 to stay consistent with the other interfaces), it seems to be added correctely and then it disappears within seconds, as I can see in dmesg :
Code: 
Jan 26 22:15:50 server02 kernel: pcib13: Attention Button Pressed: Detaching in 5 seconds
Jan 26 22:15:50 server02 kernel: pci12: <ACPI PCI bus> on pcib13
Jan 26 22:15:50 server02 kernel: vmx9: <VMware VMXNET3 Ethernet Adapter> at device 0.0 on pci12
Jan 26 22:15:50 server02 kernel: vmx9: Ethernet address: 00:50:56:86:f0:5e
Jan 26 22:15:50 server02 kernel: vmx9: link state changed to UP
Jan 26 22:15:55 server02 kernel: vmx9: detached
Jan 26 22:15:55 server02 kernel: pci12: detached
All solutions found on the Internet point to : shutdown, addition of the virtual interface, and then restart.

Although I am aware that we should upgrade everything here, I still believe I am missing a way to make this work, since the hotplug works at first. By any luck does anyone has a clue on how to disable the mechanism detaching the Interface straight away after hot plugging it?

Thank you in advance.
 
I'd look around into VM's log file on esxi to get more info about this: <vmname>/vmware.log. I can't think of anything at this time why that would be so.
 
Pouaite said:
Jan 26 22:15:50 server02 kernel: pcib13: Attention Button Pressed: Detaching in 5 seconds
Click to expand...
I suppose that this could be a bug in FreeBSD 11 PCIe hot-plug code.
Or an incompatibility between what FreeBSD expects and what VMWare does.
 
As a reference point, worked fine with FreeBSD 13.3 guest on ESXi 7.0
Code: 
pci5: <ACPI PCI bus> on pcib19
vmx1: <VMware VMXNET3 Ethernet Adapter> at device 0.0 on pci5
vmx1: Using 2048 TX descriptors and 2048 RX descriptors
vmx1: Using 4 RX queues 4 TX queues
vmx1: Using MSI-X interrupts with 5 vectors
vmx1: Ethernet address: 00:50:56:81:57:e1
 
Also, I can "hot-unplug" the interface and then I see:
Code: 
pcib19: Attention Button Pressed: Detaching in 5 seconds
vmx1: detached
pci5: detached
 
I dug up old 6.5 esxi (6.5.0 Update 2 (Build 8294253)), created two vSwitches, each with one portgroup that can be assigned to VM. Installed 11.0-RELEASE amd64 VM.

I stressed the VM a bit with adding and attempting to remove it from the configuration. While the e1000 NIC was added to the config system never saw it. And then it was not possible to remove it. Reboot to cleanup was needed.
Adding vmxnet3 does exactly what you're experiencing. VM log says just "removing ethernet". And while FreeBSD says it's removed VMware still sees the adapter up. So there's a bit of mess there then. Luckily I can remove that NIC from its config. To complete the test I did add the adapter offline - it worked as expected.

While I was at it I updated esxi to 6.7.0 Update 2 (Build 13006603) and did these tests again - no change. So as a last step I did upgrade FreeBSD to 11.4 - same issue.
So yeah, as Andriy mentioned this could be PCIe hotplug bug in FreeBSD 11.
 
Ok, well, thank you a lot for the debug and feedbacks everyone! I will just move on (reboot for now), and just upgrade when I can to let this eventual bug in the past =)
 
Pouaite said:
Ok, well, thank you a lot for the debug and feedbacks everyone! I
Click to expand...
Yw. As it takes less than 2mins to boot and upgrade the whole VM I did test this on 12.0 and 12.4 too. On 12.0 I had the same issues, 12.4 was working as expected. It was fixed somewhere around that time then.
 
You must log in or register to reply here.
Back
Top