bhyve FreeBSD 8.4 bhyve guest using ~25% CPU when idle

[spork]

In the middle of retiring some old stuff and need a holding ground, and found some issues with jailing something this old (an 8.4 jail is certainly possible, but I've hit some software that just does not behave in a jail)...

Logging into the 8.4 guest, top shows 0% CPU usage, and other than sshd, no services are running.

But on the 13.3 host, I have this:

last pid: 50502;  load averages:  1.35,  0.72,  0.45                                                                  up 0+23:24:41  20:16:27
365 processes: 1 running, 364 sleeping
CPU:  0.6% user,  0.0% nice,  1.2% system,  0.0% interrupt, 98.2% idle
Mem: 58M Active, 1360M Inact, 5777M Laundry, 51G Wired, 3496M Free
ARC: 48G Total, 5301M MFU, 41G MRU, 1881K Anon, 190M Header, 702M Other
     45G Compressed, 52G Uncompressed, 1.15:1 Ratio
Swap: 16G Total, 2825M Used, 13G Free, 17% Inuse


  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
84101 root         31  20    0  8254M   226M kqread  32 260:08  24.38% bhyve

I'm using bmd (https://github.com/yuichiro-naito/bmd) to "manage" the VMs. I get the same results whether I use a zvol or an image file, and with 1 CPU or 2, or 4. The loader is "bhveload". No VNC console, just serial. The state of bhyve generally seems to be "kqread"...

This is my first time doing much of anything with bhyve, but I did search a bit for any best practices on running older versions of FreeBSD and didn't come up with anything.

Can anyone point me in a direction here?

 

[cracauer@]

Lower kernel hz to 100.

 

[VladiBG]

Is HPET enabled?

 

[sko]

bhyve(8)

     -H          Yield the virtual CPU thread when a HLT instruction is
                 detected.  If this option is not specified, virtual CPUs will
                 use 100% of a host CPU.

 

[spork]

Lower kernel hz to 100.

Thanks! This seems to work, we're at around 4% when the VM is idle. I remember this being something vmware recommended some time back for FreeBSD guests as well many, many years ago.

Is HPET enabled?

Nope, just whatever the default is, which I assume is something I shouldn't be mucking around with?

[root@clweb6 /usr/local/etc/netdata]# sysctl -a kern.timecounter
kern.timecounter.tsc_shift: 1
kern.timecounter.smp_tsc_adjust: 0
kern.timecounter.smp_tsc: 1
kern.timecounter.invariant_tsc: 1
kern.timecounter.fast_gettime: 1
kern.timecounter.tick: 1
kern.timecounter.choice: TSC-low(1000) ACPI-fast(900) HPET(950) i8254(0) dummy(-1000000)
kern.timecounter.hardware: TSC-low
kern.timecounter.alloweddeviation: 5
kern.timecounter.timehands_count: 2
kern.timecounter.stepwarnings: 0
kern.timecounter.tc.TSC-low.quality: 1000
kern.timecounter.tc.TSC-low.frequency: 1099999927
kern.timecounter.tc.TSC-low.counter: 547252701
kern.timecounter.tc.TSC-low.mask: 4294967295
kern.timecounter.tc.ACPI-fast.quality: 900
kern.timecounter.tc.ACPI-fast.frequency: 3579545
kern.timecounter.tc.ACPI-fast.counter: 3094802
kern.timecounter.tc.ACPI-fast.mask: 16777215
kern.timecounter.tc.HPET.quality: 950
kern.timecounter.tc.HPET.frequency: 24000000
kern.timecounter.tc.HPET.counter: 2545958659
kern.timecounter.tc.HPET.mask: 4294967295
kern.timecounter.tc.i8254.quality: 0
kern.timecounter.tc.i8254.frequency: 1193182
kern.timecounter.tc.i8254.counter: 57344
kern.timecounter.tc.i8254.mask: 65535
[root@clweb6 /usr/local/etc/netdata]#

It appears "TSC-low" is preferred, with HPET being second and ACPI-fast being third.

bhyve(8)

     -H          Yield the virtual CPU thread when a HLT instruction is
                 detected.  If this option is not specified, virtual CPUs will
                 use 100% of a host CPU.

I wasn't seeing 100% CPU utilization, and while I'm using bmd, it does look like it calls bhyve with "-H" by default (after digging in the source a bit):

fprintf(fp, "/usr/sbin/bhyve\n-A\n-H\n-w\n");