Setting up a bhyve vm for kernel development and debugging

C

cmoerz

I recently set up a bhyve vm on my workstation for kernel development and figured others might be able to make use of my notes as well, because it's really quite easy.

VM setup​

  1. Download a FreeBSD disk image, i.e. 13.2-RELEASE if you're running 13.2. This version should match whatever kernel file you have compiled on your host system for debugging purposes.
  2. On the host system, compile world and kernel as you usually would and pkg install kgdb.
  3. Extract that image using unxz FreeBSD-13.2-RELEASE-amd64.raw.xz.
  4. Resize the disk image to 40GB: truncate -s 40G FreeBSD-13.2-RELEASE-amd64.raw
  5. Create a shell script that will start the new VM: touch vm.sh && chmod 755 vm.sh && ee vm.sh
  6. Put relevant VM startup code into this script, for example (you can also use tools like vm-bhyve or vmrun.sh to do this):
Code: 
#!/bin/sh

#
# Kernel development setup
#

# Enable ip forwarding
sysctl net.inet.ip.forwarding=1

DISKIMG=FreeBSD-13.2-RELEASE-amd64.raw
MEMORY=4G

# load bhyve image
bhyveload \
        -m ${MEMORY} \
        -d ${DISKIMG} \
        -c /dev/nmdm1B \
        kerndev

bhyve \
        -A \
        -H \
        -P \
        -c 4 \
        -D \
        -m ${MEMORY} \
        -l com1,/dev/nmdm1B \
        -l com2,/dev/nmdm2B \
        -s 0,hostbridge \
        -s 1,virtio-blk,${DISKIMG} \
        -s 2,virtio-net,tap0 \
        -s 31,lpc \
        kerndev &
PID=$!
sleep 1

ifconfig tap0 inet 192.168.0.1 netmask 255.255.255.252

wait ${PID}

# when we're done, we're destroying the vm
bhyvectl --vm=kerndev --destroy

Networking​

  1. Replace the IP address in the script with ones that work for your local network
  2. Either make sure to attach that tap0 interface to a bridge or implement nat, i.e. via /etc/pf.conf.

Setting up kernel​

  1. Once inside the VM, modify /usr/src/amd64/conf/GENERIC (or whatever matches your architecture) to include options GDB.
  2. Use the opportuntity to clean up /etc/rc.conf from any growfs or other statements that you don't need, disable sendmail, set up IP address and routing, etc. and do adjustments as you see fit.
  3. Then cd to /usr/src and enter the following command, replacing <NCPU> with the number of CPUs in your system.
Code: 
make clean && make -j<NCPU> buildworld && make -j<NCPU> kernel && make -j<NCPU> installworld
(thanks for the feedback+fix grahamperrin! originally I had reversed the order of kernel and world)

Setting up serial console​

  1. Fix /boot/device.hints for uart0 to be flags 0x80
  2. add debug.kdb.current=gdb in /etc/sysctl.conf
  3. change /etc/ttys to include ttyu1 "/usr/libexec/getty 3wire" vt100 onifexists secure (mark the "onifexists"!)
  4. reboot

Entering debug mode​

Connect to the host on /dev/nmdm2A (i.e. via cu -l /dev/nmdm2A), because /dev/nmdm1A will be occupied by the debug session now.

  1. On guest sysctl debug.kdb.enter=1
  2. On host
    1. cd /usr/obj/usr/src/amd64.amd64/sys/GENERIC/
    2. kgdb kernel
    3. target remote /dev/nmdm1A (or whatever console port of bhyve is used)

One side note: I went into this and thought I might be able to set arbitrary breakpoints in kernel methods with this - that did not work for me. I can react and debug dumps this way, but halting the system based on debugger conditions would not work this way. If anyone has any insight or ideas on this (aside from putting dumps into the code), I'd love to get feedback and comments.

Further Resources​

 
Last edited:
make clean && make -j<NCPU> buildworld && make -j<NCPU> buildkernel && make installworld && make installkernel
Click to expand...

Please, are you certain of the order?

It's normal to install kernel before installing world.

Whilst I'm not familiar with bhyve, an alternative command might be:

make clean && make -j<NCPU> buildworld && make -j<NCPU> kernel && make -j<NCPU> installworld

( make kernel is equivalent to make buildkernel installkernel.)
 
There is also "-G <port>" argument to bhyve. And you can attach to it from within gdb with "target remote :<port>" and you can stop/start the guest but either bhyve is missing something or I don't know the magic incantation to make the kernel talk to it so that breakpoints, single stepping etc. work. [I used to debug kernel code this way using qemu way back 2005].

cmoerz said:
-s 0,hostbridge \ -s 1,virtio-blk,${DISKIMG} \ -s 2,virtio-net,tap0 \ -s 31,lpc \ kerndev & PID=$! sleep 1 ifconfig tap0 inet 192.168.0.1 netmask 255.255.255.252 wait ${PID}
Click to expand...

  • I use nvme instead of virtio-blk as it is faster.
  • I use DHCP from within the VM and no ifconfig tapN from the host.
  • I don't destroy the vm unless some bhyve option has changed and the VM refuses to start
  • Option "-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd" will allow you to avoid a separate bhyveload, provided you installed the guest FreeBSD right.
 
Good stuff!

One thing you might find more convenient for kernel development: you can construct the image's file system from outside the VM. So you can edit the source code on the host, build it on the host and then DESTDIR=/path/to/memdisk make installkernel && DESTDIR=/path/to/memdisk make installworld (I think it's DESTDIR but you will want to make -n to confirm).

Here's an example of constructing the image file system from the host, and booting UEFI. In your case, you would remove the tar -C commands because you would you make to get the files onto the image instead.

Note that EUFI booting requires sysutils/bhyve-firmware to load UEFI firmware.
 
You must log in or register to reply here.
Back
Top