bhyve Bhyve vm folders empty on reboot, but `init 1` fixes it

174bpm

174bpm

Hey all,

Somewhat new to freeBSD here. Apologies in advance if this question is already answered somewhere (I have looked) or if this isn't the right venue. If that is the case, kindly let me know!

Anyhow, so I have a FreeBSD 13.1 server hosting a bunch of jails and some Bhyve VMs. I created and manage the VMs with CBSD. I've recently started to encounter an issue where, upon booting the host, the folders (well, technically, zfs volume mountpoints) containing VM data are empty, save for a lone `traffic/` folder. If I query them via `cbsd bls`, CBSD complains about not being able to open the local.sqlite database file for each VM (because it's gone).

Luckily, the data is not lost, and soft-restarting via `init 1` (and then bypassing the root login prompt) restores these folders, and the VMs work fine.

The host doesn't get rebooted often, in fact I was only doing so because I was messing with some stuff in /etc/gettytab (wanted something pretty to look at on tty0 and was hooking it up to display htop --readonly) and wasn't sure how to see my results without restarting everything. But then I had some weird power fluctuations today that triggered a reboot and upon booting up all the VMs were "gone".

When I am in this state, zfs list reports the volumes as mounted at their correct mountpoints. (The zvol "vhd file" for each VM is also there, with no mountpoint)

I'm not really sure what can cause a folder to appear to be empty short of unmounting the mountpoint; I can't say this is a failure mode I've seen in any of the filesystems I've worked with a bunch.

Poking around /var/log/messages, I see a bunch of what appear to be normal-looking bootup messages for bhyve (again, not very familiar with what is normal here), and nothing that appears like a VM failed to start. Yet `cbsd bstart` shows errors.

Can someone clue me in to what is happening here? Or some hints on how to go about fixing it?

Thanks!
 
Code: 
zfs mount -a
doesn't help? only `init 1` ? how healthy is your pool and has enough space ? Please check:

Code: 
zfs list -t snapshot |wc -l
zpool list      # ( check SIZE/ALLOC/FREE )
zpool status
 
So I can't reboot the machine again to test for that right now, but I will say that I have a call to `zfs status` in my .profile as a login banner, and it has never reported anything other than a healthy state ("no known data errors" and ONLINE).

Free space is sitting around 10TB, again nothing out of the ordinary on the relevant listings

`zfs list` / `zfs list -t snapshot` didn't show anything out of the ordinary when I checked it, but I will check it again.

Will reply again when I can get the server back into the bad state
 
So `zfs mount -a` brings the folders back (much better than init 1, thanks!).

`zfs list` shows the mountpoints correctly before I do this, though, which lead me to think they were already mounted. Glad to confirm the issue is they aren't mounting for some reason, and not some other failure.

CBSD's logs don't indicate anything, literally everything I see here is chatter about networking interfaces. I can see it attempting to start the three VMs I have set up, with no indication that anything is unrecoverably wrong -- what is especially weird is that one of those VMs is straight broken; I tried to have it clone its hard disk from a different VM and it never seems to have actually done that, so its disk is missing. It fails to start regardless! So I suspect CBSD isn't logging whatever is going on here...

Switching CBSD's logging to VERBOSE doesn't seem to help, log messages are pretty much identical, just a bunch of chatter from the networking system... like the most "serious" looking thing it shows is a couple instances "BGRADD bge0: Device busy"

Before this, I observed some weirdness in one of my VM folders when something went wrong in bconstruct-tui and I panicked out of it hitting ESC repeatedly... I forget what exactly but it seemed to tick through a workflow that was most definitely not backing out and quitting. Then noticed one of the VM folders was gone (it came back on a reboot). So, it seems like CBSD likes to mount+unmount zfs volumes as part of what it does. (Why would it do this??)

I'm hoping someone else has encountered something like this; this is a relatively young FreeBSD install (<1 year old) and all it runs are ezjail, cbsd, bhyve, samba, and an idle instance of nginx. Everything else it does is virtualized in some way. So it seems kind of weird to be running into what looks like some kind of edge case.
 
You must log in or register to reply here.
Back
Top