bhyve bhyve i386 OS on amd64 host?

R

rowan194

Can an amd64 host support an i386 OS? I know the CPU itself can support both (as well as, technically, 8086 real mode), but unsure whether it can freely switch between 32 and 64 bit, and if so, whether bhyve (or FreeBSD as the host OS) supports that?

I've been searching, but I cannot find a clear answer to this myself.

I tried to boot an old i386 image as a test, but it doesn't work:

Code: 
# bhyveload -m 256 -d IBM8.4-ad1.image-THROWAWAY vm0
Consoles: userboot

FreeBSD/amd64 User boot lua, Revision 1.2
Consoles: userboot

FreeBSD/amd64 User boot 4th, Revision 1.2
Segmentation fault (core dumped)
 
I don't know whether it works, I would expect it should work. But then, which OS? FreeBSD? Some very old version?

bhyveload(8) can only boot FreeBSD, and I'd expect problems with very old versions, maybe also with booting an i386 version (but then, you could try an i386 userboot.so with the -l switch).

Have you tried booting with UEFI and/or UEFI-CSM?
 
I am running bhyve on AMD64 and have i386 VM's.

Here is an example snipped from my startup script.

Code: 
bhyveload -m 4G -S -d /vm/freebsd-vm4.img -c /dev/nmdm4A freebsd-vm4 &
sleep 5
bhyve -S -c 10 -m 4G -A -H -P -s 0:0,hostbridge -s 4:0,ahci-hd,/vm/freebsd-vm4.img -s 5:0,passthru,133/0/3 -s 10:0,passthru,13/0/0 -s 31:0,lpc -l com1,/dev/nmdm4A freebsd-vm4 &
 
My other virt box:
Code: 
bhyveload -S -m 4G -c /dev/nmdm0A -d /vm/freebsd/freebsd-i386a.img freebsd-i386a &
sleep 15
bhyve -S -c 8 -m 4G -AHP -s 0:0,hostbridge -s 1:0,lpc -s 5:0,ahci-hd,/vm/freebsd/freebsd-i386a.img -s 7:0,passthru,15/0/0 -l com1,/dev/nmdm0A freebsd-i386a &
 
zirias@ said:
I don't know whether it works, I would expect it should work. But then, which OS? FreeBSD? Some very old version?
Click to expand...

That's possibly the key. I just checked, and the image is really old. Like 20+ years. I had assumed that bhyve could boot anything. I will freely admit I'm stumbling around blind.

I haven't been able to boot something more current (12.4-R and 14.0-R) either, which is why I was trying the only other image I could find!
 
rowan194 said:
bhyveload -m 256 -d IBM8.4-ad1.image-THROWAWAY vm0
Click to expand...
Notice here you don't tell it where to send console.

Use -c and send to nmdm (null modem console viewable on host with cu) or use stdio output.

I would also annotate your -m memory usage.
Is that 256M or 256G?
 
The image I tried to boot is FreeBSD, but really, really old (4.x, I think). It was really just me remembering that I happened to have a raw image of an old IDE drive sitting on my NAS, and trying it.

But to regain focus: what's the minimum version that is likely to work with bhyve? That's my main use case - occasional use of an older install of FreeBSD, with applications that are unlikely to compile cleanly on more current versions. I want to retire the hardware. It's running 9-STABLE.
 
I am not sure what the earliest FreeBSD that runs under bhyve is.

Here is the first forum post inquiring about it.
forums.freebsd.org

BHyVe

Not sure if this is the correct place for this question, but over the weekend I'm going to be testing BHyVe on my Intel i3 server and there isn't much information available on BHyve. Does anyone know the limit of the number of FreeBSD guests (since that is all that's supported at the moment)...
forums.freebsd.org forums.freebsd.org

So 10-CURRENT is when it was introduced. I bet 9-STABLE could be ran.
 
Phishfry said:
I am running bhyve on AMD64 and have i386 VM's.

Here is an example snipped from my startup script.

Code: 
bhyveload -m 4G -S -d /vm/freebsd-vm4.img -c /dev/nmdm4A freebsd-vm4 &
sleep 5
bhyve -S -c 10 -m 4G -A -H -P -s 0:0,hostbridge -s 4:0,ahci-hd,/vm/freebsd-vm4.img -s 5:0,passthru,133/0/3 -s 10:0,passthru,13/0/0 -s 31:0,lpc -l com1,/dev/nmdm4A freebsd-vm4 &
Click to expand...
This works because the CPU "knows" whether it's in 16-, 32- or 64-bit mode and whether its in real or protected mode through bits set in machine state registers. All bhyve has to do is set the state of the CPU for the VM while the VM's PID is active.
 
Host: 12.4-RELEASE-p5

Guest 9-STABLE image shows the loader screen, but just quietly exits back to the host shell prompt (no error, no coredump) at "Booting..."

I can't find much information about the "raw" format. Is it literally a disk dump, or something proprietary? I imaged a physical disk this way:

Code: 
dd if=/dev/da0 of=/nas01/backups/c1au-boot-tempimage.img bs=1m

And here's what happens when I try to invoke the VM using that image:

Code: 
root@nas01:~ # bhyveload -m 512M -c stdio -d /nas01/backups/c1au-boot-tempimage.img vm0
Consoles: userboot

FreeBSD/amd64 User boot lua, Revision 1.2
Consoles: userboot

FreeBSD/amd64 User boot 4th, Revision 1.2
Loading /boot/defaults/loader.conf
|

  ______               ____   _____ _____
 |  ____|             |  _ \ / ____|  __ \
 | |___ _ __ ___  ___ | |_) | (___ | |  | |
 |  ___| '__/ _ \/ _ \|  _ < \___ \| |  | |
 | |   | | |  __/  __/| |_) |____) | |__| |
 | |   | | |    |    ||     |      |      |
 |_|   |_|  \___|\___||____/|_____/|_____/    ```                        `
                                             s` `.....---.......--.```   -/
 +------------Welcome to FreeBSD-----------+ +o   .--`         /y:`      +.
 |                                         |  yo`:.            :o      `+-
 |  1. Boot Multi User [Enter]             |   y/               -/`   -o/
 |  2. Boot [S]ingle User                  |  .-                  ::/sy+:.
 |  3. [Esc]ape to loader prompt           |  /                     `--  /
 |  4. Reboot                              | `:                          :`
 |                                         | `:                          :`
 |  Options:                               |  /                          /
 |  5. [K]ernel: kernel (1 of 2)           |  .-                        -.
 |  6. Configure Boot [O]ptions...         |   --                      -.
 |                                         |    `:`                  `:`
 |                                         |      .--             `--.
 |                                         |         .---.....----.
 +-----------------------------------------+


/boot/kernel/kernel text=0x679e22 data=0xab420+0xe4fc8 syms=[0x8+0xca380+0x8+0xe3fb1]
/boot/kernel/zfs.ko size 0x22b048 at 0xdb9000
loading required module 'opensolaris'
/boot/kernel/opensolaris.ko size 0x5040 at 0xfe5000
/boot/kernel/geom_mirror.ko size 0x20670 at 0xfeb000
/boot/zfs/zpool.cache size=0x13a4
Booting...
root@nas01:~ #
 
rowan194 said:
But to regain focus: what's the minimum version that is likely to work with bhyve?
Click to expand...
I think any version would run, as long as it has either virtio drivers or drivers for "real" hardware bhyve emulates. My doubt was about bhyveload(8) being able to boot it, because it uses a special version of loader(8) (/boot/userboot.so from the host). You could try using the userboot.so from the guest (which works for me to boot 15-CURRENT from a 13-RELEASE host), but it won't exist in versions before bhyve was introduced. Or you could try to boot with UEFI/UEFI-CSM instead, which should work for any guest system.

If you're just starting with bhyve, I suggest having a look at sysutils/vm-bhyve, I find it very helpful to manage bhyve vms.
 
The 9.x stable disk does have userboot.so, but trying to use that bombs out:

Code: 
root@nas01:~ # bhyveload -m 512M -c stdio -l /root/userboot-STABLE9.so -d /nas01/backups/c1au-boot-tempimage.img vm0
Abort (core dumped)

I was hoping for a simple one-liner to boot a headless server with a basic network interface and block storage, but it looks like it's going to be a lot more complicated than that...
 
rowan194 said:
I was hoping for a simple one-liner to boot a headless server with a basic network interface and block storage, but it looks like it's going to be a lot more complicated than that...
Click to expand...
Not exactly what you were looking for but if the goal is to run "something" you could use VirtualBox for it (note though you can't mix hypervisors then, so either bhyve or vbox).
 
Sorry I am not familiar with userboot.so and I see no documentation.

What is it providing here? I have never used it.
 
Phishfry said:
Sorry I am not familiar with userboot.so and I see no documentation.

What is it providing here? I have never used it.
Click to expand...
If you ever used bhyveload(8) to boot a FreeBSD guest, you have used it. It's the loader(8) code in the special form to be launched from userspace. Of course this way, you can only boot FreeBSD guests.

Once again, a bhyve guest has more booting options:
The latter two options are "generic" and should be able to boot (almost) any guest.
 
rowan194 said:
The 9.x stable disk does have userboot.so, but trying to use that bombs out:

Code: 
root@nas01:~ # bhyveload -m 512M -c stdio -l /root/userboot-STABLE9.so -d /nas01/backups/c1au-boot-tempimage.img vm0
Abort (core dumped)
Click to expand...
Is this an i386 guest as well? I never tried something like that, so it's guesswork here, but I assume a 64bit bhyveload(8) binary won't be able to use a 32bit userboot.so.

rowan194 said:
I was hoping for a simple one-liner to boot a headless server with a basic network interface and block storage, but it looks like it's going to be a lot more complicated than that...
Click to expand...
I really suggest you use UEFI (or CSM) to boot. Make sure your guest has a working bootloader on its virtual disk, either an EFI loader in an EFI partition, or the MBR bootcode plus the "second stage" in a freebsd-boot partition. Then of course you can launch the virtual machine with a "one-liner", but using the bhyve(8) command directly (with lots of flags). Or you use sysutils/vm-bhyve wich simplifies usage a lot.
 
I managed to get FreeBSD-9.2-RELEASE-amd64-memstick.img (from freebsd.org) to start without needing to specify a separate userboot.so

The image I actually want to boot uses a custom kernel, and in the past I've been fairly aggressive with commenting out unused stuff - including virtio, which I guess is the reason this is failing.

Will take some fiddling around with hardware to be able to boot the disk via bare metal again, but I'll see if recompiling GENERIC fixes it, and allows me to boot via VM image instead.

Edit: Copying GENERIC from 9.2R part works, but dies at the root mount, possibly because I had to rename the image zroot in order to import it and modify the file contents. Unsure if it can be renamed back to zroot on export? Seems I may need to boot in rescue mode (eg on a UFS boot?) in order to do this. Bare metal issue again...

Or perhaps error 45 is some completely different issue, like a 9.2R kernel trying to mount a 9.3+ ZFS file system?

Code: 
Trying to mount root from zfs:zroottmp/ROOT/default []...
Mounting from zfs:zroottmp/ROOT/default failed with error 45.

Edit 2: Found some spare hardware, recompiled GENERIC, 9-STABLE now boots inside a VM.

Code: 
sh /usr/share/examples/bhyve/vmrun.sh -d /dev/da0 vm0

One possibly odd thing I did was use the disk itself as the device, rather than an image of the disk. I thought this may cause problems, since the host can see things like gmirror and zroot on the attached guest disk, but once bhyve is running the device seems to be at least partly locked out on the host? I can see the device, I can 'hd' it, but 'zpool import' doesn't show anything, and 'gpart show da0' says no such geom. The latter works fine inside the VM.
 
Wanted to bump this again to re-ask the original question about running an i386 guest.

I am having no luck trying to boot a FreeBSD i386 image, which is too old to support bhyveload.

I'm struggling to understand how bhyve-grub works, especially for booting a "not linux" OS. I have no idea what to do with a grub> prompt.

One page instructs, "This sets the grub and memory, hit enter again with the installer it will drop you to another terminal" - having trouble parsing this sentence.

The large majority of examples I can find show how to do a fresh install of a (modern!) Linux OS... but that's not what I want.

I did find a github issue saying that byhveload and bhyve-grub were unstable hacks...

Can anyone show me sample command/s that will simply boot a legacy (MBR/non UEFI) raw disk image? Much appreciated.

(Edit: It looks like the package uefi-edk2-bhyve-csm has disappeared, is that perhaps the key to a successful boot?)
 
rowan194 said:
(Edit: It looks like the package uefi-edk2-bhyve-csm has disappeared, is that perhaps the key to a successful boot?)
Click to expand...
No.EDK2 is not used at all on i386.

Phishfry

Solved Post in thread 'Bhyve FreeBSD i386 guest memory'

Sorry for the noise. I use /etc/rc.local to start my VM and I have sleep in between bhyveload and bhyve.
I found the more memory I add to the VM the more sleep time I need for bhyve.
Code: 
bhyveload -S -m 3750M -c /dev/nmdm0A -d /vm/freebsd-i386a.img freebsd-i386a &
sleep 15
bhyve -S -c 16 -m 3750M -AHP -s 0:0,hostbridge -s 1:0,lpc -s 3:0,ahci-hd,/vm/freebsd-i386a.img -s 7:0,passthru,20/0/0 -l com1,/dev/nmdm0A freebsd-i386a &
 
Step One in your quest is decide how you want to install your OS into VM.
Use a pre-defined image or use the installer.
Using the installer is harder because you must pass thru memstick installer and image file where VM will reside.
 
Phishfry said:
Step One in your quest is decide how you want to install your OS into VM.
Use a pre-defined image or use the installer.
Using the installer is harder because you must pass thru memstick installer and image file where VM will reside.
Click to expand...
Thanks for your reply. :)

I'm trying to boot an existing image of a physical HDD, but I'm frustrated by 99% of the tutorials showing how to do a new install, of a current OS. I have an old OS that I just want to run.

I succeeded earlier in the thread with booting an existing 9-STABLE image (after recompiling the kernel with virt support), but this version is even older, and does not support bhyveload.

The OS in the image is FreeBSD, but that's coincidental - I'd be happy to learn how to boot a generic non-UEFI OS...
 
rowan194 most likely you would be better off trying some other emulation software, e.g., qemu.
Based on your comments I suspect that the OS you are trying to boot predates VirtIO and even AHCI.
BHyve does not emulate IDE, so even if you somehow got past the loader issues, the kernel still wouldn't be able to access disks.
 
You must log in or register to reply here.
Back
Top