Solved Win10 guest isoboot failed but win2016&2019 succeed

Rex Ai · Feb 2, 2021

This is some dmesg output on the host:

Code:

---<<BOOT>>---
Copyright (c) 1992-2020 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.2-RELEASE-p3 GENERIC amd64
FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c
611aa2)
VT(efifb): resolution 1024x768
CPU: Intel(R) Xeon(R) CPU E5-4620 v3 @ 2.00GHz (1997.73-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306f2  Family=0x6  Model=0x3f  Stepping=2
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PS
E36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7ffefbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX1
6,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C
,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x21<LAHF,ABM>
  Structured Extended Features=0x37ab<FSGSBASE,TSCADJ,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM
,NFPUSG>
  Structured Extended Features3=0x9c000400<MD_CLEAR,IBPB,STIBP,L1DFL,SSBD>
  XSAVE Features=0x1<XSAVEOPT>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics
real memory  = 412316860416 (393216 MB)
avail memory = 401743728640 (383132 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <HP     ProLiant>
FreeBSD/SMP: Multiprocessor System Detected: 80 CPUs
FreeBSD/SMP: 4 package(s) x 10 core(s) x 2 hardware threads

I run the following command with sysutils/bhyve-firmware:

Code:

bhyve -AHP -c cpus=4,sockets=1,cores=2,threads=2 -m 16g -s 0,hostbridge -s 31,lpc \
-s 1:0,ahci-cd,/root/win10-20h2.iso \
-s 1:1,ahci-hd,/dev/zvol/zroot/win10sys \
-s 1:2,ahci-hd,/dev/zvol/zroot/win10data \
-s 2,e1000,tap0 -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
-s 3:0,fbuf,tcp=54321,w=1920,h=1080,wait -s 3:1,xhci,tablet win10

When I connect the guest with VNC and "press any key", I can see the Windows logo. After this, VNC is automatically done and the terminal shows:

Code:

fbuf frame buffer base: 0xc42600000 [sz 16777216]
Unhandled ps2 keyboard command 0x02
Unhandled ps2 keyboard command 0x02
Unhandled ps2 keyboard command 0x02
rdmsr to register 0x3a on vcpu 0
vm exit[0]
        reason          VMX
        rip             0x0000000001a40cdf
        inst_length     3
        status          0
        exit_reason     2 (Triple fault)
        qualification   0x0000000000000000
        inst_type               0
        inst_error              0
Abort trap

I verified the ISO checksum and it's right. If I replace win10 ISO with win2016&2019 ISO, everything is fine.

What's the problem? Thanks.

usdmatt · Feb 2, 2021

I'm pretty sure I've run Windows 10. Try moving the CD & disks tp slots 3/4/5 and move the other devices using them.

The UEFI firmware used to required disks to be on those slots as they were the only ones set up for it in the firmware. I would of thought that would affect 2016/2019 as well, but then the mailing lists say this:

> Depends on the o/s being booted - if it needs legacy interrupt
> support, the only available slots with routing set up are 3/4/5/6

It's possible 2016+ don't need legacy interrupts.

Raffeale · Feb 3, 2021

change the network card to virtio-net ,the e1000 has some problems for win10

Rex Ai · Feb 6, 2021

Thanks all guys and sorry for my late reply.

According to my test, the reason is that sysutils/bhyve-firmware is incompatible with Windows installation ISO after version 1909. Notice this means the installation ISO, not the OS. For example:
1) Win2016 installation ISO boots succeed because it's based on version 1607.
2) Win2019 installation ISO boots succeed because it's based on version 1809.
3) Win10 installation ISO boots succeed If using the older version like 1909, and failed if using the newer version like 2004 or 20H2.

Although I don't know what's changed in ISO after version 1909 since it's property software, the boot screen proves they aren't identical: the font, size and color of "Press any key to boot from CD or DVD" are different.

On the other hand, the information about slot 3/4/5/6 looks outdated. I tested the slots from 0 to 10 and no error.

So, if you want to install the newest version Win10 guest, the workaround is:
1) Use 1909 ISO to boot and install the guest, including virtio-blk and virtio-net drivers.

Code:

bhyve -AHP -c cpus=4,sockets=1,cores=2,threads=2 -m 16g -s 0,hostbridge -s 31,lpc \
-s 1:0,ahci-cd,/root/win10-1909.iso \
-s 1:1,virtio-blk,/dev/zvol/zroot/win10sys \
-s 1:2,virtio-blk,/dev/zvol/zroot/win10data \
-s 2,virtio-net,tap0 -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
-s 3:0,fbuf,tcp=54321,w=1920,h=1080,wait -s 3:1,xhci,tablet win10

2) Power cycle the guest and upgrade to the newest version like 20H2 by Windows Update.

Code:

bhyve -AHP -c cpus=4,sockets=1,cores=2,threads=2 -m 16g -s 0,hostbridge -s 31,lpc \
-s 1:0,virtio-blk,/dev/zvol/zroot/win10sys \
-s 1:1,virtio-blk,/dev/zvol/zroot/win10data \
-s 2,virtio-net,tap0 -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
-s 3:0,fbuf,tcp=54321,w=1920,h=1080 -s 3:1,xhci,tablet win10

Raffeale · Feb 6, 2021

i have directly installed win10 h2 without any problems

Rex Ai · Feb 7, 2021

Raffeale said:
i have directly installed win10 h2 without any problems

Could you paste the commands to let me try?

Raffeale · Feb 7, 2021

bhyve -c 2 -m 4G -H -w -s 0,hostbridge -s 3:0,ahci-cd,/root/win10_x64.iso -s 3:1,ahci-cd,virtio.iso -s 4,ahci-hd,win10.img -s 5,virtio-net,tap0 -s 29,fbuf,tcp=127.0.0.1:5900,wait -s 30,xhci,tablet -s 31,lpc -l com1,stdio -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd win10
i download the win10 international edition from microsoft website

Rex Ai · Feb 7, 2021

OK, I find the reason. The guest can boot normally if adding -w:

Code:

bhyve -AHPw -c cpus=4,sockets=1,cores=2,threads=2 -m 16g -s 0,hostbridge -s 31,lpc \
-s 1:0,ahci-cd,/root/win10-20h2.iso \
-s 1:1,virtio-blk,/dev/zvol/zroot/win10sys \
-s 1:2,virtio-blk,/dev/zvol/zroot/win10data \
-s 2,virtio-net,tap0 -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
-s 3:0,fbuf,tcp=54321,w=1920,h=1080,wait -s 3:1,xhci,tablet win10

But I can't understand completely:

Code:

-w          Ignore accesses to unimplemented Model Specific
            Registers (MSRs).  This is intended for debug
            purposes.

Raffeale · Feb 8, 2021

maybe the win10 use syscall instruction to replace system interrupt

a6h · Feb 8, 2021

Disable CVE-2017-5715 (Spectre V2) in Windows guest

1. Log in Windows with -w enabled.
2. Run file.exe or pwsh.exe or powershell.exe as administrator.
3. Run these commands:

C:\>

 reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management" /v FeatureSettingsOverride /t REG_DWORD /d 1 /f

C:\>

 reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management" /v FeatureSettingsOverrideMask /t REG_DWORD /d 3 /f

4. Turn off windows guest, and start it again without -w
5. Test, and report back.
6. (Revert commands)

C:\>

 reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management" /v FeatureSettingsOverride /t REG_DWORD /d 0 /f

C:\>

 reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management" /v FeatureSettingsOverrideMask /t REG_DWORD /d 3 /f

Rex Ai · Feb 18, 2021

According your information, I got something like:
1. CVE-2017-5715 is enabled by default on Windows Client but disabled by default on Windows Server
2. Microsoft provides a PowerShell script to verify the status of speculative execution side channel mitigations

Besides regedit, I used this script to check the result of step 3. It changed BTIDisabledBySystemPolicy from "False" to "True". The value will be reverted to "False" for step 6. The same error occured whether "True" or "False".

Then I compared the script output between 1909 and 20H2. The only difference is, BTIKernelImportOptimizationEnabled on 1909 is "False" but 20H2 is "True". Nonetheless, I don't know how to change it to "False" on 20H2.

Finally, I feel the simplest trick is "add -w"...