I had time this week to try to tackle upgrading my 13.3-RELEASE-p3 Xen server to major release 14. Sadly this did not go well.
I get a kernel panic in stage three (see below for output and CPU specs). I rolled back to 13.3 for now and everything works.
Ironically from what I can see, even on 13.3-p3 the xen-kernel is the same version as in latest 14.1 which is 4.18.0.20240201
There's a bug report from earlier this year that describes similar behavior but was suposedly fixed with this very version update of the xen-kernel.
I don't use any fancy cmdline options for my dom0 and I tried booting 14.0 without any other boot loader options except ZFS just to be as vanilla as possible, to no avail either.
Note that console=com1,vga is the cmdline default for console, I don't understand why some people tend to write out defaults in configs, it's redundant and doesn't help anybody IMHO #SorrySlightRant
As mentioned, this config works very well, with the latest Xen kernel on 13.3-RELEASE-p3
I can not recreate this on another intel based Laptop I have lying around with a fresh install of 14.1-RELEASE with the same exact config lines (except 4G mem instead of 16).
But I also didn't try to recreate the whole scenario, it might be hardware related to begin with.
I have tried to boot with the xen-debug kernel in hopes that I would get even more output, but the panic reads the same:
What I can see is that the clang version in 14 was pushed to 18.1.5 and llvm was also mentioned in the bug report I linked. The working installation on 13.3 still uses clang 17.0.6
(output above from upgrade all the way to 14.1 but the same exact output prints for 14.0)
For now this is preventing me from upgrading to 14. Any ideas on what to do next?
I get a kernel panic in stage three (see below for output and CPU specs). I rolled back to 13.3 for now and everything works.
Ironically from what I can see, even on 13.3-p3 the xen-kernel is the same version as in latest 14.1 which is 4.18.0.20240201
There's a bug report from earlier this year that describes similar behavior but was suposedly fixed with this very version update of the xen-kernel.
I don't use any fancy cmdline options for my dom0 and I tried booting 14.0 without any other boot loader options except ZFS just to be as vanilla as possible, to no avail either.
Code:
dom0_mem=16G,max:16G dom0=pvh com1=115200,8n1As mentioned, this config works very well, with the latest Xen kernel on 13.3-RELEASE-p3
I can not recreate this on another intel based Laptop I have lying around with a fresh install of 14.1-RELEASE with the same exact config lines (except 4G mem instead of 16).
But I also didn't try to recreate the whole scenario, it might be hardware related to begin with.
I have tried to boot with the xen-debug kernel in hopes that I would get even more output, but the panic reads the same:
Code:
FreeBSD 14.1-RELEASE releng/14.1-n267679-10e31f0946d8 GENERIC amd64
FreeBSD clang version 18.1.5 (https://github.com/llvm/llvm-project.git llvmorg-18.1.5-0-g617a15a9eac9)
VT(vga): text 80x25
XEN: Hypervisor version 4.18 detected.
CPU: Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz (3292.52-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x306a9 Family=0x6 Model=0x3a Stepping=9
Features=0x1fc3fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT>
Features2=0xbfba2203<SSE3,PCLMULQDQ,SSSE3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,HV>
AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
AMD Features2=0x1<LAHF>
Strucutred Extended Features=0x281<FSGSBASE,SMEP,ERMS>
Structured Extended Features3=0x20000000<ARCH_CAP>
XSAVE Features=0x1<XSAVEOPT>
IA32_ARCH_CAPS=0x4000000
AMD Extended Feature Extensions ID EBX=0x100000
TSC: P-state invariant
ACPI APIC Table: <ALASKA A M I>
Package ID shift: 4
L3 cache ID shift: 4
L2 cache ID shift: 1
L1 cache ID shift: 1
Core ID shift: 1
AP boot address 0x1000
panic: AP #1 (PHY# 1) failed!
cupid = 0
time = 1
KDB: stack backtrace:
#0 0xffffffff80b7fbfd at kdb_backtrace+0x5d
#1 0xffffffff80b32961 at vpanic+0x131
#2 0xffffffff80b32823 at panic+0x43
#3 0xffffffff80fe22c2 at start_all_aps+0x592
#4 0xffffffff80fe1d23 at cpu_mp_start+0x1a3
#5 0xffffffff80b936de at topo_analyze+0x42e
#6 0xffffffff80abb425 at mi_startup+0xb5
#7 0xffffffff80fde0bc at xen_start32+0xbc
Uptime: 1s
Rebooting...(output above from upgrade all the way to 14.1 but the same exact output prints for 14.0)
For now this is preventing me from upgrading to 14. Any ideas on what to do next?