qemu-aarch64-static 13-Current r346657 amd64 host vm, is it supposed to work? (It does, with caveats, debug guidance appreciated)

CrunchBerry

CrunchBerry

so I've been playing with a raspberry pi 3, and have it running the base system, so I tried moving on to package building with ports-mgmt/poudriere. found out I can't build 13-Head packages on my 12-releng-p3 main OS so stood up a bhyve with 13-Head r346657 . then built a ports-mgmt/poudriere jail from that source with the non-build src= option, native cross-tools built -x and the jail was happy. when I attempting to bulk build it failed after the jail was stood up with uname errors, the basic stuff to determine host build, version, etc. then noticed at first stack errors when I manually ran qemu-aarch64-static against an arm64 binary. now there is no errors, but it doesn't run the file, even statics as found in /rescue/hostname, just line feeds and returns to the shell prompt no output. technically not the same file, but the same versions built in my main 12-releng host do work with some glitches, I'm thinking due to the missing kernel calls. things like hostname works fine. Only issues I've noticed with the 13-r346657 system is witness reporting lock order reversals. not necessarily related to qemu usage.

it looks like emulators/qemu-user-static hasn't been updated for a while in so far as its source tree, so not sure if it just doesn't work Current. haven't found much searching about it, least nothing within the year on Current. I've thought the other option might be to drop back to 12, or try to run an qemu-system-aarch64 multi-core vm to do the same thing from my 12 main system, just want more HP than a pi 3 to build packages. Just curious if folks have gotten aarch64 poudriere package building on 13-Head from an amd64 host, mainly in so far as having qemu function properly, with or without bhyve .
 
[papasmurf@bhyve13 /usr]$ sudo poudriere bulk -j aarch64-Head -p HEAD ~papasmurf/adm/rpi3-ports.list
Password:
[00:00:00] Cross-building ports for arm64.aarch64 on amd64 requires QEMU
[00:00:00] Creating the reference jail... done
[00:01:14] Mounting system devices for aarch64-Head-HEAD
[00:01:14] Mounting ports/packages/distfiles
[00:01:14] Using packages from previously failed build: /usr/local/opt/poudriere/data/packages/aarch64-Head-HEAD/.building
[00:01:14] Mounting packages from: /usr/local/opt/poudriere/data/packages/aarch64-Head-HEAD
[00:01:14] Copying /var/db/ports from: /usr/local/etc/poudriere.d/aarch64-Head-HEAD-options
[00:01:14] Setting up native-xtools environment in jail... done
[00:01:15] Raising MAX_EXECUTION_TIME and NOHANG_TIME for QEMU from QEMU_ values
[00:01:15] Copying latest version of the emulator from: /usr/local/bin/qemu-aarch64-static
[00:01:15] Appending to make.conf: /usr/local/etc/poudriere.d/aarch64-Head-make.conf
/etc/resolv.conf -> /usr/local/opt/poudriere/data/.m/aarch64-Head-HEAD/ref/etc/resolv.conf
[00:01:15] Starting jail aarch64-Head-HEAD
make: "/usr/ports/Mk/bsd.port.mk" line 1154: warning: "/usr/bin/uname -s" returned non-zero status
make: "/usr/ports/Mk/bsd.port.mk" line 1159: warning: "/usr/bin/uname -r" returned non-zero status
make: "/usr/ports/Mk/bsd.port.mk" line 1203: UNAME_r () and OSVERSION (1300021) do not agree on major version number.
make: "/usr/ports/Mk/bsd.port.mk" line 1154: warning: "/usr/bin/uname -s" returned non-zero status
make: "/usr/ports/Mk/bsd.port.mk" line 1159: warning: "/usr/bin/uname -r" returned non-zero status
make: "/usr/ports/Mk/bsd.port.mk" line 1203: UNAME_r () and OSVERSION (1300021) do not agree on major version number.
[00:01:17] Logs: /usr/local/opt/poudriere/data/logs/bulk/aarch64-Head-HEAD/2019-05-03_10h44m52s
[00:01:17] WWW: http://bhyve13/poudriere//build.html?mastername=aarch64-Head-HEAD&build=2019-05-03_10h44m52s
[00:01:17] Loading MOVED for /usr/local/opt/poudriere/data/.m/aarch64-Head-HEAD/ref/usr/ports
make: "/usr/ports/Mk/bsd.port.mk" line 1154: warning: "/usr/bin/uname -s" returned non-zero status
make: "/usr/ports/Mk/bsd.port.mk" line 1159: warning: "/usr/bin/uname -r" returned non-zero status
make: "/usr/ports/Mk/bsd.port.mk" line 1203: UNAME_r () and OSVERSION (1300021) do not agree on major version number.
[00:01:20] Error: Error looking up pre-build ports vars
[00:01:20] Cleaning up
[00:01:20] Unmounting file systems

for qemu

{papasmurf@bhyve13 /usr]$ qemu-aarch64-static /usr/local/opt/poudriere/jails/aarch64-Head/rescue/hostname
[papasmurf@bhyve13 /usr]$

[papasmurf@bhyve13 /usr]$ uname -a
FreeBSD bhyve13.internal.nunya.bus 13.0-CURRENT FreeBSD 13.0-CURRENT r346657 DOMINO amd64
and from the 12 host
[papasmurf@host12 ~]$ uname -a
FreeBSD host12.internal.nunya.bus 12.0-RELEASE-p3 FreeBSD 12.0-RELEASE-p3 r346070 host12 amd64

[papasmurf@host12 ~]$ qemu-aarch64-static ~/projects/beastiepi/release/rescue/hostname
host12.internal.nunya.bus

it would only print the stack error to the console, which i wasn't always on. between the weekend and posting last night I didn't notice the issue on qemu-sbruno's source. I've disabled core dumps, but checking it out with lldb like in the issue

[papasmurf@bhyve13 ~]$ lldb -f `which qemu-aarch64-static` -- /usr/local/opt/poudriere/jails/aarch64-Head/rescue/hostname
(lldb) target create "/usr/local/bin/qemu-aarch64-static"
Current executable set to '/usr/local/bin/qemu-aarch64-static' (x86_64).
(lldb) settings set -- target.run-args "/usr/local/opt/poudriere/jails/aarch64-Head/rescue/hostname"
(lldb) run
Process 66644 launching
Process 66644 launched: '/usr/local/bin/qemu-aarch64-static' (x86_64)
May 3 14:59:09 bhyve13 qemu-aarch64-static[66644]: stack overflow detected; terminated
Process 66644 exited with status = 127 (0x0000007f)

next time I'll be sure to provide suitable mana from the aether to nourish the tools of divination, but thanks for solving this. It just doesn't work right now.
 
in the interest of full disclosure, you can build xorg, but window managers are tough. Ruby doesn't seem to want to build in any form, and many need ruby it seems, or at least the dependent libraries do. "checking whether ELF binaries are produced... Unsupported ancillary data: 1/0" which from my skimming of the qemu source seems to indicate unimplemented ioctl/syscall. Also, noticed that different host versions trigger different internal qemu structures for messages, and one presently are there for 13, which makes sense given its not expected to be supported by ports. I'm going to drop to release 12 where ruby seems to build under qemu in my tests. I tried debugging, but I don't have the skills right now to figure it out. thought setting STATIC_LINK BSD_USER would add debug symbols, but doesn't appear to be the case from my test. I know it does something as it runs slower, and it is quite likely I'm not calling lldb correctly, and or not attaching to the hung process correctly or with the right options.

Under 13 Ruby basically hangs during configure when testing if system creates Elf binaries, so I might try messing with config.site to skip the check, but haven't played with that yet. similar to https://reviews.freebsd.org/D18835 as was pointed out in another post. now if wifi comes to pi, or if what I've read about linux drivers maybe becoming supported so I can try accelerated graphics, I'll give it another go... Even still, was expecting more pain tracking Current, so it was quite a pleasant surprise its been as solid for my uses as a Release would be.

I'll gladly try debugging more, my bhyve isn't going anywhere, just need some guidance on how to get the debugging symbols included, as I'm not 733t enough for this below:
from sorted top
36880 nobody 2 52 0 222M 14M select 3 0:00 0.00% /usr/local/bin/qemu-aarch64-static head -1 conftest
37624 nobody 2 52 0 221M 14M piperd 4 0:00 0.00% /usr/local/bin/qemu-aarch64-static tr -dc \\177ELF
38067 nobody 2 52 0 221M 14M piperd 6 0:00 0.00% /usr/local/bin/qemu-aarch64-static tr \\177 .
38819 nobody 2 52 0 222M 15M select 4 0:00 0.00% head: zygote (qemu-aarch64-static)

[papasmurf@bhyve13 ~]$ sudo lldb -p 36880
Password:
(lldb) process attach --pid 36880
Process 36880 stopped

Executable module set to "/usr/local/opt/poudriere/data/.m/aarch64-Head-HEAD/01/usr/local/bin/qemu-aarch64-static".
Architecture set to: x86_64--freebsd13.0.
(lldb) bt
* thread #1, name = 'qemu-aarch64-sta'
* frame #0: 0x000000006024722a qemu-aarch64-static`___lldb_unnamed_symbol7709$$qemu-aarch64-static + 394
frame #1: 0x00000000600490ac qemu-aarch64-static`___lldb_unnamed_symbol740$$qemu-aarch64-static + 33708
frame #2: 0x0000000060038eac qemu-aarch64-static`___lldb_unnamed_symbol672$$qemu-aarch64-static + 572
frame #3: 0x0000000060038c69 qemu-aarch64-static`___lldb_unnamed_symbol671$$qemu-aarch64-static + 9
frame #4: 0x0000000060039d24 qemu-aarch64-static`___lldb_unnamed_symbol679$$qemu-aarch64-static + 2932
frame #5: 0x000000006000010f qemu-aarch64-static`___lldb_unnamed_symbol1$$qemu-aarch64-static + 271

from a normal qemu-user-static, but the STATIC_LINK BSD_USER option version is the same. obviously I don't expect someone to explain how to debug, but a quick crash on if the unnamed symbols can somehow become named would be awesome.
 
Unsupported ancillary data: 1/0 it's a bug in sendrecvmsg, I have some old patches for that here: http://mikael.urankar.free.fr/index.php?dir=qemu/ (you'll need the patches in time_t and sendrecvmsg). It's also reported here PR 235554

I don't remember quite well how I built a debug version. I think you need make WITH_DEBUG=yes and grep the source to find occurrences of -O2 or -O3 and replace them with -g

contrary to popular belief -current has always been solid and stable. Of course, you still have to be cautious when you upgrade the system.
 
acheron said:
Unsupported ancillary data: 1/0 it's a bug in sendrecvmsg, I have some old patches for that here: http://mikael.urankar.free.fr/index.php?dir=qemu/ (you'll need the patches in time_t and sendrecvmsg). It's also reported here PR 235554

I don't remember quite well how I built a debug version. I think you need make WITH_DEBUG=yes and grep the source to find occurrences of -O2 or -O3 and replace them with -g

contrary to popular belief -current has always been solid and stable. Of course, you still have to be cautious when you upgrade the system.
Click to expand...

been tracking that patch, hoping it gets applied soon... Been thinking of trying them for a while, good to know the key ones to apply though. I take that to use the patches I would need to clone the git source and/or otherwise manually apply them there or within the work folder from the port? said another way putting them in the appropriate port patch folder and make patch isn't going to work?

I'll have to look into that debug process some more. would prefer to stay on current, for my pi
 
You just need to apply them after make patch,
Code: 
cd /usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-454adc1
fetch http://blah
patch < 0001.patch
...
cd  /usr/ports/emulators/qemu-user-static
make
 
You must log in or register to reply here.
Back
Top