Solved Unable to build llvm19

R

Reaperzx

I recently upgraded from 12.4 to 13.5. Now I am in the process of rebuilding the ports.

One of the ports I have been unable to build is llvm19 (requirement for new postgresql):

Code: 
[ 87% 9175/10475] /usr/bin/c++ -DMLIR_INCLUDE_TESTS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/usr/ports/devel/llvm19/work-default/.build/tools/mlir/examples/transform-opt -I/usr/ports/devel/llvm19/work-default/llvm-project-19.1.7.src/mlir/examples/transform-opt -I/usr/ports/devel/llvm19/work-default/.build/include -I/usr/ports/devel/llvm19/work-default/llvm-project-19.1.7.src/llvm/include -I/usr/ports/devel/llvm19/work-default/llvm-project-19.1.7.src/mlir/include -I/usr/ports/devel/llvm19/work-default/.build/tools/mlir/include -O2 -pipe -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing  -isystem /usr/local/include -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -Wundef -Werror=mismatched-tags -O2 -pipe -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing  -isystem /usr/local/include  -DNDEBUG -std=c++17  -fno-exceptions -funwind-tables -fno-exceptions -funwind-tables -MD -MT tools/mlir/examples/transform-opt/CMakeFiles/mlir-transform-opt.dir/mlir-transform-opt.cpp.o -MF tools/mlir/examples/transform-opt/CMakeFiles/mlir-transform-opt.dir/mlir-transform-opt.cpp.o.d -o tools/mlir/examples/transform-opt/CMakeFiles/mlir-transform-opt.dir/mlir-transform-opt.cpp.o -c /usr/ports/devel/llvm19/work-default/llvm-project-19.1.7.src/mlir/examples/transform-opt/mlir-transform-opt.cpp
ninja: build stopped: subcommand failed.
===> Compilation failed unexpectedly.
Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to
the maintainer.
*** Error code 1

Stop.
make[1]: stopped in /usr/ports/devel/llvm19
*** Error code 1

Stop.
make: stopped in /usr/ports/devel/llvm19

Any hints how to get it compiled would be appreciated.

I have tried like 3 times by now. What makes it harder, is that compiling llvm takes hours.
 
As Cracauer said there's not much to go on here, but to share a (very!) broad guess => how much memory does your device have? Because with huge projects like these the (virtual?) memory requirements can seriously add up, and if you run out that too can cause issues with building.
 
This virtual machine has 4 cores and 8GB memory. No swap.

These were all the logs i saw in the morning. The compile was running at night in the screen program. I will try again tomorrow. Start compiling in the morning, so I don't have to close the putty window for night.
 
Yes, I used "screen" program for compiling overnight. But in the morning when i resumed, I had just the error message in 1st post, nothing else, can't see what happened before.

Tried compiling again, very long log here in pastebin:
https://pastebin.com/YzSa3b6Z

I guess there relevant part is this FAILED thing:
Code: 
[ 87% 9175/10475] /usr/bin/c++ -DMLIR_INCLUDE_TESTS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/usr/ports/devel/llvm19/work-default/.build/tools/mlir/tools/mlir-opt -I/usr/ports/devel/llvm19/work-default/llvm-project-19.1.7.src/mlir/tools/mlir-opt -I/usr/ports/devel/llvm19/work-default/.build/include -I/usr/ports/devel/llvm19/work-default/llvm-project-19.1.7.src/llvm/include -I/usr/ports/devel/llvm19/work-default/llvm-project-19.1.7.src/mlir/include -I/usr/ports/devel/llvm19/work-default/.build/tools/mlir/include -O2 -pipe -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing  -isystem /usr/local/include -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -Wundef -Werror=mismatched-tags -O2 -pipe -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing  -isystem /usr/local/include  -DNDEBUG -std=c++17  -fno-exceptions -funwind-tables -MD -MT tools/mlir/tools/mlir-opt/CMakeFiles/MLIRMlirOptMain.dir/mlir-opt.cpp.o -MF tools/mlir/tools/mlir-opt/CMakeFiles/MLIRMlirOptMain.dir/mlir-opt.cpp.o.d -o tools/mlir/tools/mlir-opt/CMakeFiles/MLIRMlirOptMain.dir/mlir-opt.cpp.o -c /usr/ports/devel/llvm19/work-default/llvm-project-19.1.7.src/mlir/tools/mlir-opt/mlir-opt.cpp
FAILED: tools/mlir/tools/mlir-opt/CMakeFiles/MLIRMlirOptMain.dir/mlir-opt.cpp.o
/usr/bin/c++ -DMLIR_INCLUDE_TESTS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/usr/ports/devel/llvm19/work-default/.build/tools/mlir/tools/mlir-opt -I/usr/ports/devel/llvm19/work-default/llvm-project-19.1.7.src/mlir/tools/mlir-opt -I/usr/ports/devel/llvm19/work-default/.build/include -I/usr/ports/devel/llvm19/work-default/llvm-project-19.1.7.src/llvm/include -I/usr/ports/devel/llvm19/work-default/llvm-project-19.1.7.src/mlir/include -I/usr/ports/devel/llvm19/work-default/.build/tools/mlir/include -O2 -pipe -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing  -isystem /usr/local/include -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -Wundef -Werror=mismatched-tags -O2 -pipe -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing  -isystem /usr/local/include  -DNDEBUG -std=c++17  -fno-exceptions -funwind-tables -MD -MT tools/mlir/tools/mlir-opt/CMakeFiles/MLIRMlirOptMain.dir/mlir-opt.cpp.o -MF tools/mlir/tools/mlir-opt/CMakeFiles/MLIRMlirOptMain.dir/mlir-opt.cpp.o.d -o tools/mlir/tools/mlir-opt/CMakeFiles/MLIRMlirOptMain.dir/mlir-opt.cpp.o -c /usr/ports/devel/llvm19/work-default/llvm-project-19.1.7.src/mlir/tools/mlir-opt/mlir-opt.cpp


Maybe I should build llvm19 with all dependencies, but how exactly would i do that?
 
To my surprise it won't even work:

Code: 
# dmesg
pid 46714 (c++), jid 0, uid 0, was killed: failed to reclaim memory

Might give the machine a restart just in case... But might still be problem of half-done upgrade process.

htop shows current memory usage: 670M/7.96G. So it is not a memory problem.

EDIT: dmesg works after restart, but of course does not show relevant information.

I guess one more compile in order...
 
Right, so you don't have enough memory for the number of cores you build with.

When you say 4 cores and 8 GB, do you mean plus hyperthreading, aka 8 hardware threads? If so you build with 8 units, which is too much for 8 GB RAM.

4 units and 8 GB comes to 2 GB/execution unit, which should be enough - unless you build in debug mode. Did you turn debug on?

Finally, llvm19 could now use more than 2 GB per execution unit even without debug.

Maybe you get away with giving it some swapspace.
 
No, this is virtual machine from netcup. 4 cores and 8GB is what is see.
dmesg shows:

CPU: AMD EPYC 7702P 64-Core Processor
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)

8GB is actually quite a lot. My last VM used to have 2 cores and 2GB or memory...
Giving swap space would require some advanced reconfiguration of VM, not easy to do after installation.

No, I don't know anything about debugging, never used it.
 
You can use swapping to a file. You don't need a partition.

Either way, apparently llvm now needs more than 2 GB/core even with a non-debug build. You have to deal with that some way or another.
 
Yes, it really uses quite a lot of memory during compiling:

Code: 
    1[||||||||||||||||||||||||||100.0%]
    2[||||||||||||||||||||||||||100.0%]
    3[||||||||||||||||||||||||||100.0%]
    4[||||||||||||||||||||||||||100.0%]
  Mem[|||||||||||||||||||||7.07G/7.96G]
  Swp[                        0K/4.00G]


Thanks, will report back tomorrow.
 
The compilation is usually not the worst thing in the llvm build.

I observed the linker processes go to 4 GB (with debug), each and several of them in parallel.
 
And llvm compile done, Thank you.

But 4G was barely enough. I witnessed time, when swap usage was 4/4 and memory usage was 7/8. Around 87-88% of the compile process.
 
Reaperzx said:
And llvm compile done, Thank you.

But 4G was barely enough. I witnessed time, when swap usage was 4/4 and memory usage was 7/8. Around 87-88% of the compile process.
Click to expand...

Good to know for future reference. So 3 GB/core is what is needed now, plus reserve.

Sound like the linking being a pig.
 
You must log in or register to reply here.
Back
Top