Solved Poudriere -> rust 5 hour build failed, 8GB ram + 8GB Swap is not enough to build rust....

nerozero

nerozero

Hello,

5 hour build marathon - ended up like this:

Code: 
  c Assemble { target_compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-freebsd", file: None } } }
  c RustInstaller { compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-freebsd", file: None } }, target: TargetSelection { triple: "x86_64-unknown-freebsd", file: None } }
  c Assemble { target_compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-freebsd", file: None } } }
Dist rustc-dev-1.60.0-x86_64-unknown-freebsd
running: "/wrkdirs/usr/ports/lang/rust/work/_build/x86_64-unknown-freebsd/stage0-tools-bin/fabricate" "generate" "--image-dir" "/wrkdirs/usr/ports/lang/rust/work/_build/tmp/tarball/rustc-dev/x86_64-unknown-freebsd/image" "--component-name=rustc-dev" "--rel-manifest-dir=rustlib" "--legacy-manifest-dirs=rustlib,cargo" "--product-name=Rust" "--success-message=rustc-dev installed." "--package-name=rustc-dev-1.60.0-x86_64-unknown-freebsd" "--non-installed-overlay" "/wrkdirs/usr/ports/lang/rust/work/_build/tmp/tarball/rustc-dev/x86_64-unknown-freebsd/overlay" "--output-dir" "/wrkdirs/usr/ports/lang/rust/work/_build/dist" "--work-dir" "/wrkdirs/usr/ports/lang/rust/work/_build/tmp/tarball/rustc-dev/x86_64-unknown-freebsd"


command did not execute successfully: "/wrkdirs/usr/ports/lang/rust/work/_build/x86_64-unknown-freebsd/stage0-tools-bin/fabricate" "generate" "--image-dir" "/wrkdirs/usr/ports/lang/rust/work/_build/tmp/tarball/rustc-dev/x86_64-unknown-freebsd/image" "--component-name=rustc-dev" "--rel-manifest-dir=rustlib" "--legacy-manifest-dirs=rustlib,cargo" "--product-name=Rust" "--success-message=rustc-dev installed." "--package-name=rustc-dev-1.60.0-x86_64-unknown-freebsd" "--non-installed-overlay" "/wrkdirs/usr/ports/lang/rust/work/_build/tmp/tarball/rustc-dev/x86_64-unknown-freebsd/overlay" "--output-dir" "/wrkdirs/usr/ports/lang/rust/work/_build/dist" "--work-dir" "/wrkdirs/usr/ports/lang/rust/work/_build/tmp/tarball/rustc-dev/x86_64-unknown-freebsd"
expected success, got: signal: 9


Traceback (most recent call last):
  File "x.py", line 27, in <module>
    bootstrap.main()
  File "/wrkdirs/usr/ports/lang/rust/work/rustc-1.60.0-src/src/bootstrap/bootstrap.py", line 1325, in main
    bootstrap(help_triggered)
  File "/wrkdirs/usr/ports/lang/rust/work/rustc-1.60.0-src/src/bootstrap/bootstrap.py", line 1311, in bootstrap
    run(args, env=env, verbose=build.verbose, is_bootstrap=True)
  File "/wrkdirs/usr/ports/lang/rust/work/rustc-1.60.0-src/src/bootstrap/bootstrap.py", line 185, in run
    raise RuntimeError(err)
RuntimeError: failed to run: /wrkdirs/usr/ports/lang/rust/work/_build/bootstrap/debug/bootstrap dist --jobs=1
*** Error code 1

Stop.
make: stopped in /usr/ports/lang/rust
=>> Cleaning up wrkdir
===>  Cleaning for rust-1.60.0
build of lang/rust | rust-1.60.0 ended at Fri May 27 22:33:22 +04 2022
build time: 04:28:53
!!! build failure encountered !!!
In /var/log/messages:
Code: 
May 27 17:33:15 backup kernel: swp_pager_getswapspace(10): failed
May 27 17:57:08 backup kernel: swp_pager_getswapspace(29): failed
May 27 22:29:30 backup kernel: swap_pager: out of swap space

Now the question, how many RAM do I need to build clamav now days ? Or is there a way to limit somehow ram usage for that ?

Thanks
 
Hi,
CPU is a very important factor in compilation time, what is your processor?
5 hours is too long. My compilation time usually takes a few minutes.
Anyway, if you can't compile, you can use a simpler method: freebsd.sh/rust
 
hbsd said:
5 hours is too long. Compilation time I do usually takes a few minutes.
Click to expand...
What's your reasoning behind this statement? Poudriere builds each port on a single CPU core. Rust takes just a little under 4 hours to clean build on my machine.
OP is clearly running out of memory (and also asking about that).

Poudriere builds can be sped up by using devel/ccache and tmpfs(5) - the latter of which will further infrease RAM usage.

As for OPs question: I won't be of any help there. My poudriere instance runs on 128 GB of RAM. And I don't think there is an easy way to check how much memory a particular port build consumed.
I guess you can always increase your swap size. Greybeards often recommend to have your swap size be double of your (physical) RAM. As far as I know, other than being a simple rule-of-thumb, this is mainly to ensure that crash/core dumps fit into swap. YMMV.
 
jbodenmann said:
What's your reasoning behind this statement? Poudriere builds each port on a single CPU core. Rust takes just a little under 4 hours to clean build on my machine.
Click to expand...
I compiled things like nginx/apache with Poudriere which lasted about 40/45 minutes. I didn't meant rust. I don't use too much with Poudriere, just to learn more about Freebsd and fun. I Always use pkg/ports.
 
C
jbodenmann said:
I guess you can always increase your swap size.
Click to expand...
Ccache would fasten builds as you do over time. I will suggest you create a huge swap. Swapfile on drive, swapd or swap on pkg, or zvol swap will help. Preferably, you create a swap file or zvol swap (if zfs). The RAM is low though. 8G cannot get you through several pkg builds. If rust fails, you cannot build firefox.
 
So, will add today a 64GB old SSD drive and will partition it as SWAP entirely .... Now days you need an entire drive as a swap just to compile antivirus....
 
Tons of swap are probably not the greatest idea. Swap is nice to get over temporary memory shortage, but a system constantly swapping will be quite slow... (still, it will most likely solve your problem anyways).

Do you have any custom port options/settings affecting rust? If not, it would be much easier to just use the binary package. ports-mgmt/poudriere-devel can automate that in a clean and easy way with its package fetch option.

Finally, ccache doesn't handle code written in rust, and as large parts of rust are written in rust, it won't help too much. There's ports-mgmt/sccache-overlay to solve that, although this doesn't work fully reliably for me (yet?)
 
At the moment I have to maintain few servers. One of them is a mail server for a small company ~100 users. Mail server works at 0.15 % average load, CPU: 6-th gen I3, RAM: was 4GB become 8GB(yesterday after fist failed attempt to compile ClamAV). So I don't see requirement for a new hardware just for weekly update. At the moment I'm making another server with poudriere just to compile customized ports and provide them for all of the servers running BSD. 24 hour or more compile time is not an issue. Yes, I have asked for a new hardware, but this will not get a green light till September (and this is a best scenario)...

Zirias said:
Tons of swap are probably not the greatest idea
Click to expand...
Hm, this is only solution I can afford at the moment, unfortunately...
 
nerozero said:
Hm, this is only solution I can afford at the moment, unfortunately...
Click to expand...
And there's nothing wrong with that temporary solution while you get long term solution in place.
Of course, a question would be if you are using a currently supported release and running the GENERIC kernel, have you thought about using packages instead of building everything? Of course, I recognize that even if you are running supported GENERIC, there are valid reasons to build ports.
 
Many thanks to everyone!

Rust compile is finally done :) - it took: 05:14:35
Now my only hope is that rust sources doesn't updating too often :)
 
nerozero said:
Now my only hope is that rust sources doesn't updating too often :)
Click to expand...
I don't mean to spoil the fun but when I look at my poudriere logs... ?

Keep in mind that poudriere does the safe thing and rebuilds everything, even if there is a port update on a dependency of a dependency of a dependency of rust it will rebuild the entire chain.
This is why you'd want to use devel/ccache if possible.
 
You must log in or register to reply here.
Back
Top