Solved lang/rust - Poudriere build - OOM Killer

rafael_grether

rafael_grether

Hello folks,

I'm absolutely done with lang/rust!

lang/rust cannot be build on a medium machine (Intel Xeon Platinum 8259CL @ 2.5 GHz with 8Gb RAM + 8 Gb Swap), and after almost 5 hours building, gets OOM:
pid 17504 (rustc), jid 117, uid 0, was killed: failed to reclaim memory
pid 44305 (php-fpm), jid 0, uid 80, was killed: failed to reclaim memory
swap_pager: out of swap space

I tried to add more than 16Gb swap space, and nothing.
But I don't want to build RUST anymore. I don't want any more fancy solutions for this.
I'm done.

I'm using Poudriere 3.3.7
So, is there a manually way to only include RUST binary into poudriere?
Or do I need poudriere-devel and use prefetch resource?

PS: Please guys, please...never include RUST into FreeBSD Kernel.

Thanks
 
rafael_grether said:
I'm using Poudriere 3.3.7
Click to expand...
rafael_grether said:
So, is there a manually way to only include RUST binary into poudriere?
Click to expand...
Yes there is! You might want to consider having a look at running ports-mgmt/poudriere-devel instead. It has a new feature (for a long time now) which allows to selectively fetch specific packages from the official FreeBSD binary repository. For ports where you don't have any custom options set this is quite helpful as poudriere will simply fetch those packages (eg. lang/rust) from the FreeBSD binary repos and just put those in your repositories. Then you can build everything else you want yourself.
I was initially very reluctant to use the devel version of poudriere but after some more consideration and research I eventually switched. It worked very well. Haven't had a single issue with it for years.
There are more details to consider tho such as the dependencies of packages you'll be fetching. AFAIK those dependencies will be fetched from the same repository as the package you're instruction poudriere to fetch.
 
Must admit, I'm weary of rust appearing in my poudriere build list. The dependency is clamav which now uses rust. Checked the clamav build options and it is the default.

Can anyone provide details on what I should do to get the rust pkg pulled in?

Oh, found a link, going to try this tomorrow.

forums.freebsd.org

poudriere-devel binary package prefetch whitelist.

I want poudriere-devel to blacklist all binary packages, offcourse except for one package where the build faild on my system. Can this be done and how ? So build every port listed from source except download one package it should blindly fetch binary into the build jails without caring too much...
forums.freebsd.org forums.freebsd.org
 
Tested it, doesn't work for me, still building rust, oh well.... I do have this in poudriere.conf. poudriere-devel-3.3.99.20220831

Code: 
# Set to always attempt to fetch packages or dependencies before building.
# XXX: This is subject to change
# Default: off; requires -b <branch> for bulk or testport.
PACKAGE_FETCH_BRANCH=latest
# The branch will be appended to the URL:
PACKAGE_FETCH_URL=pkg+http://pkg.FreeBSD.org/\${ABI}
# Packages which should never be fetched.  This is useful for ports that
# you have local patches for as otherwise the patches would be ignored if
# a remote package is used instead.
#PACKAGE_FETCH_BLACKLIST=""
# Alternatively a whitelist can be created to only allow specific packages to
# be fetched.
# Default: everything
PACKAGE_FETCH_WHITELIST="gcc* rust llvm*"
 
Last edited by a moderator:
Alain De Vos said:
Try to compile chromium-browser or electron/vscode.
Serious, two settings in poudriere.conf you can try:
Code: 
USE_TMPFS="no"
PARALLEL_JOBS=1
Click to expand...
Thanks Alain De Vos. I will try this tonight.
Is there a way to avoid only some packages (like rust) to use tmpfs?
Anyway, I'm thinking to install poudriere-devel. I was afraid that the devel version was unstable, but I saw that many colleagues are using it.
 
I use ports-mgmt/poudriere-devel, but i think i don't use a specific feature of it.
I compile with parallel jobs 10, in the end the big ports with parallel jobs 1.
Just,
PRIORITY_BOOST="llvm* gcc* rust iridium firefox-esr qt5-webkit qt5-webengine"
 
Alain De Vos , Wow!!
I didn't need to change PARALLEL_JOBS. Only using USE_TMPFS="no", the RUST is builded fine after 5 hours and some minutes, with low ram usage and only 100mb swap; I monitored every 15 minutes.

But if this was the problem (TMPFS limit usage), not RAM + SWAP exhaustion, why OOM Killer?
 
I must be getting something wrong with the invocation of poudriere, poudriere bulk -j FreeBSD_13-1 -p HEAD security/clamav just seems to want to rebuild rust instead of fetching the binary as above.
 
Any chance that you have set or unset options or default versions in your poudriere's build environment which results in either the lang/rust port itself or any of its dependencies not being eligible for recylcing from the existing binary repos?
I assume something as innocently looking as OPTIONS_UNSET+=EXAMPLES can already result in poudriere not being able to fetch the existing package as the package options would not match.

This may or may not also apply to compile flags.
 
RypPn said:
I must be getting something wrong with the invocation of poudriere, poudriere bulk -j FreeBSD_13-1 -p HEAD security/clamav just seems to want to rebuild rust instead of fetching the binary as above.
Click to expand...
Code: 
     -b branch
              Fetch binary packages from a binary package repository instead
              of building them.  The branch argument can be one of the
              following: latest, quarterly, release_X (where X is the minor
              version of a release, e.g., “0”), or url.
{...}
              See PACKAGE_FETCH_BRANCH, PACKAGE_FETCH_URL,
              PACKAGE_FETCH_BLACKLIST, and PACKAGE_FETCH_WHITELIST in
              poudriere.conf.sample.  The entries in the lists will be matched
              against package names without versions.
See poudriere-bulk(8). Option is only available in ports/poudriere-devel.
 
RypPn said:
I must be getting something wrong with the invocation of poudriere, poudriere bulk -j FreeBSD_13-1 -p HEAD security/clamav just seems to want to rebuild rust instead of fetching the binary as above.
Click to expand...
SirDice said:
[...]See poudriere-bulk(8). Option is only available in ports/poudriere-devel.
Click to expand...

Hmm, it seems RypPn is using (the most recent) ports/poudriere-devel:
RypPn said:
Tested it, doesn't work for me, still building rust, oh well.... I do have this in poudriere.conf. poudriere-devel-3.3.99.20220831
Click to expand...

Only thing I can think of is: have you checked the version of your (pkg) installed lang/rust?
I have:
Code: 
# pkg info rust | grep Version
Version : 1.66.1
Based on the info at freshports, this is a rather recent version (e.g. FreeBSD:12:amd64 1.66.1 - 2023-01-20 10:39 – repo build date)
 
There's a truckload of checks that are done too:
Code: 
              poudriere will only use packages that:
              •   come from a repository having the same or older version of
                  pkg.
              •   do not have a locally fetched package already.
              •   are not IGNORED.
              •   match the expected local version.
              •   match the expected ABI.
              •   match the expected runtime and library dependencies.
              •   match the expected OPTIONS when CHECK_CHANGED_OPTIONS is
                  enabled (default: on).
              •   is NOT listed in PACKAGE_FETCH_BLACKLIST.
              •   is NOT listed with -C, or -c, when -t is used.
              The -v flag can be used to show these decisions during build.
              Specifing twice will show more details on why some are skipped.

              WARNING: poudriere has no way of determining differences outside
              of the above list.  That is, if the local ports framework, or
              port, has custom patches or special WITH_FOO knobs (not OPTIONS)
              then it is required to add its name into
              PACKAGE_FETCH_BLACKLIST.  Otherwise a package may be fetched and
              used that lacks the custom patch or knob.
Failing those would trigger a local build instead of downloading and using the package. Adding -v should present a reason why it's not using the package.
 
Yeah, that was lingering in the back of my mind as well. (Reminds me of the same sort of "truckload-list" of conditions that has to be satisfied before an ISG (Idle Stop & Go) starts to engage in my car.)

On the rust build problems with tmpfs as mentioned by rafael_grether in the OP. I think in general activating USE_TMPFS is a good thing—when you don't have a memory constrained system*. Some builds, like lang/rust, seem to take it beyond the limit. That has been noticed; Bryan Drewery:
rust takes >20gb of disk space currently to build. When using tmpfs this can hog a lot of the system memory and also allow too high of a limit for other packages. We need to mount an extra "builder" that avoids tmpfs or allows a higher limit for select packages.
Click to expand...
This has been addressed with the introduction of TMPFS_BLACKLIST: commit. For ports-mgmt/poudriere-devel that is already available; see /usr/local/etc/poudriere.conf.sample or poudriere.conf.sample L65-L72

TMPFS_BLACKLIST for poudriere (non-devel) is announced (note: general title label WIP) in poudriere: release_notes_34—not currently landed in ports.

___
* Memory constrained is, of course, a relative thing.
 
Erichans said:
* Memory constrained is, of course, a relative thing.
Click to expand...

It really is a matter of how many cores you have, and whether you have hyperthreading (on). The first thing I would do is reduce the number of parallel jobs to real cores (not using hyperthreaded pairs).

The problem here is that there are way too many cores for the RAM on many modern machines. 8 cores and 8 GB is common and gives you 1 GB/core. But something like compiling LLVM with debug on requires 4-5 GB per linker process, so something's gotta give. Haven't measured rust, but I don't think it has this brutal of a linker stage.
 
Regarding TMPFS usage: I have a poudriere job running right now building lang/rust. Poudriere reports TMPFS usage of 9.10 GiB (1h 36min into the build).
 
Interesting

[[00:01:31] Package fetch: Skipping rust-1.66.1: deps wanted: curl-7.87.0_1 [00:01:31] Package fetch: Skipping rust-1.66.1: deps remote: curl-7.87.0 [00:01:31] Package fetch: No packages eligible to fetch
 
This would indicate that your local copy of the ports tree wants curl-7.87.0_1 while the remo repo you're using only provides curl-7.87.0.
There's not much you can do about that other than checking out a revision of the ports tree matching whatever the remote used to build which may or may not be trivial to figure out.

Alternatively, just build the lang/rust solo before running the poudriere-bulk build to prevent the system from running out of memory due to other ports being built in parallel.
I do feel like 8GB of RAM + 16GB of swap should be plenty to build this port (and others). Just don't run as many parallel builds.

Are you running anything else on this host which consumes a truck ton ton of memory? Because I just noticed this in your opening post:
rafael_grether said:
pid 44305 (php-fpm), jid 0, uid 80, was killed: failed to reclaim memory
Click to expand...
You wouldn't be running poudriere on a production web server hosting your local government's intranet, would you? :D
 
For me i found that building with less builders but more jobs works better than opposing.
Note: building lang/rust in parallel with llvm15 both requires a lot of memory.
1674581961576.png
 
Give electron18,iridium-browser,chromium,ungoogled-chromium a try ...
Currently i use only two poudriere-builders for them and four threads in make.conf.
 
You must log in or register to reply here.
Back
Top