Solved Does the poudriere bulk -b parameter actually do anything?

Lord Bung · Mar 7, 2023

I've been learning and playing around with poudriere for about a week now. The last bulk build I used -b latest with the command thinking it would greatly reduce the number of packages poudriere would need to build and consequently, the amount of time needed to build whatever packages weren't downloaded, especially since it was only a few days since I had run poudriere ports -u . Well, it would be something of an understatement to say that I was disappointed with any time savings that I may have had. My machine built 842 ports in 36 hours will all 8 cpu cores at 100% almost the entire run. Additionally, having USE_TMPFS=yes set, it used up almost all of my available 64GB of memory for the whole run as well.

I guess my question is, did setting -b latest actually save poudriere from compiling anything?

Here's the command I ran sudo time poudriere bulk -j 13_2_STABLE -f /usr/home/jon/prime-origins.txt -b latest

Code:

(ins)jon:/usr/home/jon poudriere version
poudriere-git-3.3.99.20220831

Alain De Vos · Mar 8, 2023

"-b" does something.
You can mainly choose between "quarterly" or "latest" just as with packages.
In general with "latest" you need to re-compile more frequently.
With "quarterly" it's mainly each new quarter you need to recompile.

I use "quarterly" so i need to re-compile less. But it also means the ports can be a small bit "older".

zirias@ · Mar 8, 2023

Lord Bung said:
I guess my question is, did setting -b latest actually save poudriere from compiling anything?

poudriere tells you how many packages it fetched and how many it built. This bulk build on my testing machine went exceptionally well: https://home.palmen-it.de/testbuilder/build.html?mastername=14a-default&build=2023-03-08_13h11m08s (link will go dead in a few weeks, it shows 103 packages fetched, 6 built). Normally, I end up with 30-50% of the packages fetched.

Typical reasons for "disappointing" results:

Custom options (individual port options as well as stuff in make.conf like DEFAULT_VERSIONS). Any port affected by that must be built locally. Keep in mind this also forces any dependent ports to be built locally. Of course, without any custom options it wouldn't make sense to build yourself at all. But the more you customize, the fewer official packages can be used.
Using the main ports branch ("latest"). This has a high frequency of changes, so it will often differ a lot from the state from which the "latest" package repository was built. If you want, you can work around that: On https://pkg-status.freebsd.org/ you can find the latest completed package build for your FreeBSD version and architecture. The build name will be the short git commit hash. So, just git checkout that state before building to use as many official packages as possible.

astyle · Mar 8, 2023

Lord Bung said:
I guess my question is, did setting -b latest actually save poudriere from compiling anything?

Only if you edit poudriere.conf and allow Poudriere to download the dependency packages on-demand.

poudriere/src/etc/poudriere.conf.sample at master · freebsd/poudriere

Port/Package build and test system. Contribute to freebsd/poudriere development by creating an account on GitHub.

github.com

Start at line 368... That option is OFF by default, and only makes sense with -b <branch> flag.

FWIW, I don't have that on - I'd rather compile the whole enchilada, with port options I choose. Pkg has very conservative options pre-compiled.

zirias@ · Mar 8, 2023

astyle said:
Only if you edit poudriere.conf and allow Poudriere to download the dependency packages on-demand.

Well, I didn't try it, but that's not what's documented. You can enable fetching of binary packages in the configuration (that's what I do) and control some more aspects of it like black-/whitelisting packages (I blacklist all *-kmod packages for example).

But according to the manpage, -b on the commandline enables the feature for this run.

Lord Bung · Mar 8, 2023

astyle said:
Only if you edit poudriere.conf and allow Poudriere to download the dependency packages on-demand.

poudriere/src/etc/poudriere.conf.sample at master · freebsd/poudriere

Port/Package build and test system. Contribute to freebsd/poudriere development by creating an account on GitHub.

github.com

Start at line 368... That option is OFF by default, and only makes sense with -b <branch> flag.

Interesting. I did not look at poudriere.conf.sample after switching from poudriere to poudriere-devel .

poudriere tells you how many packages it fetched and how many it built.

This run did not fetch any packages.

I'm not sure which make.conf you were referring to Zirias@.

 

(ins)jon:/usr/ports[main] cat /etc/make.conf 

WITH_CCACHE_BUILD=yes



(ins)jon:/usr/ports[main] cat /usr/local/etc/poudriere.d/make.conf 

CCACHE_DIR=/var/cache/ccache

WITH_CCACHE_BUILD=yes

(ins)jon:/usr/ports[main]

I have thought about updating ports, poudriere ports -u , and then waiting a few days before building anything to give the ports build machines a chance to do their work. At the moment, I don't have any ports with custom settings, so I don't actually need to build anything myself, but I wanted to tinker with this and get it working. I'm thinking about adopting a port that does not have a maintainer and is a few years out of date. It's one I use and I'd like to get on the latest version.

astyle · Mar 8, 2023

My reading was that the -b flag (and black/whitelisting certain packages) only works if you allow packages to be downloaded in the first place. If you don't allow packages to be downloaded at all, it doesn't make sense to specify the repo or black/whitelist certain packages... those filters will be coming up empty. I wouldn't expect the -b flag to override what's in the poudriere.conf file...

Lord Bung · Mar 8, 2023

astyle said:
I wouldn't expect the -b flag to override what's in the poudriere.conf file...

Oh, I would. Think about it. You use a command line parameter to supersede the default settings. That's one of the main things they are used for.

zirias@ · Mar 8, 2023

astyle said:
I wouldn't expect the -b flag to override what's in the poudriere.conf file...

It certainly does the other way around, I sometimes use -b '' to prevent fetching packages. As I understand it, it's just the same as the PACKAGE_FETCH_BRANCH knob in the config (which must be non-empty to even attempt fetching packages), and, as always, commandline overrides configuration.

patmaddox · Mar 8, 2023

zirias@ said:
Using the main ports branch ("latest"). This has a high frequency of changes, so it will often differ a lot from the state from which the "latest" package repository was built. If you want, you can work around that: On https://pkg-status.freebsd.org/ you can find the latest completed package build for your FreeBSD version and architecture. The build name will be the short git commit hash. So, just git checkout that state before building to use as many official packages as possible.

This is almost certainly the piece that you're missing. Keep in mind that if a package builds, any dependent packages will also build. So if you're using latest, there's a decent chance that you end up building stuff, because latest may have last been built several days ago, plus there's the time it takes for packages to sync to the servers.

It's not fully reliable because of the time it takes to sync, or maybe the repo failed to build a particular package version. So, you might consider checking out an even older version of ports repo if you find it's still building.

Beyond that, I'm not sure.

Lord Bung · Mar 9, 2023

Thanks for all the insights guys. I did check https://pkg-status.freebsd.org/ and found that default had not been built in several days. It seems the machine was tied up with quarterly builds. I naively assumed from my experience at $JOB that builds of latest would probably occur as soon as the previous build had completed. I think I'll wait a while and check the build status before starting more builds with poudriere. The novelty of locally building many hundreds of ports all with default settings has long since faded. ? I am glad to have this set up and working however.

astyle · Mar 9, 2023

Lord Bung said:
The novelty of locally building many hundreds of ports all with default settings has long since faded

BTW, I didn't play with Poudriere because I wanted to reproduce defaults with my own hardware.

I have a subset of ports that I want to keep compiling with my own options, and keep upgrading just those - against a working system. Right now, upgrading KDE means upgrading the entire machine, and it's easier to reinstall EVERYTHING from scratch than to upgrade just the KDE packages. There's stuff I want to leave alone, I'm not interested in upgrading the vast majority of the ports that I have installed - primarily because I compiled everything else with my own options, and am happy with what I have. Same problem exists on Linux, but I like what FreeBSD has available (ZFS, ports, Poudriere, and consistency), and prefer to make use of that to solve my problem