Do not use ports-mgmt/portmaster and other tools who build in the main system

Is this right?

  • Yes

    Votes: 18 27.7%
  • No

    Votes: 39 60.0%
  • Don't know

    Votes: 8 12.3%

  • Total voters
    65
As there's continued misconceptions around that topic, here's another explanation:

First, "recursive make": make is a tool using commands from a "recipe" to process input files to output files in a process that can cascade. So it might produce file_b from file_a, which is needed to produce file_c later. To do that automatically, the input to make is a set of dependency rules (here, file_c "depends" on file_b) plus recipes which are short shell scripts required to do the actual processing.

As a consequence, make might fail when you forget to list a dependency. In the scenario above, if make attempts to create file_c before it created file_b, this will obviously fail. Even worse, make uses file timestamps to know which files are outdated and must be recreated. So, if you have a file_b from a previous build and file_c misses the dependency on it, it might be built using an older version of file_b.

Now, enter "recursive make": This is a widespread idiom for organizing a large source tree. Instead of depending on individual files from a subdirectory, you depend on a pseudo-target with a recipe that will just call make again inside this subdirectory, that has its own Makefile. This is somewhat simplified, but the main consequence is that you have to be very careful about the dependencies of these pseudo-targets, so the directories are built in the correct order (if you have errors there, they will typically manifest when building in parallel with make's -j option), and it will result in whole directories being built even if only a single file would be needed. But it has a few advantages as well: The whole structure is easier to grasp than with ONE huge Makefile (possibly using includes from subdirectories), and you have an easy way to just build one individual directory by calling make from in there. So, assuming you get the ordering correct, recursive make still has the one downside that it will build too much when only little parts changed. FreeBSD /usr/src uses recursive make and solves the problem of unnecessary build steps in a different way, see meta-mode.

The ports tree also uses recursive make. But this is an entirely different thing, as it doesn't build anything itself, but instead organizes builds of individual software packages, from downloading the distribution files to packaging the result of the builds. For the actual builds, the build systems provided by the upstream packages is used. So, there is just no alternative to recursive make for a thing like the ports tree. Furthermore, this has nothing to do with the handling of dependencies between ports. These are handled in a clever way by the make framework in /usr/ports/Mk, which uses special variables set in the Makefiles like e.g. BUILD_DEPENDS. And finally, you almost never use the recursive infrastructure of the ports tree, but instead build individual ports directly from within their directory.

Portmaster also doesn't use the recursive make infrastructure, but parses the special variables in the individual ports' Makefiles and handles these dependencies itself. So, it has nothing to do with "recursive make". Also, there's nothing wrong with Portmaster per se, it does work as intended.

Now, what's the problem with Portmaster? The same as with building ports manually from the ports tree. It's neither a problem with the ports tree OR with Portmaster itself, but with the build systems of some upstream software packages.

Many software packages have build-time configurations, so they can optionally require additional libraries for optional features. The build systems used offer means for configuration, e.g. with the widespread "GNU autotools", you will have a ./configure shellscript doing that. These configuration tools are meant to do two things: They let the user decide (e.g. with commandline flags) what optional features should be enabled, and they check whether the needed libraries for these features are actually available.

Now, due to either bugs or cluelessness of upstream authors, it can happen that some optional feature is automatically enabled if the build system finds the needed libraries on the system you build on. This is called an "automagic dependency" and should be avoided, but sometimes, upstream build systems fail to do so. It happens mostly when home-brewn build systems (e.g. with large hand-written Makefiles) are used. Say the port/package "frobnicate" has an option to enable feature BAR, which should cause it to link "libbar.so". What *could* happen is that you don't enable that option, so the package you install won't have a dependency on libbar, but the buildsystem would find that libbar.so on your system and link to it nevertheless. You will end up with an implicit dependency pkg knows nothing about, and once you remove "libbar" (pkg will have no objections, nothing depends on it!), your "frobnicate" is suddenly broken.

Tools like poudriere or synth solve this problem: They run each build in a clean environment with ONLY the packages installed that are actual and intended dependencies of the port to build. This is again a BIT simplified, but it's basically the main thing…

So, all of this has nothing to do with "recursive make", and Portmaster isn't "broken" either, it's just there are better and more reliable alternatives and you should look into using one of them to profit from it.
 
Zirias : Thank you for that clear example of why a clean build environment matters. I've been using FreeBSD for something like 20 years, and never was really aware of that.

As for the core of the debate: I started out using portupgrade, since that was what the handbook said to do back in FreeBSD 3. That was a PITA, since you had to either:
  1. Keep watching the build in case something had a config screen that needed to be OK-ed or edited
  2. Use the --batch flag, which had its own problems and risks (for example: giving you large numbers of unwanted dependencies, such as with things that try to install x11 by default)
That alone was one of the reasons I started using portmaster: It moves all those config commands to the start of the process, and after that you only have to keep checking if it still looks like the Matrix in that terminal...

( --list-origins is also a very useful flag if you're replicating a server.)

I never switched to poudriere, as I never needed it. Sure, portmaster has some problems, but if you know what it's doing, they are easy to deal with. For instance, I switched to security/libressl after Heartbleed, and after the first time updating it, I knew that whenever that needs an upgrade, I need to pass the -r flag to portmaster, in case the library number has been incremented (which would cause Apache and other software depending on it be confused at its next restart).

This thread has made me take a good look at synth, and I'm currently giving it a try on one of my servers. If that works, I might integrate it into my workflow.

Interesting thing: building synth using portmaster failed on my system (devel/adacurses installs devel/ncurses, but synth itself wants ncurses from base). Using the ol' make install clean worked fine though. This seems to be somewhat related to what Zirias describes above...
 
Hi Jose,

* I rsync the ports to the server. I have my private repository on both machines set to a higher priority than the Freebsd quarterly repo. This allows me to install a mix of locally-built and standard ports

Could you please elaborate on this point?

As I understand, you are building packages from ports, so I assume that you rsync the built packages, correct? So should the last sentence not read:

This allows me to install a mix of locally-built and standard ports packages.

I am not being grammar Nazi, but it seems to be important here, because of the repeated mantra of "not mixing [packages buit form] ports and packages", which is implied.

I further do not understand why would one want to do such. I have been using ports-mgmt/synth and its options allows to fetch and install a package if the inter-dependencies do not require build it. So, I give ports-mgmt/synth the port reference and if ports-mgmt/synth does not need to build, ports-mgmt/synth just downloads and - if instructed - installs the package.

Or perhaps I am misunderstanding your work-flow?

Kindest regards,

M
 
As I understand, you are building packages from ports, so I assume that you rsync the built packages, correct? So should the last sentence not read...
I am not being grammar Nazi, but it seems to be important here, because of the repeated mantra of "not mixing [packages buit form] ports and packages", which is implied.
You are correct! Post edited.

I further do not understand why would one want to do such. I have been using ports-mgmt/synth and its options allows to fetch and install a package if the inter-dependencies do not require build it. So, I give ports-mgmt/synth the port reference and if ports-mgmt/synth does not need to build, ports-mgmt/synth just downloads and - if instructed - installs the package.
I've never tried synth, and don't intend to. I believe it builds and installs ports, not packages? My reasons for avoiding tools like portmaster and synth are:
  • I don't want to build the same stuff over and over. It's a waste of electricity
  • I want to set some top-level options and some package level options for the packages I use
  • I want my custom packages to build in a pristine and self-consistent environment
  • I like to have one beefy machine that is not on all the time do the builds. The downstream consumers of my builds are on all the time, but are more power-efficient.
HTH
 
  • Thanks
Reactions: a6h
Hi Jose,

thank you for the reply. However, by my introduction of the ports-mgmt/synth, I derailed the conversation.

So, by having your locally-built and standard repository, you do mix packages buit from ports and standard packages, correct?

Kindest regards,

M
 
So, by having your locally-built and standard repository, you do mix packages buit from ports and standard packages, correct?
That is not how I would describe it. My systems have a combination of packages built by Poudriere using standard options on Freebsd.org hardware and packages built by Poudriere using my options on my hardware.

The details are important because that's where the devil lives. When I run Poudriere, it builds not just the packages I want, but also the entire dependency tree for these pacakges every time (I use the -c option to ensure this); and it builds them on a fresh install of 12.2-RELEASE with absolutely nothing else installed in it. Packages built in this way are preferred over standard packages to ensure consistency. I do this before every run of pkg-upgrade(8).
 
Hi Jose,

I appreciate your willingness to explain your workflow, but I think that I am not knowledgeable enough to appreciate the difference. As I understand all the standard packages, to use your term, are built by Freebsd.org using Poudriere with the standard options. You are modifying options on some of the packages, but then use these packages and the standard packages. Thus, mixing packages build [with non-standard options] from ports and packages built with standard options.

Do not worry about my failure to understand, I am glad that it works for you.
Kindest regards,

M
 
I appreciate your willingness to explain your workflow, but I think that I am not knowledgeable enough to appreciate the difference. As I understand all the standard packages, to use your term, are built by Freebsd.org using Poudriere with the standard options. You are modifying options on some of the packages, but then use these packages and the standard packages. Thus, mixing packages build [with non-standard options] from ports and packages built with standard options.
Yes, but Poudriere ensures that my nonstandard options are propagated to the entire tree of packages.


Consider the following example. I do not use Mysql, and prefer Postgresql. The standard options usually do the opposite, they enable Mysql compatibility, and disable Postgresql. I have the following in /usr/local/etc/poudriere.d/make.conf.
Code:
OPTIONS_SET+=PGSQL
OPTIONS_UNSET+=MYSQL

Say I want to run application foo. Application foo depends on library bar. They both have Mysql and Postgresql options. I install them out of ports, and I have the correct options in my make.conf. Everything connects to my Postgres and things are happy.

Some time later, I decide to install package baz that also depends on library bar. I install it out of packages without thinking. Two things can happen now. I can be annoyed by the large amount of Mysql stuff that gets dragged in and realize my mistake, or I could go blissfully along and then spend some time frustratingly trying and failing to get baz to connect to my database.

This would've all been avoided if I'd just built bar and baz with Poudriere. They, and all the libraries they depend on would've all been built with the correct options.

This is a very simple example of a dependency tree with just one branch and one level of edges. Imagine how quickly this can get out of hand when you have dependency trees with many branches and levels. Imagine further that database options come and go with new application and library versions.

The term I like for servers that are lovingly hand-crafted with make.conf options, src.conf settings and the like is snowflake server. They are beautiful and unique, and a nightmare to maintain or migrate away from. Yes, been there, done that. I work hard up front now to make my server builds reproducible. It pays off in terms of time and frustration over the long run.
 
Hi Jose,

thank you for the reply, but it seems that the example that you have given with the foo, bar, and baz supports my contention of mixing (packages build from) ports with standard packages, i.e., you build the non-standard package foo with Poudriere, and all the non-standard option is propagated through. Perfect.

But, next you add a package with standard options, which causes the nightmare scenario that you describe. Thus, is this not what I described as mixing (packages build from) ports with standard packages? If not what am I missing?

I originally asked about the specific statement because I have been building all the packages from ports, exactly to avoid the scenario described in your message above. Yes, it results in a lot of rebuilding if it is needed.

Again, the fact that I do not understand does not mean that it does not work well for you.

Kindest regards,

M
 
thank you for the reply, but it seems that the example that you have given with the foo, bar, and baz supports my contention of mixing (packages build from) ports with standard packages, i.e., you build the non-standard package foo with Poudriere, and all the non-standard option is propagated through. Perfect.
This would be true iff ports options never changed. Fact is they do, and you'll wind up with an inconsistent set of options eventually if you mix packages and ports. Upgrades should be a transaction. Either all the ports in a dependency tree are updated at the same time or none are.

If your contention is that you can safely mix packages and ports if you never mess with the standard options; this is probably true modulo some time jitter in the ports tree, but why would you bother building ports at all then? You're just wasting time and effort, since the best outcome of this approach is that you'll wind up with exactly the same software as if you'd installed the standard package. Also, there are not-so-good outcomes for this approach when the standard ports tree is out of sync with the standard packages set.
But, next you add a package with standard options, which causes the nightmare scenario that you describe. Thus, is this not what I described as mixing (packages build from) ports with standard packages? If not what am I missing?
I'm confused now. I thought your thesis was that you can mix ports and packages? I'm trying to prove the opposite.
I originally asked about the specific statement because I have been building all the packages from ports, exactly to avoid the scenario described in your message above. Yes, it results in a lot of rebuilding if it is needed.
This is the pain of the ports user, but at least you wind up with a consistent set of software installed. My main criticism of this approach is above, under "snowflake server".
Again, the fact that I do not understand does not mean that it does not work well for you.
It sounds like what you're doing (installing everything from ports) works for you, and it's not inherently dangerous. The only reason I don't do that is that I like to have a small, reproducible set of steps to (re)-install and configure any Freebsd machine I maintain. You can come up with a set of steps for a ports-based approach, but it won't be small.
 
Hi Jose,

it is interesting how we could misunderstand each other.

It has never been my thesis that mixing (packages build from) ports with standard packages is a good idea. In fact, if you look at my initial post (104), I was questioning how can you use both your built packages and the standard packages because that would constitute mixing (packages build from) ports with standard packages, which many posters, you among them, demonstrated to be a bad idea.

Based no the foregoing, I still do not understand how you can use both repositories and not have the problem you described.

The only reason I don't do that is that I like to have a small, reproducible set of steps to (re)-install and configure any Freebsd machine I maintain. You can come up with a set of steps for a ports-based approach, but it won't be small.

This is another point I do not understand. You are building with Poudriere, I am building with synth. So, what is the difference you are alluding to?

BTW, we have moved quite away of the original topic.

Kindest regards,

M
 
It has never been my thesis that mixing (packages build from) ports with standard packages is a good idea. In fact, if you look at my initial post (104), I was questioning how can you use both your built packages and the standard packages because that would constitute mixing (packages build from) ports with standard packages, which many posters, you among them, demonstrated to be a bad idea.
But I'm not mixing packages and ports. I'm mixing packages and packages. This can be bad too, if you build one-off packages, but that's not what I'm doing. I'm building consistent package sets.
Based no the foregoing, I still do not understand how you can use both repositories and not have the problem you described.
Because my local package repo has higher priority than the standard packages. My local packages will be installed preferentially over the standard ones. This ensures consistency.
This is another point I do not understand. You are building with Poudriere, I am building with synth. So, what is the difference you are alluding to?
Again, I don't know how Synth works, and I'm not going to find out. Assuming Synth allows you to build ad-hoc package sets based on the current state of the host system, the difference is that Poudriere always builds packages on what is basically a clean install of the build host. In my assumed Synth case, the state of the build host changes over time, which means that builds done today may be different from builds done tomorrow.

I'm also assuming Synth handles so-called "reverse" dependencies correctly. This is not always the case for this kind of tool.
BTW, we have moved quite away of the original topic.
I think we're on topic, but am willing to start a new one if you'd like.
 
Hi Jose,

But I'm not mixing packages and ports. I'm mixing packages and packages. This can be bad too, if you build one-off packages, but that's not what I'm doing. I'm building consistent package sets.
If you re-read my quote, you will notice that I am carefully specifying mixing (packages build from) ports with standard packages.

Because my local package repo has higher priority than the standard packages. My local packages will be installed preferentially over the standard ones. This ensures consistency.
This is exactly the crux of my misunderstanding. Suppose that you issue a command to install a package. The local package repository is accessed first, but the requested package is not there. Thus the standard package repository is accessed next, the package is found and installed.

This may or may not result in the disatrous scenario that you described.

As far as I understand it, synth is using the same build logic as Poudriere and. likewise, builds packages in a clean environment. I do think that it does a great job in handling dependencies because since 2016 when I started using it, I never had a problem. Of course, as mentioned above, unlike you I do not use a standard package repository, I build everything by synth.

Kindest regards,

M
 
This is exactly the crux of my misunderstanding. Suppose that you issue a command to install a package. The local package repository is accessed first, but the requested package is not there. Thus the standard package repository is accessed next, the package is found and installed.
That cannot happen. Any package that is in the dependency graph of a package for which I want custom options will be built by Poudriere, and therefore will exist in my local package repo.
As far as I understand it, synth is using the same build logic as Poudriere and. likewise, builds packages in a clean environment. I do think that it does a great job in handling dependencies because since 2016 when I started using it, I never had a problem. Of course, as mentioned above, unlike you I do not use a standard package repository, I build everything by synth.
All I can find on the subject is:
(Synth) internally creates jail-like environments for each builder...
https://github.com/jrmarino/synth

Contrast with:
Because Poudriere performs its building in a clean jail(8) environment and uses zfs(8) features, it has several advantages over traditional testing on the host system:

  • No pollution of the host environment: No leftover files, no accidental removals, no changes of existing configuration files.
  • ...

Even if you could convince me that Synth is objectively superior to Poudriere, I still wouldn't use it. Why? Because Poudriere is used to build the standard Freebsd packages. I want to diverge as little as possible to minimize problems. Also, because so many people have a stake in having Poudriere work, I'm more likely to find solutions to any problems I run into, and have some confidence that bugs will be found and fixed quickly.

I looked at the Synth source, and I have to admit the Ada is beautiful. I'm not a fan of shell, and Poudriere is definitely an ugly duckling here, but I'm still going to use it.
 
Hi Jose,

thank you for your patience, but I still do not understand.
mefizto said:


This is exactly the crux of my misunderstanding. Suppose that you issue a command to install a package. The local package repository is accessed first, but the requested package is not there. Thus the standard package repository is accessed next, the package is found and installed.
That cannot happen. Any package that is in the dependency graph of a package for which I want custom options will be built by Poudriere, and therefore will exist in my local package repo.
As understand, all the packages that may depend on the package build from ports will also be built. As I am reading your reply, you will install only packages from your local repository. Why then do you have the standard repository?

What am I missing?

Even if you could convince me that Synth is objectively superior to Poudriere, . . .
I respectfully submit that none of may statement can be interpreted in this manner. Lest there be any misunderstanding, I am not trying to convince anybody about anything.

Kindest regards,
M
 
As understand, all the packages that may depend on the package build from ports will also be built. As I am reading your reply, you will install only packages from your local repository. Why then do you have the standard repository?
I use the standard repo packages when the standard options they were compiled with work for me.
I respectfully submit that none of may statement can be interpreted in this manner. Lest there be any misunderstanding, I am not trying to convince anybody about anything.
I didn't mean to imply that you were.
 
Hi Jose,

I use the standard repo packages when the standard options they were compiled with work for me.
Ah, but how do you know that? In post #109, you have described a disastrous scenario caused by a "package baz" installed from the standard repository. Now, admittedly, you have added a disclaimer "without thinking".

Maybe, this is what I am missing, that one has to know enough to understand whether or not such a package may cause a problem?

Kindest regards,

M
 
Ah, but how do you know that? In post #109, you have described a disastrous scenario caused by a "package baz" installed from the standard repository. Now, admittedly, you have added a disclaimer "without thinking".
You are right, my approach would not protect me in that case. The standard package would be installed, and it would not work with Mysql. However, I can never get a package bar that doesn't support Postgresql because that package is in the dependency graph for foo which is in my set of packages I build.
 
Back
Top