Global ports configuration in make.conf in 2023

[philman]

Hello there everyone! Sorry if this thread is a duplicate or violates the rules in any way, I researched as well as I could before posting.

I realize that the FreeBSD way of ports configuration is by means of an Ncurses option selection menu for each package, however I do prefer as much as possible to streamline and generalize my configs, as for example if I never plan on using Dbus for anything but want all ports that support it to be compiled with png support, I think it's logical to specify them globally. I've been using Gentoo as my go to for the last 10 years and although I believe I understand the FreeBSD way, I'd love to configure my ports using one centralized make.conf file to the largest extent possible.

I came across this forum post https://forums.freebsd.org/threads/how-to-manage-options-easily-and-globally.54508/ which 8 years ago discussed what I'm getting at. It mentioned this file from the FreeBSD source code https://cgit.freebsd.org/ports/tree/Mk/bsd.options.desc.mk which lists and describes all possible ports' options you could use under make.conf. I would love to clarify a few things.

1. I've seen in the above mentioned thread and in some other places online, that there are a few keywords that are used for what I'm describing, and they are OPTIONNAME_USE and OPTIONNAME_UNUSE, but also OPTIONNAME_WANT and OPTIONNAME_UNWANT, and PORTNAME_OPTIONNAME_SET and PORTNAME_OPTIONNAME_UNSET. I suppose the last two are used for per-port configuration. What are the differences between the other two?
2. One gentleman (or woman of course) mentions in the forum post I've mentioned that I shouldn't "go crazy with sticking things in your "OPTIONS_UNSET" line in /etc/make.conf; if do, 'll probably end up un-setting options that don't exist for any ports have installed anyway." However I would very much like to do that. It makes a lot of sense to me to set and unset options for ports that I might install in the future. Again my example, I want png and ffmpeg support, but will never want any dbus or gtk support ever, or maybe just in one single specific package.
3. As soon as I jot out all of the ports' options my heart desires, what is the recommended way to go about installing the ports? Do I simply go to the ports folder and "make"? Won't that open up the same ncurses menu? If it won't, will it take in mind all of the changes I made to make.conf? I've heard many different tools mentioned in various posts, such as poudriere, tinderbox, portmaster, etc.. What is the case for each one? Given that my goal is to get as close to the centralized Gentoo way as possible with as little interaction upon ports building as possible, as that would save me copious amounts of time and keep all of my packages in sync configuration-wise, what should I use?

Thank you very much for your patience. I've been using Gentoo for 10-13 years, started right before what I call the "Linux YouTuber explosion." I'm now almost finishing an engineering degree, and I'm continually growing disillusioned with the Linux ecosystem and the direction it's heading. FreeBSD's documentation is incredibly nice, it's one solid OS, and I would love to dive deep into it's internals and maybe contribute in the future, but for the moment this issue prevents me from doing so.

 

[SirDice]

As soon as I jot out all of the ports' options my heart desires, what is the recommended way to go about installing the ports? Do I simply go to the ports folder and "make"?

Use ports-mgmt/poudriere or ports-mgmt/Synth to build packages from those ports. Then distribute or share that package repository to the rest of your systems.

Won't that open up the same ncurses menu?

Yes. But you can switch that off by setting BATCH:

     BATCH             If defined, only operate on a port if it can be
                       installed 100% automatically.

Or run make config or make config-recusive beforehand.

     config         Configure OPTIONS for this port using dialog4ports(1).

     config-recursive     Configure OPTIONS for this port and all its
                          dependencies using dialog4ports(1).

See ports(7).

If it won't, will it take in mind all of the changes I made to make.conf?

# These variables can be used in make.conf to configure options.  They are
# processed in the order listed below, i.e. later variables override the effects
# of previous variables.  Options saved using the options dialog are processed
# right before OPTIONS_SET_FORCE.  When building a port a dialog to configure
# options will only appear if there are new options, i.e. options which have not
# been configured before either using the option dialog in a previous build or
# using the variables below.  You can force the dialog to appear by running
# "make config".
#
# OPTIONS_SET                   - List of options to enable for all ports.
# OPTIONS_UNSET                 - List of options to disable for all ports.
# ${OPTIONS_NAME}_SET           - List of options to enable for a specific port.
# ${OPTIONS_NAME}_UNSET         - List of options to disable for a specific port.
#
# OPTIONS_SET_FORCE             - List of options to enable for all ports.
# OPTIONS_UNSET_FORCE           - List of options to disable for all ports.
# ${OPTIONS_NAME}_SET_FORCE     - List of options to enable for a specific port.
# ${OPTIONS_NAME}_UNSET_FORCE
#                               - List of options to disable for a specific port.

See /usr/ports/Mk/bsd.options.mk

 

[philman]

Thank you very much SirDice, for taking the time to answer. I'd highly appreciate it if you could please clarify a few more things for me:

Regarding Poudriere and Synth, I take it that what they do is create a repository of binary packages for me to then install using "pkg install X" and such, is that correct? That's not exactly what I want, is there no other way for me to compile and install the port directly with just one command, as in, suppose, "poudriere install xorg-server" or "cd /usr/ports/x11/xorg-server; make install;"?

Regarding that snippet of bsd.options.mk that you've cited, does that mean that the USE, UNUSE, WANT and UNWANT keywords that I've mentioned earlier are obsolete, and that the only words I'll be using in me make.conf file are SET, UNSET, SET_FORCE and UNSET_FORCE?

Just to make sure, and I do apologize for reiterating the same point over and over again, what I'm looking for is a sort of setup similar to Gentoo, where I write all of my ports' configurations in make.conf or what have you, and then with one command, "emerge package", I download, configure automatically (accordingly to my make.conf file), compile and install the package. That's the most important thing to me. Could you please tell me directly if I can do just that? (of course, I don't mind cd'ing into /usr/ports and running "make", that's fine too)

 

[SirDice]

Regarding Poudriere and Synth, I take it that what they do is create a repository of binary packages for me to then install using "pkg install X" and such, is that correct?

Correct. The upshot of this is that you have the flexibility of ports and the ease of use from packages. Build once, install many

That's not exactly what I want, is there no other way for me to compile and install the port directly with just one command, as in, suppose, "poudriere install xorg-server" or "cd /usr/ports/x11/xorg-server; make install;"?

You can build ports directly with cd /usr/ports/x11/xorg; make install, sure. But this won't be a "clean" build as options can be picked up 'automagically' if something is already installed, even if you deliberately turn that option off. So the "best" way to build a port is to do this in a "clean room" situation, which is what tools like Poudriere and Synth do.

Regarding that snippet of bsd.options.mk that you've cited, does that mean that the USE, UNUSE, WANT and UNWANT keywords that I've mentioned earlier are obsolete,

No, they're not obsolete. Those variables are intended to be used in a port's Makefile, not in make.conf. A port can, for example, indicate it wants to use Perl by setting USES= perl in it's Makefile (Chapter 17. Using USES Macros).

and that the only words I'll be using in me make.conf file are SET, UNSET, SET_FORCE and UNSET_FORCE?

In make.conf you can use OPTIONS_UNSET{,_FORCE}, OPTIONS_SET{,_FORCE} to set options "globally". Or use www_apache24_SET= SSL or www_apache24_UNSET= SSL for example to set/unset options for a specific port, in this case the SSL option of www/apache24.

This is an example from my systems, that might bring some light onto the subject:

DEFAULT_VERSIONS+= mysql=10.5m samba=4.13 java=11 php=8.1 lua=54 ruby=3.1

OPTIONS_SET+= OPTIMIZED_CFLAGS

devel_cmake_UNSET= DOCS
devel_git_UNSET= GITWEB SEND_EMAIL
devel_py-molecule_SET= DOCKER
editors_neovim_SET= PYNVIM
math_gmp_SET= CPU_OPTS
net_samba413_SET= MDNSRESPONDER
net_samba413_UNSET= AD_DC AVAHI CUPS FRUIT
net_samba416_SET= MDNSRESPONDER
net_samba416_UNSET= AD_DC AVAHI CUPS FRUIT
security_strongswan_SET= CURL GCM KDF SWANCTL VICI
security_sudo_SET= INSULTS
sysutils_cpupdate_SET= INTEL
sysutils_vm-bhyve_SET= BHYVE_FIRMWARE GRUB2_BHYVE
textproc_py-docutils_SET= PYGMENTS

DEFAULT_VERSIONS is usually interesting to set. You can set which version of PHP, Ruby, Java, etc. you want to use. The 'standard' defaults are all set in /usr/ports/Mk/bsd.default-versions.mk and you can override those by setting DEFAULT_VERSIONS in your make.conf.

 

[philman]

You can build ports directly with cd /usr/ports/x11/xorg; make install, sure. But this won't be a "clean" build as options can be picked up 'automagically' if something is already installed, even if you deliberately turn that option off. So the "best" way to build a port is to do this in a "clean room" situation, which is what tools like Poudriere and Synth do.

Or rather, if I want to install for example GzDoom from the ports, and I just so happen to have Timidity installed, then GzDoom will be compiled with Timidity support regardless of whether or not I've disabled it? That is quite frustrating

Correct. The upshot of this is that you have the flexibility of ports and the ease of use from packages. Build once, install many

Can either be configured to do what I've described - configure according to my make.conf, compile and install, just like Gentoo's portage? I couldn't find anything like that in Poudriere's man page, but I did notice the "synth install" option, does it do what I'm seeking to do? i.e. will running synth "install /usr/ports/games/gzdoom" configure gzdoom accordingly to my make.conf file with all of my preferences (i.e. with timidity disabled regardless of whether or not I have it installed), then compile and install it on my system? If that's the case, is it also capable of removing installed ports, removing build dependencies, removing orphaned packages, etc., just like Gentoo's Portage would?

No, they're not obsolete. Those variables are intended to be used in a port's Makefile, not in make.conf. A port can, for example, indicate it wants to use Perl by setting USES= perl in it's Makefile (Chapter 17. Using USES Macros).

Oh OK, got it, thanks for clearing that up!

In make.conf you can use OPTIONS_UNSET{,_FORCE}, OPTIONS_SET{,_FORCE} to set options "globally". Or use www_apache24_SET= SSL or www_apache24_UNSET= SSL for example to set/unset options for a specific port, in this case the SSL option of www/apache24.

So USE/UNUSE and WANT/UNWANT are for port writers just to be used in a ports makefile, while USE/UNUSE are to be used globally. Perfect!

Now on to your example, I see that you've used many options per port, i.e. net_samba413_UNSET= AD_DC AVAHI CUPS FRUIT in order to disable AD_DC, Avahi, Cups and Fruit specifically for net_samba413. Now, say for example I want to enable ffmpeg and SDL2 support and disable Avahi and GTK for ALL ports. To do that could I write:

OPTIONS_SET = FFMPEG SDL2
OPTIONS_UNSET = AVAHI GTK

?

Also, why did you write:

OPTIONS_SET+= OPTIMIZED_CFLAGS

with '+=' instead of '='? Should I use +=? If I use just '=' what'll happen? What'll be ignored?

And I have one more question, can I specify in make.conf the amount of threads I want to be used during compilation? In Gentoo you can specify, for example, MAKEOPTS="-j10" for 10 threads to be used when compiling a port.

I thank you deeply for your pacience SirDice! Thank you very much for helping a noob weed out his confusion

 

[SirDice]

Or rather, if I want to install for example GzDoom from the ports, and I just so happen to have Timidity installed, then GzDoom will be compiled with Timidity support regardless of whether or not I've disabled it?

Yes, don't know if this exact situation will occur for this port but that is the general gist of it. It can and will happen. Nothing the ports system can do about it either, it's due to 'broken' configure or other build scripts the distfile's source uses. It can even lead to broken packages, the port/package has it registered as 'off' (and thus not registered as a dependency) but the actual executables will have the dependency when you run them. So if you uninstall that dependency the port/package system isn't going to complain and simply remove it, leaving you with a broken executable.

Can either be configured to do what I've described - configure according to my make.conf, compile and install, just like Gentoo's portage?

I'm honestly not familiar with Gentoo's portage, so I can't comment on that. But the options to set/unset for building the ports is exactly the same for all port build systems, poudriere, synth, portmaster, or just the plain ports system itself. It works exactly the same everywhere because in the end it all builds ports using the ports system. A lot has been automated, that's the biggest difference.

I couldn't find anything like that in Poudriere's man page, but I did notice the "synth install" option, does it do what I'm seeking to do?

Haven't used synth much, and never actually used Gentoo's portage, but I think it does. I'm more of a poudriere user because it is the tool that's used on FreeBSD's build clusters; https://pkg-status.freebsd.org

If that's the case, is it also capable of removing installed ports, removing build dependencies, removing orphaned packages, etc., just like Gentoo's Portage would?

That's pkg-autoremove(8). In the end a port builds a package, and it's this package you install when you do cd /usr/ports/.../...; make install. A package has registered dependencies, it will pull those in and install them automatically. Those packages are marked as 'automatic'. Any package that's 'automatic' but doesn't have anything depending on it any more will get removed with pkg-autoremove(8).

Now, say for example I want to enable ffmpeg and SDL2 support and disable Avahi and GTK for ALL ports. To do that could I write:

OPTIONS_SET = FFMPEG SDL2
OPTIONS_UNSET = AVAHI GTK

?

Yes. Exactly. Many of those OPTION 'knobs' are fairly standardized. But that might not always be the case.

with '+=' instead of '='?

I used += to add to any existing OPTIONS_SET variable. Remember, it's a variable, only the last assignment is stored.

OPTIONS_SET= FOO
OPTIONS_SET= BAR

Would result in OPTIONS_SET being "BAR".

OPTIONS_SET= FOO
OPTIONS_SET+= BAR

Will result in OPTIONS_SET be set to "FOO BAR"

 

[philman]

Yes, don't know if this exact situation will occur for this port but that is the general gist of it. It can and will happen. Nothing the ports system can do about it either, it's due to 'broken' configure or other build scripts the distfile's source uses. It can even lead to broken packages, the port/package has it registered as 'off' (and thus not registered as a dependency) but the actual executables will have the dependency when you run them. So if you uninstall that dependency the port/package system isn't going to complain and simply remove it, leaving you with a broken executable.

Ah OK got it, so just general undefined spooky behavior.

I'm honestly not familiar with Gentoo's portage, so I can't comment on that. But the options to set/unset for building the ports is exactly the same for all port build systems, poudriere, synth, portmaster, or just the plain ports system itself. It works exactly the same everywhere because in the end it all builds ports using the ports system. A lot has been automated, that's the biggest difference.

Well, with Gentoo, after you configure your make.conf file, you can run the command emerge PACKAGE that will automatically download, extract, configure (based on the make.conf file), compile, and install said package. I was looking for more of a single-command-to-install type of thing. Since you've said you're using Poudriere, does poudriere have one command that automatically configures, builds and installs a port? For example poudriere install games/gzdoom or something like that, which will then configure the port based on my make.conf file, build it and install it?

That's pkg-autoremove(8). In the end a port builds a package, and it's this package you install when you do cd /usr/ports/.../...; make install. A package has registered dependencies, it will pull those in and install them automatically. Those packages are marked as 'automatic'. Any package that's 'automatic' but doesn't have anything depending on it any more will get removed with pkg-autoremove(8).

Ah OK so that's for removing orphaned packages and build dependencies. Just out of curiosity, does this tool work with packages added with pkg or is it just for ports?

Also, I wanted to know if either poudriere or synth (which you said you don't know so well) have a command to uninstall a port that you've compiled and installed using them.

Yes. Exactly. Many of those OPTION 'knobs' are fairly standardized. But that might not always be the case.

Oh perfect then, just like Gentoo!

I used += to add to any existing OPTIONS_SET variable. Remember, it's a variable, only the last assignment is stored.

Ah I see. But just out of curiosity, if you use OPTIONS_SET = VALUES without the '+', what exactly might you delete? Or rather, what will you be overriding? Could you override for example the port's internal use options, or something else unknown entirely?

Thanks again, and I hope my questions aren't too bothersome, I'm trying to deepen my knowledge of the ports system as much as possible.

 

[SirDice]

Ah OK so that's for removing orphaned packages and build dependencies. Just out of curiosity, does this tool work with packages added with pkg or is it just for ports?

A port builds a package, so in the end they're all packages.

Also, I wanted to know if either poudriere or synth (which you said you don't know so well) have a command to uninstall a port that you've compiled and installed using them.

Both are tools to build packages from ports. Package management is done with pkg(8), deleting a package is done with pkg-delete(8).

Ah I see. But just out of curiosity, if you use OPTIONS_SET = VALUES without the '+', what exactly might you delete? Or rather, what will you be overriding? Could you override for example the port's internal use options, or something else unknown entirely?

Poudriere allows you to 'stack' make.conf, so I have a make.conf that applies to everything and a server-make.conf and desktop-make.conf. I have specific settings that are common and settings that are specific to 'server' or 'desktop' builds.

   Create optional make.conf
     You can also specify a global make.conf which will be used for all the
     jails.  Any of the following are allowed and will all be used in the
     order shown:

           /usr/local/etc/poudriere.d/make.conf
           /usr/local/etc/poudriere.d/<setname>-make.conf
           /usr/local/etc/poudriere.d/<tree>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-make.conf
           /usr/local/etc/poudriere.d/<tree>-<setname>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-<tree>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-<setname>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-<tree>-<setname>-make.conf
           /usr/local/etc/poudriere.d/hooks/plugins/<plugin>/make.conf

poudriere(8)

 

[philman]

A port builds a package, so in the end they're all packages.

Both are tools to build packages from ports. Package management is done with pkg(8), deleting a package is done with pkg-delete(8).

Well, I don't know, having to compile all ports into package repositories only to later install them again using pkg honestly seems redundant for my use case. I have no plans of compiling for multiple devices or environments. I just want to compile ports and immediately install them unto the same computer.

With Gentoo, for example, I have the same make.conf file, in which I would write USE flags, which are the same as OPTIONS_SET in FreeBSD. Then, I would use emerge, its package manager. I could run, for example, emerge gzdoom, for emerge to download the source code for GzDoom, configure it automatically according to the USE flags I wrote in make.conf, build it and install it on my computer. All automatically with just one single program. If I wanted to then delete gzdoom, I could write emerge -c gzdoom.

This is the sort of setup that I'm seeking - just one program that takes care of port installations in my own personal computer, without compiling all programs into a repository or anything like that. I'm basically looking for something like FreeBSD's pkg, except that instead of downloading prebuilt binary packages from a binary repo, it configures a port from the ports tree based on what's written in make.conf, builds the port and installs it.

Is there any tool like this at all, or any way to achieve this? Are you familiar with portsmaster? Will I be able to do what I'm looking for with it? If not, is there any other tool or way at all to achieve this?

I really really want to get into FreeBSD and explore it deeply with all its powerful features, but honestly I just can't see myself neither using a binary based package manager, nor going through an ncurses configuration menu for every new port I want to install, nor compiling all of my ports into some repository on my computer, to then install them again onto the same computer with pkg. I realize that a tool like poudriere and its ability to compile ports with multiple make.conf files is very powerful for people with multiple setups who want complete control and automation and unity for all of their setups, but that's just not my use case.

 

[SirDice]

Well, I don't know, having to compile all ports into package repositories only to later install them again using pkg honestly seems redundant for my use case.

No, that's not what I meant. I was talking about a port in the ports tree (/usr/port). It's what the ports system does, without using any additional tools. Short explanation of the steps involved building and installing a port:

1. Download the distfile(s) if they're not available already ( make fetch)
2. Configure any port options ( make config)
3. Extract the distfiles in a work area ( make extract)
4. Do some FreeBSD specific patching to the source code ( make patch)
5. Run ./configure if that's appropriate ( make configure)
6. Build the source code using whatever tool is appropriate (gmake, meson, etc) ( make; will automatically run all the previous steps and build the code)
7. Install into a 'staging' area
8. Build a package from that staging area (uses pkg-create(8))
9. Install that package on the host ( make install; this, in essence, simply does pkg-add(8))

A package is always created. An installed port is a package. That's what I meant.

 

[Tieks]

I'm basically looking for something like FreeBSD's pkg, except that instead of downloading prebuilt binary packages from a binary repo, it configures a port from the ports tree based on what's written in make.conf, builds the port and installs it.

Command cd /usr/ports/games/gzdoom && make install clean will do that, it does all the steps SirDice just described.
Two ways to remove it: pkg delete gzdoom and cd /usr/ports/games/gzdoom && make deinstall. Configuration settings per port will be stored under /var/db/ports.

 

[philman]

Oh, I see. This is what I had in mind originally - I said I don't mind cd'ing into the port's directory and running make install. The only thing is, I don't want to be presented with the Ncurses menu every time I install a port, rather I want to put lots of OPTIONS_SET and OPTIONS_UNSET statements beforehand in my make.conf file and have make install automatically look at my make.conf file and configure my port accordingly. I'm looking for a "set it and forget it" sort of setup.

You can build ports directly with cd /usr/ports/x11/xorg; make install, sure. But this won't be a "clean" build as options can be picked up 'automagically' if something is already installed, even if you deliberately turn that option off. So the "best" way to build a port is to do this in a "clean room" situation, which is what tools like Poudriere and Synth do.

Here SirDice said that if I do that, I won't get a port perfectly configured accordingly to my make.conf file, rather I'll get some unexpected result, for example a port enabling Xorg support just because I have Xorg installed, even though I've explicitly disabled Xorg support in make.conf.

What I'm basically looking for is a "set and forget" setup like I've had on Gentoo for years. Basically upon installing Gentoo I would write down all of the global port options I want to enable or disable in make.conf, in the form of USE flags. Then, when I go about installing the app, I can simply run emerge packagename, where emerge is the port manager and packagename is the package I want to install, and I never have to deal with port configuration again. Of course I can later make changes to make.conf or make a separate configuration for one specific port that needs something that I want globally disabled otherwise, but generally I can write a make.conf file once with lots of USE flags or OPTIONS_SET values, and then never have to manually configure ports myself.

I don't care which program I use, I don't mind cd'ing into the /usr/ports/ directory either, but I just want to be able to never have to manually deal with ports configuration. I want to write all of my OPTIONS_SET and OPTIONS_UNSET values in make.conf once, and never worry about them again, and if I ever want to change my ports configuration, I want to be able to only change make.conf, and nothing else, no ncurses port configuration menu or anything of the like. Everything globally configured and stored in make.conf.

Is this conceivable in FreeBSD, in any way?

 

[SirDice]

Here SirDice said that if I do that, I won't get a port perfectly configured accordingly to my make.conf file, rather I'll get some unexpected result, for example a port enabling Xorg support just because I have Xorg installed, even though I've explicitly disabled Xorg support in make.conf.

Yes. This doesn't always happen. Lots of ports are reasonably well behaved. It's a problem that can unexpectedly pop up without noticing.

to never have to manually deal with ports configuration.

I would suggest ports-mgmt/portmaster and a good /etc/make.conf. Portmaster will take away some of the 'tediousness' that comes with updating what you have installed. Give that a shot. Build away. Honestly. You'll get the hang of it.

 

[Tieks]

Everything globally configured and stored in make.conf.

That is what I tried to achieve some 10 years ago and I am stil sort of working on it...

The ncurses menu will not show up if EITHER all the ports options are in make.conf OR if its options are available in a file /var/db/ports/portname/options. To gather all the options in make.conf you have to check all the makefiles or the ncurses menus at least once to find them. Then either type or copy the options to make.conf or (in the ncurses menu) set the options and simply press Enter to write them to /var/db/ports/portname/options. The latter worked much faster for me. So I have a /etc/make.conf with a number of mostly global options as well as a number of entries under /var/db/ports. The end result is similar to what you are looking for, I have a script with ports that will be build and installed without the ncurses menu popping up. I just have to back up both make.conf and /var/db/ports.

And a script to convert the entries under /var/db/ports to make.conf is still on my bucket list. And will most likely remain there for another 10 years.