Ports manpage: Need for clarification of make target function and behavior

Snurg · Jan 14, 2021

Reading the ports(7) manpage, I am having difficulties to find out how I can do an unattended scripted install.

For example, when I do make extract, I get a build options dialog(1), and after exiting the dialog, the port was apparently built.
What I expected, however, was make just extracting the archived files, like with any packer.
Nothing more. In particular, no passing through the subsequent steps.

Side note, what worried me somewhat also was that I didn't see the screen notices I expected to see from the 'patch' step. I don't know whether it is intended behavior If the 'patch' step apparently gets skipped if one does a make fetch; make extract; make install sequence?!?

Anyway, what I need is to find how I can do a completely non-interactive scripted install of Apache, about this way:

 make fetch

make extract [without interactive dialog]

make patch

<some source file editing, either via perl or sed>

<edit the make configuration (change from '-suexec' to '+suexec' or the like) via script>

make build

make install

I also failed to find out where the build configuration I entered in the dialog I got after make extract is stored.

Thank you for any assistance

SirDice · Jan 14, 2021

Snurg said:
Side note, what worried me somewhat also was that I didn't see the screen notices I expected to see from the 'patch' step. I don't know whether it is intended behavior If the 'patch' step apparently gets skipped if one does a make fetch; make extract; make install sequence?!?

The step isn't skipped because it's also part of the install target. In order to 'install' it has to have done 'build', in order for 'build' to complete it has to complete 'configure', 'configure' needs 'patch', 'patch' needs 'extract' etc. So each step is eventually executed. It keeps track of these steps by setting 'flag' files in WRKDIRPREFIX:

Code:

root@molly:/usr/ports/security/sshguard # ll work/.[^.]*
-rw-r--r--  1 root  wheel    0 Jan 14 22:48 work/.build_done.sshguard._usr_local
-rw-r--r--  1 root  wheel    0 Jan 14 22:48 work/.configure_done.sshguard._usr_local
-rw-r--r--  1 root  wheel    0 Jan 14 22:48 work/.extract_done.sshguard._usr_local
-rw-r--r--  1 root  wheel  209 Jan 14 22:48 work/.license-catalog.mk
-rw-r--r--  1 root  wheel   81 Jan 14 22:48 work/.license-report
-rw-r--r--  1 root  wheel    0 Jan 14 22:48 work/.license_done.sshguard._usr_local
-rw-r--r--  1 root  wheel    0 Jan 14 22:48 work/.patch_done.sshguard._usr_local

(I picked security/sshguard because it was easy, did all the steps and didn't require any dependencies)

Snurg said:
I also failed to find out where the build configuration I entered in the dialog I got after make extract is stored.

The configuration options ( make config) are stored in PORT_DBDIR which is /var/db/ports/ by default. The build itself happens in WRKDIRPREFIX, which defaults to the work directory in the port's directory.

As for the specific question to enable SUEXEC on www/apache24, add this to your /etc/make.conf:

Code:

.if ${.CURDIR:M*/www/apache24}
  OPTIONS_SET+=SUEXEC
.endif

Or

Code:

SUEXEC_SET

You can also use it on the command line: make -DWITH_SUEXEC install

Have a look through /usr/ports/Mk/bsd.options.mk.

Snurg · Jan 14, 2021

Wow this answer points at just the right points!
Thank you so much again!

I probably missed the patch messages, as the following dialog quickly overwrote the screen. My mistake then.

Is there a way to prevent make pass/fall through to the next target(s)?
Reason is, I would like to have my script call make fetch; make extract; make patch one-by-one to do these steps non-interactively.
Then I would like to skip the make config step, letting my script do the changes itself, and touching the appropriate 'flag file'.
After that I would like my script to call make build; make install.

This whole thing confuses me so much because, after make fetch make did not pass/fall through to the next target 'extract'.
This was the behaviour I expected the other targets to run, too.

So I feel very confused and stupid, as I do not understand why the other targets seem to pass/fall through.
How can I call a non-interactive make extract; make patch that does just this, without doing 'config', 'build' and 'install' afterwards?

ljboiler · Jan 14, 2021

From list of targets in bsd.port.mk:

Code:

# config                - Configure options for this port (using ${DIALOG}).
#                                 Automatically run prior to extract, patch, configure, build,
#                                 install, and package.

Defining the make variable BATCH turns off the automatic running of the 'config' target, and make will run using the
default options for the port). The 'config' is also skipped if options have already been saved for the port.

The following should do what you're looking for:

Makefile:

make -DBATCH extract
make -DBATCH patch

Snurg · Jan 14, 2021

Somehow I missed the part about suexec the first time I read your answer...
Another ?‍

Thank you again ?

In particular I am glad to learn I can avoid modifying /etc/make.conf by just invoking with option make -DWITH_SUEXEC install!
Reading /usr/ports/Mk/bsd.options.mk, I see not only WITH, but also things like WITHOUT, _SET, _UNSET and more

So no need to touch things in /etc just to build a port!

I'll test tomorrow whether I just can prepend env ASSUME_ALWAYS_YES=YES and call the apache24 make with -DWITH_SUEXEC -DBATCH, and report back

SirDice · Jan 15, 2021

Snurg said:
I'll test tomorrow whether I just can prepend env ASSUME_ALWAYS_YES=YES

That's for pkg-install(8), it doesn't do anything when building ports, make install never asks if you're sure.

Snurg · Jan 19, 2021

Finding 1:
For some reason the presence of the -j option breaks the make run.
It apparently only works well in the make build step.

Finding 2:
Furthermore, I found out that make extract failed when trying to install the build and library dependencies (see the freshports info).
To me, it seems sensible to install these separately before doing the work on the port.
So it is made sure that packages get installed and "no more ports are being opened".
These build and library dependencies can be easily looked up at freshports.org.

Now only the last step, starting Apache, still fails.

I seem unable to service start or onestart Apache.

Code:

httpd does not exist in /etc/rc.d or the local startup
directories (/usr/local/etc/rc.d), or is not executable
executer: Got nonzero result from 'service httpd onestart' at ./bootie.pl
root@wurst:~ #

When I check, I find apache24_enable="YES" in /etc/rc.conf.
In /usr/local/etc/rc.d I find two scripts, apache24 and htcacheclean.
In /etc/rc.d there is no script of apache.
In /usr/local/etc/apache24 there is the updated httpd.conf.

So I do not understand what is wrong/missing.

Below you can see what happened until the failing last step service httpd onestart.

Code:

zfs snapshot -r wurstpool@bootie_snap_before_install_2021-01-19-16:57:20
env ASSUME_ALWAYS_YES=YES PAGER=cat freebsd-update fetch
[0 1 2]env ASSUME_ALWAYS_YES=YES PAGER=cat freebsd-update install
env ASSUME_ALWAYS_YES=YES pkg update
env ASSUME_ALWAYS_YES=YES pkg upgrade
env ASSUME_ALWAYS_YES=YES pkg install devel/autoconf
env ASSUME_ALWAYS_YES=YES pkg install devel/automake
env ASSUME_ALWAYS_YES=YES pkg install devel/libtool
env ASSUME_ALWAYS_YES=YES pkg install textproc/expat2
env ASSUME_ALWAYS_YES=YES pkg install devel/apr1
env ASSUME_ALWAYS_YES=YES pkg install devel/pcre
env ASSUME_ALWAYS_YES=YES pkg install www/libnghttp2
env ASSUME_ALWAYS_YES=YES pkg install ftp/curl
env ASSUME_ALWAYS_YES=YES pkg install devel/jansson
env ASSUME_ALWAYS_YES=YES pkg install www/libnghttp2
env ASSUME_ALWAYS_YES=YES pkg install converters/libiconv
env ASSUME_ALWAYS_YES=YES pkg install textproc/libxml2
cd /usr/ports/www/apache24/
make -DWITH=SUEXEC -DBATCH fetch
make -DWITH=SUEXEC -DBATCH extract
make -DWITH=SUEXEC -DBATCH patch
_back_up_and_patch_suexec_c_
make -j12 -DWITH=SUEXEC -DBATCH build
make -DWITH=SUEXEC -DBATCH install
env ASSUME_ALWAYS_YES=YES pkg install ap24-mod_perl2
cp /usr/local/etc/apache24/httpd.conf /usr/local/etc/apache24/httpd.conf.orig
_write_httpd_conf_
mkdir -p /cgi-bin
_write_the_helloworld_cgi_
_update_rc_conf_for_apache_
env ASSUME_ALWAYS_YES=YES pkg autoremove
service httpd onestart

Any idea which steps are still missing so Apache can be started using service start or onestart?

SirDice · Jan 19, 2021

Snurg said:
For some reason the presence of the -j option breaks the make run.
It apparently only works well in the make build step.

Most of the other steps need to be run sequentially and can't work concurrently.

Code:

env ASSUME_ALWAYS_YES=YES pkg install devel/autoconf
env ASSUME_ALWAYS_YES=YES pkg install devel/automake
env ASSUME_ALWAYS_YES=YES pkg install devel/libtool
env ASSUME_ALWAYS_YES=YES pkg install textproc/expat2
env ASSUME_ALWAYS_YES=YES pkg install devel/apr1
env ASSUME_ALWAYS_YES=YES pkg install devel/pcre
env ASSUME_ALWAYS_YES=YES pkg install www/libnghttp2
env ASSUME_ALWAYS_YES=YES pkg install ftp/curl
env ASSUME_ALWAYS_YES=YES pkg install devel/jansson
env ASSUME_ALWAYS_YES=YES pkg install www/libnghttp2
env ASSUME_ALWAYS_YES=YES pkg install converters/libiconv
env ASSUME_ALWAYS_YES=YES pkg install textproc/libxml2

You can replace all of this with env ASSUME_ALWAYS_YES= pkg install -y -A $(make -C /usr/ports/www/apache24 missing). That will make it more resilient to changes in the dependency chain. You also want to add the -A (automatic) option so things are cleaned up with a pkg-autoremove(8). No sense in keeping a bunch of build dependencies around or when you decide to remove Apache later on.

Code:

make -DWITH=SUEXEC -DBATCH fetch
{and a bunch of other invocations}

It's WITH_SUEXEC, not WITH=SUEXEC

And to not have to add the variables with every single line, define them once:

Code:

export ASSUME_AWAYS_YES=
export WITH_SUEXEC=
export BATCH=
export PAGER=/bin/cat

The variables just have to exist, they don't need to have anything assigned to them. Except the PAGER variable, that does need a value.

SirDice · Jan 19, 2021

Snurg said:
Code:

mkdir -p /cgi-bin

That will create a cgi-bin directory in your root, probably not what you intended here. For websites create a /usr/local/www/mysite/ for example. Put everything in that.

SirDice · Jan 19, 2021

Snurg said:
_back_up_and_patch_suexec_c_

If you create a proper patch file you can just drop it in the files/ directory of the port. It'll be automatically applied during the patch stage. Then you only have to run make build and make install.

Chapter 4. Slow Porting

Description about creating a FreeBSD Port when the program need some modifications

www.freebsd.org

Snurg · Jan 20, 2021

Thank you very very much, SirDice !!!

Your snippet to find and install the dependencies is great! ?
I instantly added it, it works great

Your thoughts and tips regarding proper clean-up after deinstallation are extremely helpful too ?

Regarding the filepaths (their sources and destinations) all is still variable.
Atm I just have lazy short paths for dry testing. mostly /tmp/...
All file/pathnames are stored in (constant) variables so they can be easily changed.
As soon as CGI basic function is there, clean-up and adaptation to conventions will follow.

Now I am no longer battling with the make process, but with AHxxxxx.
So progress is going on

As soon testing shows all works as intended, I'll make a patchfile.
So no more wild patching. ?
Thank you so much ?

Snurg · Jan 20, 2021

Regarding export, can I really use that?
The "batch" is being processed by a system() call, one for (almost) each line).
I know too little about shell

So I'd like to ask, will the exports remain between the perl system() calls?
This would be very useful tidying the resulting installation "batch script".

SirDice · Jan 20, 2021

Snurg said:
Regarding export, can I really use that?

If it's a POSIX/Bourne shell script, yes, that's how you pass variables to child shells (which is basically what make(1) does).

Snurg said:
The "batch" is being processed by a system() call, one for (almost) each line).

Why do it like that? In that case an export isn't going to work. The variables are passed to child shells (a system(3) call spawns a child shell), the export is executed there but as the system call returns to the parent those environment variables are forgotten. In that case a execve(2) is a better option as you can pass environment variables with it.

Snurg said:
I know too little about shell

You have most of it already right there. Just put all the commands in a file and execute the whole lot with sh <script file>. Or add the #!/bin/sh as the first line, make the file executable and run it directly. Shell scripts don't have to be complex.

POSIX Shell Tutorial

Snurg · Jan 21, 2021

Now that's a coincidence... when searching about shell and export I found and bookmarked that Posix Shell Tutorial

It's great and full of tricks!
But there are a few reasons why for now I stick with Perl, even without execve (which afaik isn't available in Perl).

I think I just have to sweep a rug over the ugly calls with many many environment parameters. ?‍

Maybe just cover with splash screen or progress bar and all is good

[*]

Thank you again, SirDice !

([*]I just ask myself, are there maybe hooks in FreeBSD make system for updating dialog() progress bars while doing builds?)

Ports manpage: Need for clarification of make target function and behavior

Snurg

SirDice

Administrator

Snurg

ljboiler

Snurg

SirDice

Administrator

Snurg

SirDice

Administrator

SirDice

Administrator

SirDice

Administrator

Chapter 4. Slow Porting

Snurg

Snurg

SirDice

Administrator

Snurg