How to mix port and packages with poudriere?

Nick-6

Nick-6

I'm running FreeBSD-12.2-RELEASE on VirtualBox (RAM - 5000 MiB , Processor - 1, SSD - 18 GiB) hosted by Debian Buster. Almost all of the available binary packages in FreeBSD satisfy my need. I just need to build only one package with custom options which is ffmpeg. I won't remove it's default options, I'll just add some extra options, namely lame, sdl and fdk-aac . Since mixing port and packages is not recommended and poudriere is suggested for this purpose. I followed a guide from DigitalOcean. Please see the my questions below.

(I) Updating system
freebsd-update fetch install
portsnap fetch extract

(II) Installing required packages
pkg install portmaster poudriere nginx
rehash

(III) Creating SSL cert and key
mkdir -p /usr/local/etc/ssl/{keys,certs}
chmod 0600 /usr/local/etc/ssl/keys
openssl genrsa -out /usr/local/etc/ssl/keys/poudriere.key 4096
openssl rsa -in /usr/local/etc/ssl/keys/poudriere.key -pubout -out /usr/local/etc/ssl/certs/poudriere.cert

(IV) Configuring Poudriere
mkdir /usr/ports/distfiles
$EDITOR /usr/local/etc/poudriere.conf
NO_ZFS=yes #uncomment for UFS
FREEBSD_HOST=https://download.freebsd.org
DISTFILES_CACHE=/usr/ports/distfiles
POUDRIERE_DATA=${BASEFS}/data #uncomment
CHECK_CHANGED_OPTIONS=verbose #uncomment
CHECK_CHANGED_DEPS=yes #uncomment
PKG_REPO_SIGNING_KEY=/usr/local/etc/ssl/keys/poudriere.key #edit path
URL_BASE=http://localhost/

(V) Creating Build Environment
poudriere jail -c -j freebsd_12-2x64 -v 12.2-RELEASE #it'll take a while, 11G is enough
poudriere ports -c -p HEAD

(VI) Creating Port Building List and setting Port options
$EDITOR /usr/local/etc/poudriere.d/port-list
multimedia/ffmpeg # I add only one package

$EDITOR /usr/local/etc/poudriere.d/freebsd_12-2x64-make.conf
OPTIONS_UNSET+= DOCS NLS TEST EXAMPLES #adding things I don't need

poudriere options -c -j freebsd_12-2x64 -p HEAD -f /usr/local/etc/poudriere.d/port-list

(VII) Building the Ports
poudriere jail -u -j freebsd_12-2x64 #updating jail
poudriere ports -u -p HEAD #updating port tree
sudo poudriere bulk -j freebsd_12-2x64 -p HEAD -f /usr/local/etc/poudriere.d/port-list #it took about 15hrs and 18GB of space
Click to expand...
Questions:
1. Since I'm setting up for Desktop use, can I omit strike-outed steps (i.e. creating SSL cert and key)?
Deleting them don't break anything for me. [self-test]
2. Is checking the port building process with Ctrl + T get the same result as checking from browser? If so, can I omit installing nginx and related setups?
Removing nginx don't break anything for me. [self-test]
3. How to set the poudriere.conf file?
poudriere: {
url: "file:///usr/local/poudriere/data/packages/freebsd_12-2x64-PORTS",
mirror_type: "srv",
signature_type: NONE,
enabled: yes,
priority: 1
}
If priority is set, pkg install will install package from poudriere repo other than FreeBSD repo. If priority is not set (i.e, commenting or deleting that line), pkg install will install from poudriere repo only if the package is not available in FreeBSD repo [self-test].
#When priority is set to 1
pkg install mpv
New packages to be INSTALLED:
fdk-aac: 2.0.1 [poudriere]
ffmpeg: 4.3.1_3,1 [poudriere] #also available in FreeBSD repo
libdvdnav: 6.1.0 [FreeBSD]
libdvdread: 6.1.0 [FreeBSD]
libplacebo: 2.72.0 [FreeBSD]
lua52: 5.2.4 [poudriere]
mpv: 0.32.0_5,1 [FreeBSD]
opus: 1.3.1 [FreeBSD]
shaderc: 2020.0 [FreeBSD]
uchardet: 0.0.7 [FreeBSD]
vulkan-loader: 1.2.135.0 [FreeBSD]

Installed packages to be REINSTALLED:
pkg-1.15.10 [poudriere] (options changed)
===========
#When priority is not set
pkg install mpv
New packages to be INSTALLED:
fdk-aac: 2.0.1 [poudriere] #only available poudriere repo
ffmpeg: 4.3.1_3,1 [FreeBSD] #also available in poudirere repo
libdvdnav: 6.1.0 [FreeBSD]
libdvdread: 6.1.0 [FreeBSD]
libplacebo: 2.72.0 [FreeBSD]
lua52: 5.2.4 [FreeBSD]
mpv: 0.32.0_5,1 [FreeBSD]
opus: 1.3.1 [FreeBSD]
shaderc: 2020.0 [FreeBSD]
uchardet: 0.0.7 [FreeBSD]
vulkan-loader: 1.2.135.0 [FreeBSD]

Number of packages to be installed: 11

4. I want to install other packages from pkg except poudriere ports (ffmpeg in my case). What do I have to do after this step (VII)Building the Ports ?
Select priority (see answer of question #3) and done.
5. After the process, which files are safe to delete and which files should be kept for further usage?

Other Questions:
Since the process took about 15 hours and fetch and build about 180 packages and need the total of about 18 GiB storage (I've installed pkg install xfce xorg), I feel a little bit overkill for just a single package ( ffmpeg in my case).
5. Is that normal (15hrs, 180 packages, 18GiB)? Answered
6. What are the recommended/official methods for mixing port and packages? Answered
7. I've read that Synth is more suitable for my purpose. So, I've installed it and read man synth. Since I'm a newbie, I need step-by-step setup guide for Synth. Where can I find it?
8. The last one. A noob's thoughts. If FreeBSD could provide another port system which has the same version as binary packages, will it prevent the conflicts? Since people are trying to mix port and package just because of one or two packages, can't we have one extra package for each package with full options enable (i.e. ffmpeg-full )?

Thanks in advance.
 
Last edited:
Nick-6 said:
Is that normal (15hrs, 180 packages, 18GiB)?
Click to expand...
Depends on the packages. Chromium alone often takes 8 hours to complete on my build server. But if those 180 packages are all Ruby gems for example, it would probably all be done in less than an hour.

Nick-6 said:
What are the recommended/official methods for mixing port and packages?
Click to expand...
Not doing that, that's the official/recommended strategy.
 
Nick-6 said:
What are the recommended/official methods for mixing port and packages?
Click to expand...

To stay save i install the build, runtime etc... dependencies from the packages and then build the port. After building and installing the port, i remove the build dependencies again because they are useless then.

The only thing you should not do is: Removing options from already installed packages or options that are already selected because that could be dependencies for other packages.
 
I mix quarterly packages with quarterly ports build with poudriere , only a handfull and just for functionality like sndio.
Fyi, compiling iridium-browser with poudriere takes 36 hours on my P.C. It's a bit too much of the good.
 
While the official strategy is not to do it, some people who know what they’re doing do it anyway. I elaborated on dual mode updating of installed ports and packages in one of my BLog posts - https://obsigna.com/articles/1519780374.html

I employed this strategy with the latest packages and latest ports now for several years and on several FreeBSD machines (servers and desktops) without any major hassle, which I wouldn’t have had anyway, for example the python switch.
 
Addition from me:
a) You don't need portmaster when using poudriere.
b) nginx is recommended to install (albeit you can use Apache too), if you want to point your FreeBSD.conf to your own repository.
In that case you wouldn't be able to install the "regular" binaries. Check out manpage for pkg to point it to a local package.
c) As for the time needed: With only one CPU that time-effort is pretty much normal, since poudriere will use only one jail to build the packages
 
Alexander88207 said:
To stay save i install the build, runtime etc... dependencies from the packages and then build the port. After building and installing the port, i remove the build dependencies again because they are useless then.

The only thing you should not do is: Removing options from already installed packages or options that are already selected because that could be dependencies for other packages.
Click to expand...
Correct me if I'm wrong. I've got the following files and I think if I delete any of the files in /usr/local/poudriere/ , I've to repeat the process all over again when I update ffmpeg. But files in /usr/ports/disfiles may be safe to delete coz it is a cache file.
root@FreeBSD:~ # du -sh /usr/local/poudriere/*
357M /usr/local/poudriere/data
2.6G /usr/local/poudriere/jails
1.1G /usr/local/poudriere/ports
root@FreeBSD:~ # du -sh /usr/ports/distfiles
735M /usr/ports/distfiles
Click to expand...
Zvoni said:
Addition from me:
a) You don't need portmaster when using poudriere.
b) nginx is recommended to install (albeit you can use Apache too), if you want to point your FreeBSD.conf to your own repository.
In that case you wouldn't be able to install the "regular" binaries. Check out manpage for pkg to point it to a local package.
c) As for the time needed: With only one CPU that time-effort is pretty much normal, since poudriere will use only one jail to build the packages
Click to expand...
a) I removed the portmaster.
b) I removed nginx too. It don't break anything. And I don't have FreeBSD.conf in that location. (I updated my question.)
 
Nick-6 said:
6. What are the recommended/official methods for mixing port and packages?
Click to expand...
I use three repositories:
FreeBSD - for binary packages with default options
Poudriere-quarterly - for packages with custom options
Poudriere-head - for security-patched packages that haven't made it to the quarterly ports tree

My repos/local.conf:
Bash: 
# awk '/Free|Poudriere-|^\}|priority/' /usr/local/etc/pkg/repos/local.conf
FreeBSD: {
        priority: 15
}
Poudriere-quarterly: {
        priority: 10
}
Poudriere-head: {
        priority: 5
}

I lock any packages built from Poudriere-quarterly to avoid newer FreeBSD versions (with default options) from removing my custom options, and update in this order:
pkg upgrade
pkg unlock -y <packages>
pkg upgrade -r Poudriere-quarterly <packages>
pkg lock -y <packages>

That gets tedious quickly, so I use a Python script to handle the upgrades of locked packages.

To speed package building, I disable building documents on all locked packages, but to avoid dependencies conflicts I only modify necessary options, and don't recurse through all Poudriere build- and runtime-dependency packages.

This allows me to get what we all want: the speed of binary-package upgrades, customizable packages, and security.
 
You must log in or register to reply here.
Back
Top