Updating with poudriere

[chrbr]

I have tried ports-mgmt/poudriere because some unexpected issue related to python and curses has crept into my ports system after using ports-mgmt/portmaster quite a long time without having it noted early. I hope I have strictly followed /usr/ports/UPDATING all the time . Now I have a few questions. Today there was an update of graphics/tiff. Update of the ports by poudriere ports -u ... and running poudriere bulk ... caused the update of almost all ports. This is of course the safest way. I have not tested up to now what ports-mgmt/portmaster would do. From my experience I guess it would update the new ports only just to update the relevant libraries. Now come the questions.

1. How to hande advises in /usr/ports/UPDATING? May be the result above has answered the question. All dependencies are always build again. The file can be ignored. Or not?

2. Would the use of devel/ccache minimize the re-build of the untouched dependencies as www/firefox or print/texlive-base just to building without compiling the same lines of code again?

3. Is it a good way of working to separate a few huge ports as www/firefox in a different base or is it very unsafe?

4. I like to say thank you for the documentation. Everything is just as described beside one item with might be super-minor. In my first test I have seen a note that the jail must not be newer than the system. Re-compiling the kernel did cause the note dissappear. Is the note about OS important?

Thank you!
Christoph

 

[kpa]

1. You can and you have to ignore /usr/ports/UPDATING in almost all cases. Poudriere builds a binary repository that can be used with pkg upgrade which can make use of the dependency resolver that can do decisions that ports-mgmt/portmaster could never do. The difference is that when using a binary repository pkg has a set of packages to be upgraded but with portmaster the upgrade is done one port a time without any knowledge of the other ports being upgraded.

2. It is only for speeding up a compilation of a port that has been built previously on source code level. It will never have the knowledge of how ports depend on each other and can not offer any help there. Ports are either compiled completely starting from scratch or not at all.

3. Clarify what you mean?

4. Can you show the uname -a output from the host and the poudriere jail -l output.

 

[chrbr]

Dear kpa,
thank you for your reply.

2. It is only for speeding up a compilation of a port that has been built previously on source code level.

In this case it should help if the cache is big enough to hold everything, if the cache is not cleared after some time. This is my current understanding. The chache must be filled once of course.

3. Clarify what you mean?

I have used the search function and somebody mentioned the option to use separate jails for common ports and other items. In my understanding this could be useful to have the re-builds separated. The common ports are usually small and it is ok to re-build often. Monsters as www/firefox could be triggered only if really necessary by updating the "other jail" only if really necessary.
EDIT: This is not what I have found first but it describes the same question even better: Thread can-i-avoid-duplicating-packages-across-multiple-pkg-repositories.52325. Your answer might indicate that this is not a good idea.

4. Can you show the uname -a output from the host and the poudriere jail -l output.

Below is the upper part error of the first log file.

====>> Building mail/abook
build started at Sun Jan  3 18:36:13 CET 2016
port directory: /usr/ports/mail/abook
building for: FreeBSD pkg.esprimo.local 10.2-STABLE FreeBSD 10.2-STABLE amd64
maintained by: ports@FreeBSD.org
Makefile ident:  $FreeBSD: head/mail/abook/Makefile 356128 2014-06-01 14:39:32Z pawel $
Poudriere version: 3.1.10
Host OSVERSION: 1002502
Jail OSVERSION: 1002504

!!! Jail is newer than host. (Jail: 1002504, Host: 1002502) !!!
!!! This is not supported. !!!
!!! Host kernel must be same or newer than jail. !!!
!!! Expect build failures. !!!

Now it should be ok because I have updated the systems kernel. The warning did not re-appear. Below are the two outputs.

#  poudriere jail -l
JAILNAME VERSION  ARCH  METHOD TIMESTAMP  PATH
10amd64  10.2-STABLE amd64 ftp  2016-01-03 17:03:17 /usr/local/poudriere/jails/10amd64
# uname -a
FreeBSD esprimo.local 10.2-STABLE FreeBSD 10.2-STABLE #4 r293110: Sun Jan  3 19:11:33 CET 2016  root@esprimo.local:/usr/obj/usr/src/sys/ESPRIMO  amd64

 

[chrbr]

I have made a test to verify the effect of devel/ccache. Below are some specs of the system which is no more state of the art.

CPU: Intel(R) Pentium(R) D CPU 3.40GHz (3391.60-MHz K8-class CPU)
Origin="GenuineIntel"  Id=0xf65  Family=0xf  Model=0x6  Stepping=5
real memory  = 2147483648 (2048 MB)
avail memory = 2050654208 (1955 MB)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
ada0: <KINGSTON SV300S37A120G 583ABBF0> ATA8-ACS SATA 3.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 512bytes)

The drive ada0 holds the system and a swap file of 8GB.I have started using a clear cache. The build by poudriere testport -j 10amd64 -p local -z workstation -o www/firefox has been run two times. The difference should be related to the effect of devel/ccache. The first run took 63 minutes, the second run 13 minutes.

The speed up for www/firefox is quite high. I post the results because the port is easy to install and configure. This might encourage others to test and take advantage of devel/ccache.

 

[garry]

I have tried ports-mgmt/poudriere ... Now come the questions.

I build my own package repository using poudriere and it has proved very reliable with FreeBSD 10-stable and 11-current. I don't know if this is necessary or the best way, but the example may help:

To avoid having to build the entire repository in one step, and to make it easier to keep track of which packages I'm supporting, I keep lists of all of the packages I'm building in "piles" such as

• basic.pile
• developer.pile
• gtk-desktop.pile
• kde-desktop.pile
• gnome-desktop.pile
• math.pile
• science.pile

I disable any pile by renaming, e.g. to math.pile.disabled. I try to keep all non-default options in the jail's make.conf to get better visibility of exactly how I'm building packages for that jail. e.g. my current_11-make.conf contains lines like this:

OPTIONS_SET+=     AVX
OPTIONS_SET+=     OPTIMIZED_CFLAGS
OPTIONS_SET+=     PYTHON
OPTIONS_SET+=     SSE
OPTIONS_SET+=     X11
OPTIONS_UNSET+=     AVAHI
OPTIONS_UNSET+=     LDAP LDAPS
OPTIONS_UNSET+=     MYSQL
OPTIONS_UNSET+=     PAM
OPTIONS_UNSET+=     PHP
OPTIONS_UNSET+=     PULSEAUDIO
audio_ardour_SET=
audio_aumix_SET+=GTK2
audio_openal-soft_SET=
editors_emacs24_SET+=OSS
editors_emacs24_UNSET+=ALSA

When I've set the options for the jail, pile by pile using poudriere options -j current_11 -p ports -f basic.pile, I cat the enabled piles together in one big PILE and bulk build it:

sed -e '/^#/d'  *pile >PILE
poudriere bulk -j current_11 -p ports -f PILE

When the repository finally contains every supported package then, and only then, I update local machines from that repository. I re-install ALL packages. I don't rely on pkg(8) to be smart about my style of rolling updates since doing a complete re-install takes so little time:
pkg upgrade -f

You can build one pile at a time or even one package at a time

poudriere bulk -j current_11 -p ports -f gnome-desktop.pile
poudriere bulk -j current_11 -p ports www/firefox

but I don't upgrade computers from that repository until all packages are perfectly built and ready to go. Once the jail is well-defined and I've built the repository once I find it relatively simple to upgrade the ports every two weeks.

 

[junovitch@]

1. You can and you have to ignore /usr/ports/UPDATING in almost all cases.
....

Some /usr/ports/UPDATING entries are handled by the solver but some are more along the lines of "Port foo has been updated from 1.x to 2.x and this requires the user to update config files or do X, Y, or Z". So unless you follow the upstream of all the software you install it's still of value to read.

 

[chrbr]

Dear junovitch@,
this was also my understanding from the statement of kpa, especially since a discussion about a change of a default version of some port in a different thread. /usr/ports/UPDATING does not change often. Therefore it does not require much effort to check it if changes occur. In the meantime, I really learned to appreciate the possibility to test things in the jails of ports-mgmt/poudriere and the effect to have a clean system.