Is it possible to get the ports tree that pkg uses to build binaries, as opposed to the current ports tree?

H

hxfischer

I do hope this questions hasn't been asked before, but I've seen no indication that is has.

Here is my dilemma: I have two packages (google-authenticator and nginx) where I literally need to change only 5 lines of code in the source before compiling. Otherwise, binary packages work very well for my needs.

Initially, I decided to go the route of using ports only, as it is not considered good practice to mix packages and ports. It worked nicely when building most ports, but when I had to install shellcheck it literally took 12+ hours to compile. Not to mention, I couldn't get it to build on a VPS with 1G of RAM, so I have had to upgrade my VPS to have 2G of memory for the sole purpose of compiling a single port, which dollar-wise would add up very quickly if/when I need to deploy many VPSs in production.

To my understanding, while pkg uses the ports tree to build packages, it is always using an older snapshot of the ports that is not aligned with the current ports tree (which makes mixing ports and packages a bad idea as it can cause dependency hell). Is it possible to get the ports tree that pkg is using to build packages, as opposed to the current ports tree?

If it is not possible to do so, is there any other way I might safely mix packages with ports? It just seems like a terrible waste of resources to build every single package from the ports tree when I am literally changing a couple of lines of code in a single package or two before compiling. I am aware of poudriere, and it sounds great (and I also intend to learn it), but again, it just seems like overkill to build my own repository when I am making such minuscule modifications to one or two packages.
 
If ports is under source control you should theoretically be able to rollback any port application to a previous state. Is it gonna be easy. Hell no.
 
hxfischer said:
To my understanding, while pkg uses the ports tree to build packages, it is always using an older snapshot of the ports that is not aligned with the current ports tree (which makes mixing ports and packages a bad idea as it can cause dependency hell). Is it possible to get the ports tree that pkg is using to build packages, as opposed to the current ports tree?
Click to expand...
The current ports tree is the ports tree from which the packages are built. That is, the latest package repositories are built from it. You may be slightly confused regarding the quarterly packages (which is the default on all recent releases). Those are built from a ports tree that's more or less frozen in time at regular intervals (every 3 months) and only get security or stability updates. You can find those under /ports/branches/: https://svnweb.freebsd.org/ports/branches/

It is perfectly acceptable to get your ports tree from there: svnlite co https://svn.freebsd.org/ports/branches/2018Q3/ /usr/ports
Just keep in mind it changes every three months. So for the next cycle you would need to switch to 2018Q4. While Q3 is active you can keep it updated using a regular svnlite update.

This is fine for a single machine but if you really need to deploy to many other machines I definitely recommend researching how to set up your own repository. Which, if you want, could track that same quarterly branch, or latest, or even your own snapshot of the ports tree. But besides those you also get to control which default versions to use, what flavors, which options, etc. And, best of all, it allows you to assure you have the exact same packages installed everywhere.
 
hxfischer said:
Initially, I decided to go the route of using ports only, as it is not considered good practice to mix packages and ports. It worked nicely when building most ports, but when I had to install shellcheck it literally took 12+ hours to compile. Not to mention, I couldn't get it to build on a VPS with 1G of RAM, so I have had to upgrade my VPS to have 2G of memory for the sole purpose of compiling a single port, which dollar-wise would add up very quickly if/when I need to deploy many VPSs in production.
Click to expand...

I'm not an admin, but the good practice is supposed to be about the custom package repo with dedicated building and staging environments. Building and testing packages directly on each production server is definitely not recommended.

hxfischer said:
To my understanding, while pkg uses the ports tree to build packages, it is always using an older snapshot of the ports that is not aligned with the current ports tree (which makes mixing ports and packages a bad idea as it can cause dependency hell). Is it possible to get the ports tree that pkg is using to build packages, as opposed to the current ports tree?
Click to expand...

Pkg does not build packages. The latest official repo is built with Poudriere from the latest revision of the ports tree. Quarterly is branched from the latest on a quarterly basis.

hxfischer said:
If it is not possible to do so, is there any other way I might safely mix packages with ports?
Click to expand...

The main issue with mixing packages and ports is that pkg has limited ability to manage dependencies between packages downloaded from a repository and packages built locally. It doesn't know how to rebuild local packages, it can override locally built packages with packages from a repository, etc. Of course, the default advice on this forum is geared toward servers, which means zero tolerance towards unplanned downtime policy. If you are just running FreeBSD server/desktop as a hobby project, you might as well mix anything with anything.
 
FWIW, FreeBSD is a hobby for me. I used to have a server I ran professionally (among a bunch of IBM AIX servers) but for about 15 years, it's just a hobby.

I generally avoid mixing ports and packages. As it sits now, I need drm-next-kmod for 2 of my 3 FreeBSD systems. I don't want to compile everything (especially for the laptop). So, I mix. I do so as follows:

On the build system, a cron job updates the ports tree once a week (Friday night). I upgrade my packages the next day (Saturday or Sunday). I then compile the one port I need. I put the compiled port into my own repository (it is really easy to set-up your own package repository). My package then gets updated along with all the other packages.

This is by no means perfect. I have to watch the compile as it may decide to compile something that I already have installed as a package. In such a case I need to figure out what to do (wait for the package or let the port compile).

Of course, if I were running FreeBSD servers professionally, I wouldn't get packages from the "latest" or "quarterly" repository. I would get them from my own repository after they had been tested in QA and confirmed to work as expected.
 
SirDice said:
This is fine for a single machine but if you really need to deploy to many other machines I definitely recommend researching how to set up your own repository. Which, if you want, could track that same quarterly branch, or latest, or even your own snapshot of the ports tree. But besides those you also get to control which default versions to use, what flavors, which options, etc. And, best of all, it allows you to assure you have the exact same packages installed everywhere.
Click to expand...

Thank you very much for taking the time to reply, you have cleared up a lot of the confusion I had about pkg and ports. Is there any chance you know how quickly the FreeBSD team pushes new packages after a revision is made to the quarterly ports tree?

As an aside, when I took a look at the link you gave me for the branches, it looks like revisions are still being applied to 2018Q1 and 2018Q2. I guess I'm just a bit confused because you said that they only update the latest working quarter.
 
hxfischer said:
how quickly the FreeBSD team pushes new packages after a revision is made to the quarterly ports tree?
Click to expand...
There is no revision made to quarterly. Bugs and all. You use the same repository for 3 months. It also has less churn. But then every quarter you have all the churn at once when all the packages are updated, versus latest build where only a handful a week are updated. If you want the most secure then use latest.

Just to add to this excellent body of work I would like to add that the Ports tree is the freshest. FreeBSD has Package Building Clusters that build packages and push them out to mirrors. So There is a couple of days lag between Ports and Latest Packages.
 
Ya, that might not be true. I thought twice about it after I wrote it. I do not think they get updates.

junovitch@ said:
The quarterly branch should get security fixes right after the head branch. That's part of the goal to get critical security fixes along with stability. If you ever see something fixed in the head branch but not the quarterly then please bring it up
Click to expand...
 
You must log in or register to reply here.
Back
Top