Can someone explain to me whats going on with pkg?

[mer]

Is there an explanation of the pipeline from Poudriere to pkg.FreeBSD.org somewhere? The bulk build has finished and `pkg update` still isn't picking up any updates on my side. Same with `pkg update -f` - I'm getting zero results for `pkg search gopls`, despite it succeeding in the previous bulk build. I'd just like to understand how all this is connected; it's not intuitive as a newcomer.

Very roughly/high level:
Start with source code (git repos/branches/tags)
Then we have build servers
Then we jails on the build servers, a "jail" represents an architecture to build for
Poudriere is how the source tree gets built, so a jail runs a "Poudriere" job
Poudriere finishes building, end result is a bunch of binary packages.
These binary packages need to get pushed out to the public package repos before your local machine can see them.
There are a number of public package repos, they do not all update "instantly" so once everything builds it takes a finite amount of time for them to get completely updated.

 

[kota905]

For the quarterly repo there is still an issue with full fat gnome being wiped out because of gnome-chess. gnome-lite, kde, xfce, cinnamon, & mate all built fine. There is 1 package still building that will probably take another 6.5 hours to complete.
For the latest repo kde failed due to qt6-webengine-6.8.3_1. gnome, gnome-lite, xfce, cinnamon, & mate all built fine. A new build has been kicked off so there's hope for a quick fix on kde with latest now that maintainers can focus on port specific issues instead of all the go related failures. https://pkg-status.freebsd.org/beefy22/build.html?mastername=142amd64-default&build=e3d1564d6c13

I am new to freeBSD and have no idea how https://pkg-status.freebsd.org/ works.
Can you please explain how to read data ? There is a bunch of things, bunch of servers running. It’s difficult for a novice to understand what all of them are doing at this moment.

For instance, I can go here : https://pkg-status.freebsd.org/beefy22/
I can see that the last build (the last for now at least) is : 38f68942469a
From there, how can I see which package failed to build and what is the reason of the failure ?
Like, let’s say KDE failed to build and I want to see why ?

Also, I have to wait and check the next build and see if everything work or I can find this information before the process is done ?

My request may sound obvious for lot of you, but as I said, I am now to this OS and have never dug that deep into the process.

 

[mer]

What I do, others may have different workflows.
I usually scroll down to the Package Builds section, look at the last one for which "ports" (quarterly).
Under the Build column, just to the left of the hash, do a mouse hover. That gets you to the log of the Poudiere Build.
The build log page, scoll down to the failed ports section. I usually set it to "all entries". Then you can look at the list of failed ports. The "log" column is the log of the build, scroll to the bottom and you can see why it failed.
If there are other packages that depend on the failed one, they are skipped. Example is things that depend on webkit2-gtk were skipped when it failed.

If the failure is because of a SIGSEGV that is typically not a "the port needs to be fixed" that is usually environmental.

 

[gotnull]

-CURRENT is usually pretty reliable.

In my mind CURRENT is associated to the development branch which means not fully reliable but what you must be true at some point because I've read this type of comment very often about CURRENT strangely (Netflix also use it if I am not mistaken).
Still it must be a hell of a job to keep it running bug free in production.

But in this case some backward compatibility for 14.x seems to have been broken, which I suppose is hard to spot even for a committer that tests -CURRENT for functionality.

Thank you for the details.

Very roughly/high level:
Start with source code (git repos/branches/tags)
Then we have build servers
Then we jails on the build servers, a "jail" represents an architecture to build for
Poudriere is how the source tree gets built, so a jail runs a "Poudriere" job
Poudriere finishes building, end result is a bunch of binary packages.
These binary packages need to get pushed out to the public package repos before your local machine can see them.
There are a number of public package repos, they do not all update "instantly" so once everything builds it takes a finite amount of time for them to get completely updated.

Simply and nicely explained, thank you.

 

[mer]

Bugs in an OS are often a very hard to reproduce corner case, the kind that seem to be triggered by stress/load.
Building 34724 ports, is a good stress: stress of the jails, stress of file systems, memory/VM stress.
Someone like NetFlix running -CURRENT often has a team dedicated to actually tracking what is being done and integrating/deferring upstream changes based on need.

Of course these problems are not unique to FreeBSD, every bit of software has similar issues.

 

[gotnull]

Someone like NetFlix running -CURRENT often has a team dedicated to actually tracking what is being done and integrating/deferring upstream changes based on need.

Yep that's what I was thinking too, The FreeBSD Foundation certainly doesn't have the same budget to run that kind of operation.

 

[mer]

Companies that base products on Linux typically do the same thing; maintain local repos, sets of packages because sometimes it's quicker to fix it yourself (if you have the skills) and then push patches upstream.

 

[SirDice]

As far as I know Microsoft typically has a large part of their own infrastructure running on beta builds of the next Windows version. It's not uncommon to run your development versions in your own production pipelines. Strict testing procedures will get you quite far, but real world usage will often find some edge-case bugs that were missed during testing.

The biggest problem was this insane cascading effect the recent issue had. The go compiler itself failed to build, and as a consequence everything that required a go compiler failed as well (you can't build a go application if there's no go compiler). Then many non-go ports got taken out because one or more of their dependencies are written in go.

 

[kota905]

What I do, others may have different workflows.
I usually scroll down to the Package Builds section, look at the last one for which "ports" (quarterly).
Under the Build column, just to the left of the hash, do a mouse hover. That gets you to the log of the Poudiere Build.
The build log page, scoll down to the failed ports section. I usually set it to "all entries". Then you can look at the list of failed ports. The "log" column is the log of the build, scroll to the bottom and you can see why it failed.
If there are other packages that depend on the failed one, they are skipped. Example is things that depend on webkit2-gtk were skipped when it failed.

If the failure is because of a SIGSEGV that is typically not a "the port needs to be fixed" that is usually environmental.

Perfect. Is much more clear now.
I can see in "Skipped" tableau that full Gnome DE is skipped because of "gnome-chess-43.0_2".
As KDE is not in this tableau I guess everything is ok now.

Can you clarify one last thing for me ?
More concretely, there is no more error for KDE but when this package will be available in my system, in order to update broken packages ?
I just downloaded and installed the latest RELEASE 14.2 the last week.

Thank you.

(I do not ask like I can't not wait for this update. It's just about learning about the process)

 

[mer]

"When will KDE be available". I'm not sure; that is the part "packages pushed out to package repositories".
I'm waiting on a few things that depended on webkit, so what I do is:
pkg update -f
pkg search gnucash
pkg upgrade -n

pkg update -f make sure the repo metadata is up to date (may not be strictly needed)
pkg search put whatever package name you are waiting on. This goes and searches the package repos for anything with that name in it.
pkg upgrade -n tells you what would be upgraded/deleted/reinstalled without actually doing anything.

pkg autoremove is the command that typically gets everyone in a bad situation. I always run it with -n and look at what it's going to remove and then rerun without the -n and answer y when I'm ready.

Others may have different ideas, but again this is what I do.

 

[gotnull]

Companies that base products on Linux typically do the same thing; maintain local repos, sets of packages because sometimes it's quicker to fix it yourself (if you have the skills) and then push patches upstream.

As far as I know Microsoft typically has a large part of their own infrastructure running on beta builds of the next Windows version. It's not uncommon to run your development versions in your own production pipelines. Strict testing procedures will get you quite far, but real world usage will often find some edge-case bugs that were missed during testing.

The biggest problem was this insane cascading effect the recent issue had. The go compiler itself failed to build, and as a consequence everything that required a go compiler failed as well (you can't build a go application if there's no go compiler). Then many non-go ports got taken out because one or more of their dependencies are written in go.

I didn't expect it to be that much common, I've learned something, thank you guys.

 

[byrnejb]

I cannot wait for the next time somebody tells me that I should 'just keep everything up to date and you will not have problems'. I will point them back here.

 

[mer]

I cannot wait for the next time somebody tells me that I should 'just keep everything up to date and you will not have problems'. I will point them back here.

But don't forget it's the sys admin's responsibility to determine when to apply any updates.

 

[cy@]

Set up your own repository, build once, install many. This is especially beneficial if you have to provide for a bunch of machines. Added benefit, you can set your own default versions for mysql, perl, php, python, ruby, java, etc. And you get to set/unset options for everything. Meaning you can really tailor made the packages for your environment. And, last but not least, you get to decide what to update and what not, and when this happens. With a local repository you get the best of both worlds, the flexibility of ports and the ease of management from packages.

This is exactly what I do here. I run 15-CURRENT here.

My local poudriere repo is based on my local ports git repo. It contains patches I have no intention of pushing to FreeBSD. In addition, I use poudriere-options to set my own options instead of using defaults. This saves a lot of time and CPU when you're building ports for multiple machines for yourself.

You can also build a package outside of poudriere on another machine and simply copy it into the package repo. The next time you run poudriere to build packages it will notice the new .pkg file and inventory it. This is not documented anywhere. It's a little trick you can do when building many packages on your build machine and are impatient for it to finish, letting another machine help building packages.

 

[CeXP1917]

But don't forget it's the sys admin's responsibility to determine when to apply any updates.

Sometimes sysadmin's responsibility is installing software in sutuation when building from src not acceptable.

 

[mer]

Sometimes sysadmin's responsibility is installing software in sutuation when building from src not acceptable.

I completely agree; but in that case delaying the update until the upstream has built everything needed is the wise course of action.

I also think that in a business environment, maintaining your own package environment (poudiere build) is a wise choice. "You" (corporate) controls what updates to pull in based on your needs, build them all locally, distribute locally. It may also provide some good regulatory (government interference ) documentation:
"We saw this CVE issued, we investigated, we pulled upstream fixes in locally and deployed to our systems before anything else was affected or released".

That can earn the business extra cookies (peanut butter my preference) for being proactive.

 

[g++]

I'm a FreeBSD noob, but I do have some hobby-time on the side and would like to know what can be done to mitigate this situation in the future for the public repos. If it's something I can help with, great. If not, awareness is the first step, anyway.

I just so happen to start a new job this week, and it so happens to be Go development on FreeBSD. It's a brand new system, so there aren't any ZFS snapshots to fall back on. Building from source isn't an option due to reasons. It stings.

 

[mariachi]

This is exactly what I do here. I run 15-CURRENT here.

My local poudriere repo is based on my local ports git repo. It contains patches I have no intention of pushing to FreeBSD. In addition, I use poudriere-options to set my own options instead of using defaults. This saves a lot of time and CPU when you're building ports for multiple machines for yourself.

You can also build a package outside of poudriere on another machine and simply copy it into the package repo. The next time you run poudriere to build packages it will notice the new .pkg file and inventory it. This is not documented anywhere. It's a little trick you can do when building many packages on your build machine and are impatient for it to finish, letting another machine help building packages.

Any good guide on setting this up you recommend? https://eugene-andrienko.com/en/it/2025/03/30/freebsd-pkgs-wout-unwanted-deps this is tangentially related, but doesn't go so much in depth. I know there's an official handbook, but I like to start with something small/tutorial-like before jumping into longer-form reading.
I'm thinking about setting up a git repo with all the config I need, including the ports I need modified the way I need them and write a long init script (vultr) to book FreeBSD, setup poudriere, build and install the packages I need, setup jails etc and start. Sort of Ansible without Ansible, because I don't need an always on server, but now I'm hesitant about relying on binary packages.

 

[cy@]

Any good guide on setting this up you recommend? https://eugene-andrienko.com/en/it/2025/03/30/freebsd-pkgs-wout-unwanted-deps this is tangentially related, but doesn't go so much in depth. I know there's an official handbook, but I like to start with something small/tutorial-like before jumping into longer-form reading.
I'm thinking about setting up a git repo with all the config I need, including the ports I need modified the way I need them and write a long init script (vultr) to book FreeBSD, setup poudriere, build and install the packages I need, setup jails etc and start. Sort of Ansible without Ansible, because I don't need an always on server, but now I'm hesitant about relying on binary packages.

You need Ansible, Puppet, Salt, Chef, cfengine, or some other tool. I use Ansible because we use it at $JOB and using Ansible for at home was a no-brainer.

You could use a shell script though a procedural language, like shell scripts or python, isn't great for configuration like a declarative language like Ansible is. Your ansible can run shell scripts and python scripts.

My ansible doesn't buildworld nor does it build ports or schedule poudriere bulk builds. My ansible only handles package installs an updates, and other configuration tasks. As a developer I still do installworld by hand. I recommend people use freebsd-update. I suppose you could write an ansible role or playbook to perform variouis freebsd-update tasks.

 

[fernandel]

IMO the problem with packages is because we have to many FreeBSD versions. If will be once a year, they will have more time for preparing everything what users need it without all those mess. And time i going, half of the month, June is close and again the same story.
And to use packages was also some push from FreeBSD Team.

 

[evgandr]

Any good guide on setting this up you recommend? https://eugene-andrienko.com/en/it/2025/03/30/freebsd-pkgs-wout-unwanted-deps this is tangentially related, but doesn't go so much in depth.

Hello! This blogpost is about creating a very local poudriere repository just for a laptop. After I wrote it, I met the problems with updating this repo. Especially after the "mass deletion" of packages, mentioned in this topic.

I was forced to rebuild a lot of packages, because poudriere couldn't fetch them from FreeBSD repo. So, I definitely not recommend to use that post for something "production ready" .

 

[blackhaz]

Another thought - why would a failed repo build trash the entire repo? I understand very little in behind the scenes, but I would imagine the process would be organized something like this. Repo build is launched, then some automatic QC. Monitor percentage of failed builds and critical languages/system components. If QC fails, leave the old repo and report to devs. If QC passes, put the new repo up. Just curious, are we basically overwriting old repos on the go while building them?

 

[mer]

Another thought - why would a failed repo build trash the entire repo?

I don't think this is a correct view. packages try to build about 34,724 different things. Some fail, some are skipped because of the failed, the rest build. Why not push the ones that actually build?
Next iteration ones that build still build, maybe some that failed now build which means some of the skipped now build.
Ones that still build are same versions so won't upgrade. The failed-that-now-build can upgrade and skipped-now-not-skipped build.

My opinion, it doesn't "trash the entire repo". You wind up with a repo that does not have everything. That means when you upgrade you may get most packages upgraded but maybe some that did not build are not available. So some things upgraded, some things not. This means your local machine may have problems but the repos "have what they have"

 

[gpw928]

DEar gpw928:
now this pkg upgrade destroy much more apps, that cause i can't work normal . why the department of pkg don't push all new apps at same time ? they just push part of app to news, part of app to disappear ...that was very bad . in the production system. we don't like that. thanks for your reply.. jedi of freebsd .

If this does not resonate with people, I don't know what would.

The FreeBSD application environment has been chronically unstable for most of calendar 2025.

pkg upgrade can't be trusted. Applications are broken. I have had to move one of my most important daily tasks to Linux because of two common applications that are core dumping or infinite looping.

Running Poudriere requires a commitment beyond what many (actually most) users are likely to commit. The recommendation to run FreeBSD 15-CURRENT, with jails, and master Ansible, Puppet, Salt, Chef, cfengine, or some other tool goes to my point. [It's helpful, but not to most users.]

On another thread people are being asked about what new features are desirable in new releases of FreeBSD.

The feature that I want to see most is a commitment to implement a stable path for kernel and applications for those who want and need it.

I hope this post reaches somebody who will give it serious thought.

 

[kpedersen]

This is also where the "single prefix, spray with files" approach doesn't quite work.

Imagine having a /usr/local2025q1 and a /usr/local2025q2. Things would be so much more flexible and resilient.

This is also why I put big packages like Libreoffice, Xorg in /opt/libreoffice, /opt/X11R7 respectively.