idea to make GUI for rebuild packages from sources

[ironudjin]

Hello all,

I'd like to share idea for community discussion about concept of a program which shold help desktop users to handle custom ports builds.

Let's say there is a user who installed FreeBSD desktop and setup all required application use common and the most stable way: pkg install <app>. After some time (s)he realized that the app is used doesn't have needed option. What is the common recommendation than? Yes, go to read documentation and setup ports-mgmt/poudriere or ports-mgmt/synth, build your own set of packages. To many complicated things for just rebuild one or several programs. But what if the user don't have dedicated hardware or/and expiriance or just don't want to do that? The easiest way is just go to /usr/ports/<port origin> and do: make config && make reinstall clean. But hey, what about rebuild dependecies? Ok, more proper way is to do portmaster /usr/ports/<port origin>. And everything will work correctly if you use quartaly updated repo and ports tree.

The idea is to make this process a little bit simplier for desktop user. There could be a simple desktop GUI program which allow to comfortable browse portage tree, change port options (wrapper around make showconfig-recursive), launch rebuild process ( make reinstall clean or portmaster <app>) and lock rebuilded packages and run-depends ( pkg lock) to avoid package overwrite from repo after pkg update && pkg upgrade. To save time and disk space it would be good idea to use portmaster --packages-build to use build depends from packages. After build process is finished, there could be an option to clean up build depends. There also could be some other additional features like:
- make security audit of installed packaged (wrapper around pkg audit -F) and display it in pretty way.
- show FreeBSD News
- possibly to manage all installed packages, but it's not so good idea to duplicate ports-mgmt/octopkg functionality.
- other things that could make complicated things a little bit simplier (please add your ideas to the list)

So... the question is: is this idea make sense?

This is just a window with ports tree written in TCL/TK with corrently no options:

 

[SirDice]

The easiest way is just go to /usr/ports/<port origin> and do: make config && make reinstall clean. But hey, what about rebuild dependecies? Ok, more proper way is to do portmaster /usr/ports/<port origin>. And everything will work correctly if you use quartaly updated repo and ports tree.

Actually, no. This will work but you're very likely to pull in unwanted dependencies. Quite often third party software is built using a configure script. A lot of times you can enable certain options that way, but many times you want to disable a feature. Which won't work because the configure script detects a certain library and will automagically enable that feature. That's why poudriere and synth both build packages in a 'clean room', i.e. a clean jail to prevent unwanted features from becoming enabled when they should have been disabled.

The idea is to make this process a little bit simplier for desktop user. There could be a simple desktop GUI program which allow to comfortable browse portage tree, change port options (wrapper around make showconfig-recursive), launch rebuild process ( make reinstall clean or portmaster <app>) and lock rebuilded packages and run-depends ( pkg lock) to avoid package overwrite from repo after pkg update && pkg upgrade.

You're putting the cart before the horse. You're going to end up with a bunch of incorrect dependency chains which will only result in a horrendous dependency hell.

 

[Deleted member 43773]

Frankly, to me not really.
But this is my own personal opinion, only, of course.
Instead of this Linux "sudo...apt get... sudo..apt add......sudo...apt... - warning, warning, warning, error, error, error, aborted... " fumble handicraft s..tuff you overstretch every Windows-user with,
I find package installations under FreeBSD even easier than to install anything under Windows ("Ok, Yes, Yes, Agree, OK, Yes, Yes!, YES!!, DO IT!!!")

Since there is already a page for to browse which ports are available,
and doing a pkg search is as easy as an installation via ports or pkg install (two short lines in a shell),
I don't see it simplify something complex that can be simplified.

And if you're doing installation by ports, it makes only sense if you do not want the default config.
Then you have to do more complex things than a GUI can handle without becoming an additional burden.

Sorry for not giving heads up,
but - again - this is my own personal opinion, only,
and you asked for feedback.

However,
I could imagine that GUI-addicted Mousenudgers very much like to have such a thing within their desktop environment (if there ain't such a thing like that already yet.)

 

[ironudjin]

This will work but you're very likely to pull in unwanted dependencies.

What do you mean in "unwanted dependencies"? Whan I enable some feature then it requires existing lib to be rebuilt with other feature and/or other libtary. In that case I just rebuild existing program and all required run-depends libraries. I wouldn't say that the new libraries which will install are unwanted.

You're putting the cart before the horse. You're going to end up with a bunch of incorrect dependency chains which will only result in a horrendous dependency hell.

It's not a problem to rebuild all incorrectly working programs/libtaties from source. It's possible to check ldd </path/to/lib_ot prodg> and rebuild all broken installed packages. Finally, the only thing that willl solve all the problems for sure - portmaster -af but that's require time and patient. BTW, I use packages built from latest ports tree for 10 years and didn't have a lot of problems with it. If something go wrong with newly installed package from source, thanks to portmaster's package save feature - there is always possiblity to roll back to previous version of certain package(s).

 

[ironudjin]

And if you're doing installation by ports, it makes only sense if you do not want the default config.
Then you have to do more complex things than a GUI can handle without becoming an additional burden.

That's the main target of GUI - to provide possibility to rebuild one or more programs from source easily with a few clicks + to simplify and make more compoftable process of doing "more complex things" with no need to dig into console.

 

[SirDice]

What do you mean in "unwanted dependencies"? Whan I enable some feature then it requires existing lib to be rebuilt with other feature and/or other libtary. In that case I just rebuild existing program and all required run-depends libraries. I wouldn't say that the new libraries which will install are unwanted.

Sure. Now think about the situation when you disable a feature. Application A pulls in application B as a dependency. I turn off that dependency. You rebuild the application. You tell the upstream source build system to turn off B. Upsteam build system detects B is installed and silently turns on the feature, even when you told it to turn it off. Application gets built with B dependency. Package doesn't register this dependency (it was turned off, right?). Application croaks when you run pkg-autoremove(8), which removes B (it's not a dependency anymore). Building ports 'in-place' is not a good way to deal with the problem.

BTW, I use packages built from latest ports tree for 10 years and didn't have a lot of problems with it. If something go wrong with newly installed package from source, thanks to portmaster's package save feature - there is always possiblity to roll back to previous version of certain package(s).

Building from ports is awesome. Don't get me wrong. I've been doing it for the past 25 years. I've built my own packages since FreeBSD first introduced the concept of a jail(8) with 4.0. Because a jail allowed me to set up a 'clean' environment so I can assure I do not get any unwanted dependencies in my packages. I've built packages using the old pkg_tools. I've been building packages for pkg(8) since it was first introduced. Creating packages correctly takes a lot more effort than you think. The reason Poudriere and Synth are complex is because this is a complex problem. And both tools tend to err on the safe side.

 

[myr]

for me it's a nice idea to have a window gui program, it helps with encouraging new variegated problem solvin' approaches on freebsd

 

[ironudjin]

Sure. Now think about the situation when you disable a feature. Application A pulls in application B as a dependency. I turn off that dependency. You rebuild the application. You tell the upstream source build system to turn off B. Upsteam build system detects B is installed and silently turns on the feature, even when you told it to turn it off. Application gets built with B dependency. Package doesn't register this dependency (it was turned off, right?). Application croaks when you run pkg-autoremove(8), which removes B (it's not a dependency anymore).

Faced with the such things a lot of times: after reinstall program A which doesn't require program/lib B - cleanup all thing that program A doesn't need anymore before (re)install anything else.

 

[PMc]

Actually, no. This will work but you're very likely to pull in unwanted dependencies. Quite often third party software is built using a configure script. A lot of times you can enable certain options that way, but many times you want to disable a feature. Which won't work because the configure script detects a certain library and will automagically enable that feature.

Yes, and this is a nuisance.
There is also some of our port maintainers who have their Makefile go stalking around in /usr/local and draw assumptions from there on what might be desired.

The way I came up to solve this is:
1. collect all the dependencies of the requested port(s)
2. walk through these and update the options for each as desired
3. if any change in 2. (plus or minus) go back to 1.

With a now accurate dependency list, start building and/or installing from bottom-up; make a zfs snapshot after each build or install, and afterwards rollback everything that is not required for the next build.
So make sure that no port can ever see anything they have not explicitely demanded as their own required dependency.

That's why poudriere and synth both build packages in a 'clean room', i.e. a clean jail to prevent unwanted features from becoming enabled when they should have been disabled.

Yepp. These appeared later, when the problem became all too obvious. So, due to "not invented here", I preferred to continue improving my own script set.

I don't think it is a bad idea to put a gui on top of the scheme - but it will be a lot of work to get it right.
(My script set is public on https gitr.daemon.contact/sysup, You're welcome to draw inspiration from it.)

 

[astyle]

Even Linux suffers from dependency hell, and has exact same dependencies. I'm currently trying to make KDE upgradable, and my idea is to compile everything with Poudriere, and then just pull in only updated KDE-related ports as needed. Just based on a curated list of packages. And it's a LOT of work just to have a working proof-of-concept, right now the project is on the back burner due to real-life distractions. But even so, I'm willing to put in the work, buy the hardware, connect the dots on the technical matters... I'm not gonna just pitch an idea and expect someone else to do the heavy lifting.

 

[ironudjin]

The way I came up to solve this is:
1. collect all the dependencies of the requested port(s)
2. walk through these and update the options for each as desired
3. if any change in 2. (plus or minus) go back to 1.

My current idea regarding how to deal with things is like:
If port option was changed - save a list of run depends of current package version and rebuild it. After that compare list of run depends of old package and new one. In case some of packages are dissappeared from run depends list of new version of package - check if those packages are used by any other port. If not - prompt user to delete them as they aren't required by any package anymore.
Is this correct logic? (the question for all who read this topic)

With a now accurate dependency list, start building and/or installing from bottom-up; make a zfs snapshot after each build or install, and afterwards rollback everything that is not required for the next build.
So make sure that no port can ever see anything they have not explicitely demanded as their own required dependency.

Hmm... It's nice idea to use zfs snapshots.

 

[PMc]

My current idea regarding how to deal with things is like:
If port option was changed - save a list of run depends of current package version and rebuild it. After that compare list of run depends of old package and new one. In case some of packages are dissappeared from run depends list of new version of package - check if those packages are used by any other port. If not - prompt user to delete them as they aren't required by any package anymore.
Is this correct logic? (the question for all who read this topic)

Lets consider a practical usecase: some desktop is used as a DAW. There is ardour installed, and it works with jack, and that is connected to a professional soundcard.
There is also firefox installed, but it should not spam into the professional equipment, it should instead output to the standard mainboard soundcard. So we remove option JACK from firefox - because otherwise, when firefox finds a jack server running, it will automatically connect to it.
Now the question is: will disabling the JACK option actually work? Or, when firefox is rebuilt, might the configure script detect the installed jack libraries and enable jack nevertheless?
I don't know. One would need to look into the configure script in the firefox source and figure out what it actually does. Occasionally I have to do that when something does not build the way I need it. That's why I wrote my script set in a way to prevent most of these cases.
Btw, the same scheme applies to desktop environments, and to various drivers for whatever features.

 

[ironudjin]

Now the question is: will disabling the JACK option actually work? Or, when firefox is rebuilt, might the configure script detect the installed jack libraries and enable jack nevertheless?

It depends of build settings and nature of firefox configure settings. I think in case you disable jack support option, Makefile should pass --disable-jack configure parameter (if it's exists, of course) to disable it for sure. If not - there is always an option which will works for sure: pkg delete -f jack, rebuild firefox and install jack back.

That's why I wrote my script set in a way to prevent most of these cases.

You have a nice scripts but it would be great to have more detailed comments for each there because some logic of them are difficult to understand. But it's a little bit offtopic.

Note that I'm not going to predict any problems what could happen with port(s) rebuild. The main goal is to create tool which will help to do complex operations more comfortably and simplier. For example, if you don't like default GUI behaviour or commands it run - you can go to settings tab and edit commands and other operations for fit your needs. For example, by default double click on port will open configuration dialog. After you change any option - there is "Build" button light up. In case you need to replace configure operation by build operation - you can go to settings and change it. If you need to build and install right after that - there should be an option to set it up. That's the main idea.

And the question is: does anybody require such tool?

#include <brain.h>

if (true) {                                                                                                             
    what features it has to have to be helpful?
} else
{
    let's close this topic.
}

...that's the "code" of this topic.

 

[zirias@]

This is, in its current form, a horrible idea, mainly for two reasons:

• You're attempting to build ports directly on the machine (unclean environment). While this works most of the time, it will eventually fail when a build picks something up from your machine and you don't notice .. could be optional dependencies, conflicts with older versions, etc pp. Then it might fail to build, or, even worse, fail to run correctly later.
• You have no way to verify the whole dependency tree, so packages built locally will sooner or later not match packages installed from official repos, which will also lead to all kinds of "funny" problems.

The only sane and robust way to have custom port options is to build your own packages repository, and poudriere is the standard tool for that. The -devel version of poudriere has an option to fetch binary packages and use them instead of rebuilding when they are a 100% match. To build as little as possible yourself, you should check out the commit of the ports tree that was used to build the official package repository. Unfortunately, this is a bit cumbersome to find out (you have to browse https://pkg-status.freebsd.org/ and find the build, its name is the commit hash).

So, if you want to create a helpful tool, start with the existing tooling (poudriere) and process, and build on top of that. I would certainly love to have some tooling that automatically finds the correct commit hash and sets my ports tree on it. I'm personally not interested in some GUI (neither for that utility function nor for browsing ports etc ....), but others might be.

 

[Deleted member 43773]

That's the main target of GUI - to provide possibility to

That's absolutely right.
And I'm fully aware of that.

The crucial - and often not seen or understimated - point is,
that this can be reached only by limitations.

Either you pick a small selection only of things to do from the whole shabang of possibilities.
Or the GUI becomes so complex, bloated and overloaded, that the whole idea of it is led ad absurdum.
That's what I ment with "becomes an additional burden."

Let me make this clear with a simple example.
You may know ffmpeg - very powerful and flexible, because of the many options it offers.
But CLI only.
If you came from DE (e.g. Windows) GUI-Computers only, you think:"shit. How old school."
If you are bond with shells, you see no problems, but opportunities.

Let me show you why:
If you're putting this into a GUI (there are several attemps doing that),
you either have to make a selection, which option you offer the user (e.g. by checkboxes),
or the GUI is that overloaded, it's not really usable anymore.
(most GUIs are designed as an combination of that: limited pre-selection of choices while still bloated)

Explantion:
Within a GUI you may stumble over a checkbox named e.g. "vn"
"What the heck is VN?"
Balloontip:"VN" (Because someone was too lazy to make the proper doc - again.)
"Thank you so very much!!"
So you look into the manpage.
"Aha. Now it's clear. Don't need it for my current task."
And you're sitting infront of the GUI full of options you don't know, are being only confused.
If the result is not, what you wanted (e.g. there is no output at all), you invest way more time in learning the usage of the GUI than if you had if you just used the cli-version,
because:

Otherwise you are working in the shell:
You have an clear idea what you need to do.
So you consult the manpage (anyway),
not reading it all, only looking for the things you need (and maybe stumble over other things:"Ah.. - cool! - good to know")
and for every paramter you directly see the explanation besides it.

So, conclusion:
You have to learn what vn is anyway.
The GUI makes you click a checkbox []vn,
in the shell you type ffmpeg -vn
So, there is no big deal after all.

or, to convince mathematical:
With a Mouse you have 4 signs:
left-click, right-click, middle-click, left-double-click
With keyboard I have 26 signs.
I rest my case.

There are usages, and advantages of GUIs, no question there.
But one also has to understand the limits and disadvantages of those.
Because that's one of the two core and main points, why MS Windows sucks:
It simply tries to offer everything to be done within the DE's GUI.
A) That is impossible - the idea is doomed from the very start.
B) Users don't know, what optiones, possibilities, freedom they are missing
C) Users who never had a real taste of anything else stay on a stupid level of believing this restricted, slow and highly inefficient way of working with a computer would be easy, innovative, and without any learning effort.
D) So everywhere they want to have GUIs for everything, not knowing about their limitations.

To be not misunderstood:
I don't want to derate your skills as a programmer.
I just want to avoid you make a mistake by start programming something, which will be of no much use.
And that was your question afterall.

 

[PMc]

Thanks for the writeup, Profighost - this is nicely explained, but it is only one side of the matter.
The other: GUI is two-dimensional, simultanous, multi-shape, multi-color, while text is one-dimensional and works only in linear time (see movie "Arrival").

You have a nice scripts but it would be great to have more detailed comments for each there because some logic of them are difficult to understand.

You're right. If I only had time...

 

[Ogis]

The idea may be good, but who will need it? who needs to complicate this process if there is a console and simple commands?

 

[ironudjin]

Let me show you why:
If you're putting this into a GUI (there are several attemps doing that),
you either have to make a selection, which option you offer the user (e.g. by checkboxes),

No, you don't have to make a selection. You have to offer end user an options with clear explanation what are they do and how affect on other options. That's not more or less then you'll find in man page.

or the GUI is that overloaded, it's not really usable anymore.

If there are a lot of options they can be groupped by some parameters. It user need some of them - user expand certain group and find needed option.

Explantion:
Within a GUI you may stumble over a checkbox named e.g. "vn"
"What the heck is VN?"

GUI should not provide options without explanation what they do. That's totally wrong way to organize GUI structure.

Otherwise you are working in the shell:
You have an clear idea what you need to do.
So you consult the manpage (anyway),
not reading it all, only looking for the things you need (and maybe stumble over other things:"Ah.. - cool! - good to know")
and for every paramter you directly see the explanation besides it.

It's the the same as you do in GUI where you have option tree with explanation of each one.

or, to convince mathematical:
With a Mouse you have 4 signs:
left-click, right-click, middle-click, left-double-click
With keyboard I have 26 signs.
I rest my case.

Do you use any of graphical environment like kde, xfce4, etc? I assume yes. Possibly because it's a much comfortable do some things with "4 signs" instead of console with "26 signs".
Do you use web-browser like firefox or chromium? Why do you use them if in console you have much more options to read web pages? You can curl -dump https://... | less or use w3m, lynx, etc...

That's why sometimes people don't want to read long man pages when they want to do one sipmple operations. In contest of this topic - (re)build package with needed options.

I just want to avoid you make a mistake by start programming something, which will be of no much use.

I understand and appreciate for any opinion.

 

[ironudjin]

This is, in its current form, a horrible idea, mainly for two reasons:

• You're attempting to build ports directly on the machine (unclean environment). While this works most of the time, it will eventually fail when a build picks something up from your machine and you don't notice .. could be optional dependencies, conflicts with older versions, etc pp. Then it might fail to build, or, even worse, fail to run correctly later.
• You have no way to verify the whole dependency tree, so packages built locally will sooner or later not match packages installed from official repos, which will also lead to all kinds of "funny" problems.

Thaks for explanation. I clearly undestand your position.

 

[Deleted member 43773]

this is nicely explained, but it is only one side of the matter.

Do you use any of graphical environment like kde, xfce4, etc? I assume yes. Possibly because it's a much comfortable do some things with "4 signs" instead of console with "26 signs".

Of course.
You're both right.
And of course I'm not a pure console-user only (WM only [fvwm], but no DE.)

I didn't want to get a fully detailed distinguish presentation of the pros and contras of GUIs.

It's way - way - more comfortable to simply click on a button instead of typing "firefox" in a shell, of course,
what would be a "single sign operation."

I just wanna have said:
- GUIs have not advantages only, and so are not the perfect choice for every purpose
(that's why we unixlike-users having one of a choice of GUIs additionally to a choice of powerful shell - we know, love and use both.
[Instead of most Windows-users, which know the GUI only, and many are already overstreched by giving operating instruction without showing pictures where to click at e.g. like "make a copy of that file to a disk." ? ])

- at a certain point, when it's getting more complex than "things you may do with 4...5 signs", a usage of a GUI will become contraproductive,
and it's way better using a shell
- many people neither know, nor understand, nor willing to accept that fact
- many underestimate the effort it needs to produce a good GUI ("skip, elide, leave out, don't do it", plus doing additional documentation, because just because it's a GUI doesn't mean it's self-explaining [another fact, many don't see])...)
As a result many GUIs are in an unfinished/unsatisfactory state, using options in checkboxes without any explanation; that's what I ment with "moronic balloon tips."

Thank you all guys for your professional level of disussing things.
Since there are other concerns, and I made my point (too) clear (I don't see this can be done with a good GUI; but I may be totally wrong about that),
I'm peacfully out of here.

Good results, and high efficiency!

P.

 

[Kael]

I've been using FreeBSD since 3.x, and I think the CLI tools have reached a very usable level. Especially poudriere(8), it has good control of separate build environments, port option controls, and parallel builds. It's web UI tracks the build processes in n-jails simultaneously and make it very easy to track down problems when they arise. I often need ports for which there is no FreeBSD package built, or I require a non-standard configuration. Sometimes my builds have to rolled out to multiple servers on the fly.

I could benefit some from a lightweight GUI to track and set options in poudriere jails, i.e. wraps poudriere-options(8), perhaps with a prettier options dialog, and feedback about what dependencies the user is pulling or a tree that shows what is happening when they click/unclick a box. It might be nice to see it all laid out before the GUI triggered poudriere-bulk(8).

I'm happy with the what we have now, but ngl, I'd give a GUI that gave me more info about port and port-option interactions and helped me search for settings I didn't want at a glance a try.