Updating portstree with git sometimes not up to date

B

bsd_akuma

Dear FreeBSD Community,

I hope that this is the right forum to ask for. Since portsnap is not supported in FreeBSD 14.0-RELEASE anymore I've decided it's time to switch to git for updating my port trees across the different servers. I did this a few weeks ago like described in the FreeBSD Handbook (https://docs.freebsd.org/en/books/handbook/ports/#ports-using). But for some unknown reason my portstree sometimes isn't up to date.

It sometimes occures when I make a sudo -E git pull in /usr/ports git will update, merge and commit changes, but when I take a look at the Makefile of a port it sometimes is a minor or major version behind the latest one. I also checked that I'm not using any quarterly branch and instead using the main / latest branch.

For example I updated my portstree a few days ago arround 01. december and wanted to install www/node20. When I tried to find it in my portstree there wouldn't be an entry with www/node20, but when I look at the official github release (https://github.com/freebsd/freebsd-ports/tree/main) there is clearly a www/node20 port in the latest (main) branch.
So when I try updating my portstree again with sudo -E git pull in /usr/ports nothing happens. No update is downloaded. But when I delete the whole ports/ directory and make a sudo -E git clone https://github.com/freebsd/freebsd-ports/tree/main /usr/ports in /usr and then check if the www/node20 port is available it suddenly exists.

Another example would be ftp/wget. After updating the portstree with git my version in the Makefile was 1.20.3 although the latest version is 1.21.4.

Unfortunately I can't recreate this behaviour since it randomly appears.

This behaviour, that git doesn't always update my portstree correctly is quite a nuisance and I don't know why it does not update it correctly. With this I often need to delete the entire portstree and download it with git again which, over time, is quite frustrating and tedious to do.

Maybe I miss a special git configuration? What can I do to further debug this?

Thank you in advance!

For your informations:

- I am behind a proxy (that's why I use -E in sudo calls)
- I am a novice when it comes to git. I can do simple pull, push, merge, commit but beyond that I am clueless
- The same problem also appears when using poudriere ports, which should not an intregal part of this thread
 
Also note if you make local changes, those won't get updated, commit your changes to your own branch.
 
SirDice said:
Also note if you make local changes, those won't get updated, commit your changes to your own branch.
Click to expand...
Yes I know, but thank you for the info. I don't make any changes on the ports tree aside from building ports. Does building ports in the git branch also "change" it, so I need to commit (or stash) the changes?
 
bsd_akuma said:
Does building ports in the git branch also "change" it, so I need to commit (or stash) the changes?
Click to expand...
No, ports tree could actually be entirely read-only. The work directories can be created somewhere else, as do distfiles and packages.
 
cracauer@ said:
github is not to original location of the ports tree. https://git.FreeBSD.org/ports.git is.

github is a mirror and scripts can be delayed.
Click to expand...
But if the github mirror is up to date with the https://git.FreeBSD.org/ports.git and I pull from the ports tree from github I expect that my ports tree should also be up to date. Which, in some cases, isn't. And that's what's bugging me.
Something needs to be wrong if my git pulls from github won't be updated sometimes. But I think there won't be a direct answer for this?

How do you update your ports tree? Maybe I'm doing something entirely wrong.
 
Any chance you have checked out a quarterly branch?
Then you'd have to manually switch to the newer branch after it has been created or use this (ugly) one-liner:
Code: 
/usr/local/bin/git -C /usr/ports pull && /usr/local/bin/git -C /usr/ports checkout $(/usr/local/bin/git -C /usr/ports branch -r | grep 202 | sort | tail -n1 | cut -d'/' -f2)

You *could* run this as a cronjob or part of a build-script that e.g. triggers your poudriere builds - but if you tend to make local changes to the ports tree (and rebase to keep them) this will either just leave them behind in the older branch or error out every now and then.
Because of this I usually switch to a newer quarterly by hand and carry over my local changes via cherry-pick. It's a bit tedious, but this forces me to keep the delta in the git tree as small as possible and create proper patches that can be used with poudriere or even try to upstream them...
 
bsd_akuma said:
How do you update your ports tree? Maybe I'm doing something entirely wrong.
Click to expand...

My setup is somewhat more complicated.

The actual working trees are not checked out from the public FreeBSD repositories. Instead I have a central mirror repository (no checkout) and the working trees are checkout out from there. The purpose is that changes I make in the working trees can get pushed to my central repository, from where other working trees can check out the modified tree. Pushing from the central repositories would make an official commit.
 
sko said:
Any chance you have checked out a quarterly branch?
Then you'd have to manually switch to the newer branch after it has been created or use this (ugly) one-liner:
Code: 
/usr/local/bin/git -C /usr/ports pull && /usr/local/bin/git -C /usr/ports checkout $(/usr/local/bin/git -C /usr/ports branch -r | grep 202 | sort | tail -n1 | cut -d'/' -f2)

You *could* run this as a cronjob or part of a build-script that e.g. triggers your poudriere builds - but if you tend to make local changes to the ports tree (and rebase to keep them) this will either just leave them behind in the older branch or error out every now and then.
Because of this I usually switch to a newer quarterly by hand and carry over my local changes via cherry-pick. It's a bit tedious, but this forces me to keep the delta in the git tree as small as possible and create proper patches that can be used with poudriere or even try to upstream them...
Click to expand...
I can't say for sure, but since I went with the instructions the FreeBSD handbook provides, I don't think that I am on a quarterly branch:

Code: 
$ git remote -v
origin  https://git.FreeBSD.org/ports.git (fetch)
origin  https://git.FreeBSD.org/ports.git (push)

$ git branch
* main

$ git branch -r
  # output omitted [...]
  origin/HEAD -> origin/main
  origin/main

I also don't want to to focus this threads topic on poudriere since this is another case where the same happens. I only want to focus on my "problematic" update behaviour of git for general updating the ports tree.
 
Thanks,

bsd_akuma said:
Code: 
$ git remote -v
origin  https://git.FreeBSD.org/ports.git (fetch)
origin  https://git.FreeBSD.org/ports.git (push)
Click to expand...

So, that's the freebsd.org origin (not GitHub, although I see that you had, previously, cloned from GitHub).

bsd_akuma said:
- I am a novice when it comes to git. I can do simple pull, push, merge, commit but beyond that I am clueless
Click to expand...

You will never be clued up. I certainly am not. The attached comedy is brought to you by Julia Evans.
 

Attachments

  • 1701994497901.png
    1701994497901.png
    201.7 KB · Views: 128
bsd_akuma said:
But if the github mirror is up to date with the https://git.FreeBSD.org/ports.git and I pull from the ports tree from github I expect that my ports tree should also be up to date. Which, in some cases, isn't. And that's what's bugging me.
Something needs to be wrong if my git pulls from github won't be updated sometimes. But I think there won't be a direct answer for this?

How do you update your ports tree? Maybe I'm doing something entirely wrong.
Click to expand...
My git configuration is complex. I use separate directory trees for development and push than I do as a basis for my own "prod" builds.

My working tree is in /home/.../somehwere whereas my prod/build tree is in /opt/.../somewhere. I own my tree while the account fgit owns the "prod" tree. This guarantees my personal account does not have write access to the tree my poudriere uses. Keeping personal account files separate from system account files, which the "prod" tree is, is best security practice. (Poudriere is also configured to build using my development tree for testing. But that repo is regularly cleaned out.)

Work is done within branches. And in worktrees, not so much in ports but certainly in src.

For my "prod" tree my /usr/ports default branch equivalent is cherry-picked from main while my next "release" of my "prod" tree is in a worktree as well, because I maintain 19 patches locally that I will not push. Poudriere knows these ports trees as prod and new-ports. It knows my development tree as cy-ports.

src is even more complex because I'm working on multiple projects each with its own branch. The most active branches are separate worktrees so I don't need to switch from one to another. I do the same for my ports development but to a much lesser extent.

Typically you should do git pull --rebase --autostash. If you maintain your own branch you will need to git fetch first, then git rebase --autostash origin/main. This brings the authoritative FreeBSD main into your branch. This way you can keep your own modifications in your tree without altering main. And you can switch between your branch and main very easily, and if you do a lot of that a worktree might make more sense. Worktrees avoid the lengthy wait as it rewrites files while switching branches.

Just do git pull --rebase --autostash and you will be fine. Use git.freebsd.org. It should have its updates sooner than the GH copy. If you see differnces after a while, this would suggest that your tree is corrupted in some way. You may want to remove it and clone again.

Git has its warts but is generally a good tool to work with.
 
bsd_akuma said:
It sometimes occures when I make a sudo -E git pull in /usr/ports git will update, merge and commit changes, but when I take a look at the Makefile of a port it sometimes is a minor or major version behind the latest one. I also checked that I'm not using any quarterly branch and instead using the main / latest branch.
Click to expand...
"merge"? Are you making local changes? If so, it is possible that you changed a file that was also modified in the repo and git will refuse to merge. I suggest you keep a lot of all your git pulls with something like

git pull |& tee -a gitpull.log

and when you notice a discrepancy look through the log to see if git complained (you might miss them as usually there are many changes in ports). "tee" so that you can also see git pull output on your terminal.

Also, make your local changes on a separate branch, not main.
 
grahamperrin said:
Thanks,

So, that's the freebsd.org origin (not GitHub, although I see that you had, previously, cloned from GitHub).
Click to expand...
Yes that's my mistake. Initially I cloned from Github, but since my problem started I, somewhere in the process, changed to the FreeBSD Git-Portstree.

grahamperrin said:
At any time:

git -C /usr/ports status

I mean, you can run it any time, however each run (particularly the first) may be noticeably disk-intensive.
Click to expand...

I can try that thank you.

cy@ said:
Typically you should do git pull --rebase --autostash. If you maintain your own branch you will need to git fetch first, then git rebase --autostash origin/main. This brings the authoritative FreeBSD main into your branch. This way you can keep your own modifications in your tree without altering main. And you can switch between your branch and main very easily, and if you do a lot of that a worktree might make more sense. Worktrees avoid the lengthy wait as it rewrites files while switching branches.

Just do git pull --rebase --autostash and you will be fine. Use git.freebsd.org. It should have its updates sooner than the GH copy. If you see differnces after a while, this would suggest that your tree is corrupted in some way. You may want to remove it and clone again.

Git has its warts but is generally a good tool to work with.
Click to expand...
I will try that out in future git pulls thank you very much.

bakul said:
"merge"? Are you making local changes? If so, it is possible that you changed a file that was also modified in the repo and git will refuse to merge. I suggest you keep a lot of all your git pulls with something like

git pull |& tee -a gitpull.log

and when you notice a discrepancy look through the log to see if git complained (you might miss them as usually there are many changes in ports). "tee" so that you can also see git pull output on your terminal.

Also, make your local changes on a separate branch, not main.
Click to expand...
No as I mentioned I don't do local merges. Maybe my term is bad here. I learned that if I do a git pull git will merge changes from the the remote repository into my local (cloned) repository.
But the idea of writing to a log on every pull sounds like a great idea. I didn't know I could do that this easily. I will do that for new pulls and see when and why my ports tree might get corrupted.
 
bsd_akuma said:
Dear FreeBSD Community,

I hope that this is the right forum to ask for. Since portsnap is not supported in FreeBSD 14.0-RELEASE anymore I've decided it's time to switch to git for updating my port trees across the different servers. I did this a few weeks ago like described in the FreeBSD Handbook (https://docs.freebsd.org/en/books/handbook/ports/#ports-using). But for some unknown reason my portstree sometimes isn't up to date.

It sometimes occures when I make a sudo -E git pull in /usr/ports git will update, merge and commit changes, but when I take a look at the Makefile of a port it sometimes is a minor or major version behind the latest one. I also checked that I'm not using any quarterly branch and instead using the main / latest branch.

For example I updated my portstree a few days ago arround 01. december and wanted to install www/node20. When I tried to find it in my portstree there wouldn't be an entry with www/node20, but when I look at the official github release (https://github.com/freebsd/freebsd-ports/tree/main) there is clearly a www/node20 port in the latest (main) branch.
So when I try updating my portstree again with sudo -E git pull in /usr/ports nothing happens. No update is downloaded. But when I delete the whole ports/ directory and make a sudo -E git clone https://github.com/freebsd/freebsd-ports/tree/main /usr/ports in /usr and then check if the www/node20 port is available it suddenly exists.

Another example would be ftp/wget. After updating the portstree with git my version in the Makefile was 1.20.3 although the latest version is 1.21.4.

Unfortunately I can't recreate this behaviour since it randomly appears.

This behaviour, that git doesn't always update my portstree correctly is quite a nuisance and I don't know why it does not update it correctly. With this I often need to delete the entire portstree and download it with git again which, over time, is quite frustrating and tedious to do.

Maybe I miss a special git configuration? What can I do to further debug this?

Thank you in advance!

For your informations:

- I am behind a proxy (that's why I use -E in sudo calls)
- I am a novice when it comes to git. I can do simple pull, push, merge, commit but beyond that I am clueless
- The same problem also appears when using poudriere ports, which should not an intregal part of this thread
Click to expand...
 
You must log in or register to reply here.
Back
Top