buildworld WITHOUT_TOOLCHAIN, buildkernel and installworld

Cath O'Deray

Cath O'Deray

 
From the attached file:

Code: 
root@mowa219-gjp4-8570p-freebsd:/usr/src # cd /usr/src && time make -j2 WITHOUT_TOOLCHAIN=yes buildworld > buildworld.log 2>&1 && make -j2 buildkernel KERNCONF=GENERIC-NODEBUG
Ambiguous output redirect.
root@mowa219-gjp4-8570p-freebsd:/usr/src # sh
# cd /usr/src && time make -j2 WITHOUT_TOOLCHAIN=yes buildworld > buildworld.log 2>&1 && make -j2 buildkernel KERNCONF=GENERIC-NODEBUG

2>&1

From <https://unix.stackexchange.com/a/118310/13260> I see that csh does not have this operator, so how should I modify the first command?

If you want to redirect both stdout and stderr portably, the syntax is

cmd > file 2>&1
Click to expand...

I'm confused, because the relevant part of my first command did seem to follow that syntax.

My preferred shell for root is, and will remain: /bin/csh

Background

The recent WITHOUT_TOOLCHAIN=yes hint under <https://www.freebsd.org/status/report-2021-07-2021-09/#_current_compilation_time_analysis>.
 

Attachments

Is the answer to simply not have redirection to /dev/null in my string of commands?

So:

Code: 
root@mowa219-gjp4-8570p-freebsd:/usr/src # rm buildworld.log
root@mowa219-gjp4-8570p-freebsd:/usr/src # cd
root@mowa219-gjp4-8570p-freebsd:~ # cd /usr/src && time make -j2 WITHOUT_TOOLCHAIN=yes buildworld > buildworld.log && make -j2 buildkernel KERNCONF=GENERIC-NODEBUG
^C0.051u 0.008s 0:01.60 3.1%    35468+9433k 42+143io 0pf+0w
root@mowa219-gjp4-8570p-freebsd:/usr/src # head ./buildworld.log
--- buildworld ---
make[1]: "/usr/src/Makefile.inc1" line 330: SYSTEM_COMPILER: Determined that CC=cc matches the source tree.  Not bootstrapping a cross-compiler.
make[1]: "/usr/src/Makefile.inc1" line 335: SYSTEM_LINKER: Determined that LD=ld matches the source tree.  Not bootstrapping a cross-linker.
--- buildworld_prologue ---
--------------------------------------------------------------
>>> World build started on Tue Dec 21 05:55:47 GMT 2021
--------------------------------------------------------------
--- _worldtmp ---
--------------------------------------------------------------
>>> Rebuilding the temporary build tree
root@mowa219-gjp4-8570p-freebsd:/usr/src #
 
grahamperrin said:
I see that csh does not have this operator, so how should I modify the first command?
Click to expand...
What Bourne shell's > buildworld.log 2>&1 does is direct STDERR (2) to STDOUT (1) then direct STDOUT to buildworld.log (so both STDERR and STDOUT end up in buildworld.log). The tcsh(1) equivalent is >& buildworld.log

Code: 
       > name
       >! name
       >& name
       >&! name
               The file name is used as standard output.  If the file does not
               exist then it is created; if the file exists, it is truncated,
               its previous contents being lost.

               If the shell variable noclobber is set, then the file must not
               exist or be a character special file (e.g., a terminal or
               `/dev/null') or an error results.  This helps prevent
               accidental destruction of files.  In this case the `!' forms
               can be used to suppress this check.  If notempty is given in
               noclobber, `>' is allowed on empty files; if ask is set, an
               interacive confirmation is presented, rather than an error.

               The forms involving `&' route the diagnostic output into the
               specified file as well as the standard output.  name is
               expanded in the same way as `<' input filenames are.
 
Thanks.


Don't forget to do ``make cleandepend && make depend''
Click to expand...

I never noticed that line before.

Is it an instruction to me to perform those actions? Or does e.g. make -j2 buildkernel KERNCONF=GENERIC-NODEBUG perform them for me?
 
Please, can anyone answer the question about the "Don't forget …" line?


Also, with added emphasis:

… Using WITHOUT_TOOLCHAIN=yes is fine as long as there are no major LLVM compiler changes. …
Click to expand...

The subsequent make installworld failed.

Is it reasonable to assume that failure was a result of WITHOUT_TOOLCHAIN=yes for the preceding make buildworld?

(No record of the failure. Single user mode at the time.)

What would be an example of a major LLVM compiler change?

Output from bectl(8) (below) reminds me that the attempt was from d109559ddbf to 1654b51455c (2021-11-29, 2021-12-20).

Code: 
% bectl list -c creation
BE                    Active Mountpoint Space Created
n250511-5f73b3338ee-d -      -          4.82G 2021-11-13 15:43
n251146-d109559ddbf-g -      -          67.7M 2021-12-16 23:20
n251852-1654b51455c-a -      -          1.60G 2021-12-21 04:29
n251146-d109559ddbf-h NR     /          78.1G 2021-12-24 08:59
%
 
covacat said:
… it's probably done by make buildkernel
Click to expand...

Thanks, so I'll not follow the on-screen advice.


Where WITHOUT_TOOLCHAIN=yes (on 21st December) resulted in a build time of less than two hours, the more traditional approach today took more than six:

Code: 
root@mowa219-gjp4-8570p-freebsd:~ # cd /usr/src && time make -j2 buildworld >& buildworld.log && time make -j2 buildkernel KERNCONF=GENERIC-NODEBUG >& buildkernel.log
38570.186u 1355.667s 6:21:36.34 174.3%  73151+10590k 279382+568517io 173858pf+64w

grahamperrin said:
The subsequent make installworld failed.
Click to expand...

Fingers crossed for no failure today …
 
Oh right! I missed that.

I had some peculiarities towards the tail end of today's install, but ultimately successful. Might write about them tomorrow. Christmas lunch today :)
 
grahamperrin said:
?… peculiarities towards the tail end …
Click to expand...

etcupdate resolve

Edition of just one file – /etc/group – completed the resolution.

etcupdate -p && cd /usr/src && time make installworld >& installworld.log && etcupdate -B

– a combination of three commands – resulted in the need to re-run etcupdate resolve and repeat edition of the same file.

What's wrong with that trio? I seem to have output from etcupdate -B at the tail of the log:

Code: 
% tail -n 47 /usr/src/installworld.log
install -l rs -o root -g wheel -m 755 -S   libgssapi_spnego.so.10 /usr/lib32/libgssapi_spnego.so
        3.58 real         0.26 user         0.14 sys
--------------------------------------------------------------
>>> Installing everything completed on Sat Dec 25 09:18:39 GMT 2021
--------------------------------------------------------------
      391.46 real        62.69 user        33.71 sys
Scanning /usr/share/certs/untrusted for certificates...
Scanning /usr/share/certs/trusted for certificates...
Skipping untrusted certificate /usr/share/certs/trusted/AddTrust_External_Root.pem (/etc/ssl/untrusted/157753a5.0)
Skipping untrusted certificate /usr/share/certs/trusted/AddTrust_Low-Value_Services_Root.pem (/etc/ssl/untrusted/861a399d.0)
Skipping untrusted certificate /usr/share/certs/trusted/Camerfirma_Chambers_of_Commerce_Root.pem (/etc/ssl/untrusted/f90208f7.0)
Skipping untrusted certificate /usr/share/certs/trusted/Camerfirma_Global_Chambersign_Root.pem (/etc/ssl/untrusted/cb59f961.0)
Skipping untrusted certificate /usr/share/certs/trusted/Certum_Root_CA.pem (/etc/ssl/untrusted/442adcac.0)
Skipping untrusted certificate /usr/share/certs/trusted/Chambers_of_Commerce_Root_-_2008.pem (/etc/ssl/untrusted/c47d9980.0)
Skipping untrusted certificate /usr/share/certs/trusted/D-TRUST_Root_CA_3_2013.pem (/etc/ssl/untrusted/0b7c536a.0)
Skipping untrusted certificate /usr/share/certs/trusted/EC-ACC.pem (/etc/ssl/untrusted/349f2832.0)
Skipping untrusted certificate /usr/share/certs/trusted/EE_Certification_Centre_Root_CA.pem (/etc/ssl/untrusted/128805a3.0)
Skipping untrusted certificate /usr/share/certs/trusted/GeoTrust_Global_CA.pem (/etc/ssl/untrusted/2c543cd1.0)
Skipping untrusted certificate /usr/share/certs/trusted/GeoTrust_Primary_Certification_Authority.pem (/etc/ssl/untrusted/480720ec.0)
Skipping untrusted certificate /usr/share/certs/trusted/GeoTrust_Primary_Certification_Authority_-_G2.pem (/etc/ssl/untrusted/116bf586.0)
Skipping untrusted certificate /usr/share/certs/trusted/GeoTrust_Primary_Certification_Authority_-_G3.pem (/etc/ssl/untrusted/e2799e36.0)
Skipping untrusted certificate /usr/share/certs/trusted/GeoTrust_Universal_CA.pem (/etc/ssl/untrusted/ad088e1d.0)
Skipping untrusted certificate /usr/share/certs/trusted/GeoTrust_Universal_CA_2.pem (/etc/ssl/untrusted/8867006a.0)
Skipping untrusted certificate /usr/share/certs/trusted/Global_Chambersign_Root_-_2008.pem (/etc/ssl/untrusted/0c4c9b6c.0)
Skipping untrusted certificate /usr/share/certs/trusted/LuxTrust_Global_Root_2.pem (/etc/ssl/untrusted/def36a68.0)
Skipping untrusted certificate /usr/share/certs/trusted/OISTE_WISeKey_Global_Root_GA_CA.pem (/etc/ssl/untrusted/b1b8a7f3.0)
Skipping untrusted certificate /usr/share/certs/trusted/QuoVadis_Root_CA.pem (/etc/ssl/untrusted/080911ac.0)
Skipping untrusted certificate /usr/share/certs/trusted/Sonera_Class_2_Root_CA.pem (/etc/ssl/untrusted/9c2e7d30.0)
Skipping untrusted certificate /usr/share/certs/trusted/Staat_der_Nederlanden_Root_CA_-_G2.pem (/etc/ssl/untrusted/5c44d531.0)
Skipping untrusted certificate /usr/share/certs/trusted/Staat_der_Nederlanden_Root_CA_-_G3.pem (/etc/ssl/untrusted/5a4d6896.0)
Skipping untrusted certificate /usr/share/certs/trusted/SwissSign_Platinum_CA_-_G2.pem (/etc/ssl/untrusted/a8dee976.0)
Skipping untrusted certificate /usr/share/certs/trusted/Symantec_Class_1_Public_Primary_Certification_Authority_-_G4.pem (/etc/ssl/untrusted/62744ee1.0)
Skipping untrusted certificate /usr/share/certs/trusted/Symantec_Class_1_Public_Primary_Certification_Authority_-_G6.pem (/etc/ssl/untrusted/26312675.0)
Skipping untrusted certificate /usr/share/certs/trusted/Symantec_Class_2_Public_Primary_Certification_Authority_-_G4.pem (/etc/ssl/untrusted/4d4ba017.0)
Skipping untrusted certificate /usr/share/certs/trusted/Symantec_Class_2_Public_Primary_Certification_Authority_-_G6.pem (/etc/ssl/untrusted/1320b215.0)
Skipping untrusted certificate /usr/share/certs/trusted/Taiwan_GRCA.pem (/etc/ssl/untrusted/6410666e.0)
Skipping untrusted certificate /usr/share/certs/trusted/Trustis_FPS_Root_CA.pem (/etc/ssl/untrusted/d853d49e.0)
Skipping untrusted certificate /usr/share/certs/trusted/VeriSign_Class_3_Public_Primary_Certification_Authority_-_G4.pem (/etc/ssl/untrusted/7d0b38bd.0)
Skipping untrusted certificate /usr/share/certs/trusted/VeriSign_Class_3_Public_Primary_Certification_Authority_-_G5.pem (/etc/ssl/untrusted/b204d74a.0)
Skipping untrusted certificate /usr/share/certs/trusted/VeriSign_Universal_Root_Certification_Authority.pem (/etc/ssl/untrusted/c01cdfa2.0)
Skipping untrusted certificate /usr/share/certs/trusted/Verisign_Class_1_Public_Primary_Certification_Authority_-_G3.pem (/etc/ssl/untrusted/ee1365c0.0)
Skipping untrusted certificate /usr/share/certs/trusted/Verisign_Class_2_Public_Primary_Certification_Authority_-_G3.pem (/etc/ssl/untrusted/dc45b0bd.0)
Skipping untrusted certificate /usr/share/certs/trusted/Verisign_Class_3_Public_Primary_Certification_Authority_-_G3.pem (/etc/ssl/untrusted/c0ff1f52.0)
Skipping untrusted certificate /usr/share/certs/trusted/thawte_Primary_Root_CA.pem (/etc/ssl/untrusted/2e4eed3c.0)
Skipping untrusted certificate /usr/share/certs/trusted/thawte_Primary_Root_CA_-_G2.pem (/etc/ssl/untrusted/c089bbbd.0)
Skipping untrusted certificate /usr/share/certs/trusted/thawte_Primary_Root_CA_-_G3.pem (/etc/ssl/untrusted/ba89ed3b.0)
Scanning /usr/local/share/certs for certificates...
%

I worked around by not having && etcupdate -B as part of a trio. etcupdate -B alone succeeded.
 
Last edited:
etcupdate(8)

Below, where am I going wrong? Why does the conflict recur?

1641002659223.png
 
look for /etc/master.passwd with editor, it is highly likely you will see <<<< and >>> which separates old and new lines in that file. you review and edit the file to reflect upstream changes. whether
etcupdate resolve supposed to resolve it automatically, is unknown to me. this conflicts are pretty similar to ones, you find with git. So, normally, what i do to understand what is changed git log -- /etc/master.passwd or browse online https://cgit.freebsd.org/src/log/etc/master.passwd. Once you edited the file to reflect upstream changes, save, mark as resolved in etcupdate and it should be ok.
 
There's no problem with edition (I use nano); my resolution is followed by the run of etcupdate -p, which finds no conflict.
 
Cross-reference:

forums.freebsd.org

Solved - Update from source: etcupdate -B fails

I've just compiled world & kernel from the latest stable/13 branch. I've done this several times over the last two months and didn't run into any issues following the procedure outlined by the handbook. This time however, running etcupdate -B fails: Conflicts remain from previous update...
forums.freebsd.org forums.freebsd.org
 
You must log in or register to reply here.
Back
Top