Poudriere: How to Clean up a Dropped SSH Session During a Build?

SomePoorBastard · Jan 9, 2022

Hello FreeBSD Forums,

I'm having some issues cleaning up after a "failed" poudriere build. To summarize, I initiated a poudriere bulk over an ssh connection, and the network connection was interrupted. When I was able to ssh back into the machine running poudriere, none of the build processes were running, but all of the build files were still there.

I attempted to remove the poudriere jail using poudriere jail -d -j 13amd64, thinking this would free up the most recent build environment so I could run poudriere bulk again. However, during the process I noticed that there were a lot of errors; specifically, where the removal of files were not able to be completed since they were mounted as read-only. An excerpt is below:

Code:

rm: /pool/var/poudriere/data/.m/13amd64-server/05/usr/ports/audio/midifile: Read-only file system
rm: /pool/var/poudriere/data/.m/13amd64-server/05/usr/ports/audio/miniaudio/pkg-descr: Read-only file system
rm: /pool/var/poudriere/data/.m/13amd64-server/05/usr/ports/audio/miniaudio/Makefile: Read-only file system
rm: /pool/var/poudriere/data/.m/13amd64-server/05/usr/ports/audio/miniaudio/distinfo: Read-only file system

I interrupted the removal process using [CTRL][C]. After this point, any attempt to remove the jail using the previous command will report there is no such jail.

However, using jls, I can see a list of jails with the jail name combined with the port tree name. Again, an excerpt is below.:

JID	IP Address	Hostname	Path
31	127.0.0.1	build-host.	/pool/var/poudriere/data/.m/13amd64-server/ref
32		build-host.	/pool/var/poudriere/data/.m/13amd64-server/ref
33	127.0.0.1	build-host.	/pool/var/poudriere/data/.m/13amd64-server/06
34		build-host.	/pool/var/poudriere/data/.m/13amd64-server/06

So far I haven't been able to figure out from the man pages how to remove these jails using either `jls` or the `jail` commands. I'll be looking closer to determine how and post here if I figure it out.

Other ways I attempted to remove the jail used the follwing commands, which also both failed.

Code:

zfs destroy -R pool/poudriere/jails/13amd64
cannot unmount '/pool/var/poudriere/jails/13amd64': unmount failed

Code:

umount pool/poudriere/jails/13amd64
umount: unmount of /pool/var/poudriere/jails/13amd64 failed: Device busy

Right now I'm at a loss of what to do next. What I want to do is to clean up the mess I've made so far, re-create the jail, and start building the packages again for my systems. For the next time I attempt to build over a wireless or remote connection, I'll try to use tmux before starting the job so I can pick up where I left off with the build rather than repeating this painful process.

zirias@ · Jan 9, 2022

Normally, poudriere cleans up itself on termination (and SIGPIPE should terminate), so you were a bit unlucky.

You will have some null-mounts inside the jails poudriere creates. List them with mount and unmount them manually, after that, removing the jail should work.

To avoid these problems in the future, always run poudriere e.g. inside sysutils/tmux.

Erichans · Jan 9, 2022

For poudriere, I think that should be mentioned in the FreeBSD handbook, probably at 4.6.1. Initialize Jails and Port Trees or just preceding it.

Something like:

When remotely controlling a poudriere build (for example via SSH), the connection might get interrupted and break the current poudriere build process. Please consider an appropriate tool for remotely controlling a poudriere build that allows you to reconnect to an interrupted connection. For example sysutils/tmux or sysutils/screen

It may even be worth considering elevating such a text; e.g. in an Important or Note text block.

astyle · Jan 10, 2022

Ah, the nice thing about Poudriere is that it can survive even a reboot. You can kill the process, and whatnot, reboot the machine, and then just restart your poudriere-bulk(8) command. It will pick up where it left off, as though nothing happened. The only thing to pay attention to is the filepath to the list of ports that you feed to poudriere-bulk(8)

Alain De Vos · Jan 10, 2022

I have a small script destroying the hung zfs datasets:

cat zfs_destroy

Code:

zfs destroy -f -R ZT/jails/a/apou/jails/ap-ports-ref/01
zfs destroy -f -R ZT/jails/a/apou/jails/ap-ports-ref/02
zfs destroy -f -R ZT/jails/a/apou/jails/ap-ports-ref/03
zfs destroy -f -R ZT/jails/a/apou/jails/ap-ports-ref/04
zfs destroy -f -R ZT/jails/a/apou/jails/ap-ports-ref/05 
zfs destroy -f -R ZT/jails/a/apou/jails/ap-ports-ref/06

SomePoorBastard · Jan 10, 2022

I followed Ziria's advice and attempted to remove the null mounts listed. There were a ton of mount points:

Code:

pool/poudriere on /pool/poudriere (zfs, local, nfsv4acls)
pool/poudriere/ports/server on /pool/var/poudriere/ports/server (zfs, local, noatime, nfsv4acls)
pool/var/poudriere-base on /pool/var/poudriere-base (zfs, local, nfsv4acls)
pool/poudriere/jails/13amd64 on /pool/var/poudriere/jails/13amd64 (zfs, local, noatime, nfsv4acls)
pool/poudriere/data on /pool/var/poudriere/data (zfs, local, noatime, nfsv4acls)
pool/poudriere/data/packages on /pool/var/poudriere/data/packages (zfs, local, noatime, nfsv4acls)
/pool/var/poudriere/jails/13amd64/rescue on /pool/var/poudriere/data/.m/13amd64-server/ref/rescue (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/jails/13amd64/usr/share on /pool/var/poudriere/data/.m/13amd64-server/ref/usr/share (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/jails/13amd64/usr/tests on /pool/var/poudriere/data/.m/13amd64-server/ref/usr/tests (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/jails/13amd64/usr/lib32 on /pool/var/poudriere/data/.m/13amd64-server/ref/usr/lib32 (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/jails/13amd64/usr/src on /pool/var/poudriere/data/.m/13amd64-server/ref/usr/src (nullfs, local, noatime, read-only, nfsv4acls)
devfs on /pool/var/poudriere/data/.m/13amd64-server/ref/dev (devfs)
fdescfs on /pool/var/poudriere/data/.m/13amd64-server/ref/dev/fd (fdescfs)
procfs on /pool/var/poudriere/data/.m/13amd64-server/ref/proc (procfs, local)
linprocfs on /pool/var/poudriere/data/.m/13amd64-server/ref/compat/linux/proc (linprocfs, local)
tmpfs on /pool/var/poudriere/data/.m/13amd64-server/ref/.p (tmpfs, local)
tmpfs on /pool/var/poudriere/data/.m/13amd64-server/ref/wrkdirs (tmpfs, local)
/pool/var/poudriere/ports/server on /pool/var/poudriere/data/.m/13amd64-server/ref/usr/ports (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/packages/13amd64-server/.building on /pool/var/poudriere/data/.m/13amd64-server/ref/packages (nullfs, local, noatime, read-only, nfsv4acls)
/pool/source/FreeBSD/ports/distfiles on /pool/var/poudriere/data/.m/13amd64-server/ref/distfiles (nullfs, local, nfsv4acls)
tmpfs on /pool/var/poudriere/data/.m/13amd64-server/ref/var/db/ports (tmpfs, local)
/pool/var/poudriere/data/.m/13amd64-server/ref/rescue on /pool/var/poudriere/data/.m/13amd64-server/06/rescue (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/share on /pool/var/poudriere/data/.m/13amd64-server/06/usr/share (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/tests on /pool/var/poudriere/data/.m/13amd64-server/06/usr/tests (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/lib32 on /pool/var/poudriere/data/.m/13amd64-server/06/usr/lib32 (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/src on /pool/var/poudriere/data/.m/13amd64-server/06/usr/src (nullfs, local, noatime, read-only, nfsv4acls)
devfs on /pool/var/poudriere/data/.m/13amd64-server/06/dev (devfs)
fdescfs on /pool/var/poudriere/data/.m/13amd64-server/06/dev/fd (fdescfs)
procfs on /pool/var/poudriere/data/.m/13amd64-server/06/proc (procfs, local)
linprocfs on /pool/var/poudriere/data/.m/13amd64-server/06/compat/linux/proc (linprocfs, local)
tmpfs on /pool/var/poudriere/data/.m/13amd64-server/06/.p (tmpfs, local)
tmpfs on /pool/var/poudriere/data/.m/13amd64-server/06/wrkdirs (tmpfs, local)
/pool/var/poudriere/ports/server on /pool/var/poudriere/data/.m/13amd64-server/06/usr/ports (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/packages/13amd64-server/.building on /pool/var/poudriere/data/.m/13amd64-server/06/packages (nullfs, local, noatime, read-only, nfsv4acls)
/pool/source/FreeBSD/ports/distfiles on /pool/var/poudriere/data/.m/13amd64-server/06/distfiles (nullfs, local, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/var/db/ports on /pool/var/poudriere/data/.m/13amd64-server/06/var/db/ports (nullfs, local, read-only)
/pool/var/poudriere/data/.m/13amd64-server/ref/rescue on /pool/var/poudriere/data/.m/13amd64-server/04/rescue (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/rescue on /pool/var/poudriere/data/.m/13amd64-server/07/rescue (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/rescue on /pool/var/poudriere/data/.m/13amd64-server/05/rescue (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/rescue on /pool/var/poudriere/data/.m/13amd64-server/02/rescue (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/rescue on /pool/var/poudriere/data/.m/13amd64-server/03/rescue (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/rescue on /pool/var/poudriere/data/.m/13amd64-server/08/rescue (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/rescue on /pool/var/poudriere/data/.m/13amd64-server/01/rescue (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/share on /pool/var/poudriere/data/.m/13amd64-server/04/usr/share (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/share on /pool/var/poudriere/data/.m/13amd64-server/08/usr/share (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/share on /pool/var/poudriere/data/.m/13amd64-server/01/usr/share (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/share on /pool/var/poudriere/data/.m/13amd64-server/03/usr/share (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/share on /pool/var/poudriere/data/.m/13amd64-server/07/usr/share (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/share on /pool/var/poudriere/data/.m/13amd64-server/05/usr/share (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/tests on /pool/var/poudriere/data/.m/13amd64-server/04/usr/tests (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/tests on /pool/var/poudriere/data/.m/13amd64-server/08/usr/tests (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/tests on /pool/var/poudriere/data/.m/13amd64-server/05/usr/tests (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/tests on /pool/var/poudriere/data/.m/13amd64-server/03/usr/tests (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/tests on /pool/var/poudriere/data/.m/13amd64-server/01/usr/tests (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/lib32 on /pool/var/poudriere/data/.m/13amd64-server/04/usr/lib32 (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/lib32 on /pool/var/poudriere/data/.m/13amd64-server/08/usr/lib32 (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/lib32 on /pool/var/poudriere/data/.m/13amd64-server/05/usr/lib32 (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/lib32 on /pool/var/poudriere/data/.m/13amd64-server/01/usr/lib32 (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/lib32 on /pool/var/poudriere/data/.m/13amd64-server/03/usr/lib32 (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/src on /pool/var/poudriere/data/.m/13amd64-server/04/usr/src (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/src on /pool/var/poudriere/data/.m/13amd64-server/08/usr/src (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/src on /pool/var/poudriere/data/.m/13amd64-server/05/usr/src (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/src on /pool/var/poudriere/data/.m/13amd64-server/01/usr/src (nullfs, local, noatime, read-only, nfsv4acls)
devfs on /pool/var/poudriere/data/.m/13amd64-server/04/dev (devfs)
devfs on /pool/var/poudriere/data/.m/13amd64-server/08/dev (devfs)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/share on /pool/var/poudriere/data/.m/13amd64-server/02/usr/share (nullfs, local, noatime, read-only, nfsv4acls)
devfs on /pool/var/poudriere/data/.m/13amd64-server/01/dev (devfs)
devfs on /pool/var/poudriere/data/.m/13amd64-server/05/dev (devfs)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/tests on /pool/var/poudriere/data/.m/13amd64-server/02/usr/tests (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/lib32 on /pool/var/poudriere/data/.m/13amd64-server/02/usr/lib32 (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/src on /pool/var/poudriere/data/.m/13amd64-server/02/usr/src (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/src on /pool/var/poudriere/data/.m/13amd64-server/03/usr/src (nullfs, local, noatime, read-only, nfsv4acls)
devfs on /pool/var/poudriere/data/.m/13amd64-server/02/dev (devfs)
devfs on /pool/var/poudriere/data/.m/13amd64-server/03/dev (devfs)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/tests on /pool/var/poudriere/data/.m/13amd64-server/07/usr/tests (nullfs, local, noatime, read-only, nfsv4acls)
fdescfs on /pool/var/poudriere/data/.m/13amd64-server/01/dev/fd (fdescfs)
fdescfs on /pool/var/poudriere/data/.m/13amd64-server/08/dev/fd (fdescfs)
fdescfs on /pool/var/poudriere/data/.m/13amd64-server/04/dev/fd (fdescfs)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/lib32 on /pool/var/poudriere/data/.m/13amd64-server/07/usr/lib32 (nullfs, local, noatime, read-only, nfsv4acls)
procfs on /pool/var/poudriere/data/.m/13amd64-server/01/proc (procfs, local)
fdescfs on /pool/var/poudriere/data/.m/13amd64-server/05/dev/fd (fdescfs)
procfs on /pool/var/poudriere/data/.m/13amd64-server/08/proc (procfs, local)
procfs on /pool/var/poudriere/data/.m/13amd64-server/04/proc (procfs, local)
procfs on /pool/var/poudriere/data/.m/13amd64-server/05/proc (procfs, local)
linprocfs on /pool/var/poudriere/data/.m/13amd64-server/01/compat/linux/proc (linprocfs, local)
linprocfs on /pool/var/poudriere/data/.m/13amd64-server/08/compat/linux/proc (linprocfs, local)
linprocfs on /pool/var/poudriere/data/.m/13amd64-server/04/compat/linux/proc (linprocfs, local)
/pool/var/poudriere/data/.m/13amd64-server/ref/usr/src on /pool/var/poudriere/data/.m/13amd64-server/07/usr/src (nullfs, local, noatime, read-only, nfsv4acls)
tmpfs on /pool/var/poudriere/data/.m/13amd64-server/01/.p (tmpfs, local)
linprocfs on /pool/var/poudriere/data/.m/13amd64-server/05/compat/linux/proc (linprocfs, local)
tmpfs on /pool/var/poudriere/data/.m/13amd64-server/08/.p (tmpfs, local)
tmpfs on /pool/var/poudriere/data/.m/13amd64-server/04/.p (tmpfs, local)
devfs on /pool/var/poudriere/data/.m/13amd64-server/07/dev (devfs)
tmpfs on /pool/var/poudriere/data/.m/13amd64-server/05/.p (tmpfs, local)
tmpfs on /pool/var/poudriere/data/.m/13amd64-server/08/wrkdirs (tmpfs, local)
tmpfs on /pool/var/poudriere/data/.m/13amd64-server/01/wrkdirs (tmpfs, local)
tmpfs on /pool/var/poudriere/data/.m/13amd64-server/04/wrkdirs (tmpfs, local)
tmpfs on /pool/var/poudriere/data/.m/13amd64-server/05/wrkdirs (tmpfs, local)
/pool/var/poudriere/ports/server on /pool/var/poudriere/data/.m/13amd64-server/04/usr/ports (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/ports/server on /pool/var/poudriere/data/.m/13amd64-server/08/usr/ports (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/ports/server on /pool/var/poudriere/data/.m/13amd64-server/05/usr/ports (nullfs, local, noatime, read-only, nfsv4acls)
fdescfs on /pool/var/poudriere/data/.m/13amd64-server/02/dev/fd (fdescfs)
fdescfs on /pool/var/poudriere/data/.m/13amd64-server/03/dev/fd (fdescfs)
/pool/var/poudriere/data/packages/13amd64-server/.building on /pool/var/poudriere/data/.m/13amd64-server/08/packages (nullfs, local, noatime, read-only, nfsv4acls)
procfs on /pool/var/poudriere/data/.m/13amd64-server/02/proc (procfs, local)
procfs on /pool/var/poudriere/data/.m/13amd64-server/03/proc (procfs, local)
/pool/var/poudriere/data/packages/13amd64-server/.building on /pool/var/poudriere/data/.m/13amd64-server/04/packages (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/packages/13amd64-server/.building on /pool/var/poudriere/data/.m/13amd64-server/05/packages (nullfs, local, noatime, read-only, nfsv4acls)
/pool/source/FreeBSD/ports/distfiles on /pool/var/poudriere/data/.m/13amd64-server/08/distfiles (nullfs, local, nfsv4acls)
/pool/source/FreeBSD/ports/distfiles on /pool/var/poudriere/data/.m/13amd64-server/04/distfiles (nullfs, local, nfsv4acls)
/pool/source/FreeBSD/ports/distfiles on /pool/var/poudriere/data/.m/13amd64-server/05/distfiles (nullfs, local, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/var/db/ports on /pool/var/poudriere/data/.m/13amd64-server/08/var/db/ports (nullfs, local, read-only)
/pool/var/poudriere/data/.m/13amd64-server/ref/var/db/ports on /pool/var/poudriere/data/.m/13amd64-server/05/var/db/ports (nullfs, local, read-only)
/pool/var/poudriere/ports/server on /pool/var/poudriere/data/.m/13amd64-server/01/usr/ports (nullfs, local, noatime, read-only, nfsv4acls)
fdescfs on /pool/var/poudriere/data/.m/13amd64-server/07/dev/fd (fdescfs)
/pool/var/poudriere/data/packages/13amd64-server/.building on /pool/var/poudriere/data/.m/13amd64-server/01/packages (nullfs, local, noatime, read-only, nfsv4acls)
procfs on /pool/var/poudriere/data/.m/13amd64-server/07/proc (procfs, local)
linprocfs on /pool/var/poudriere/data/.m/13amd64-server/07/compat/linux/proc (linprocfs, local)
/pool/source/FreeBSD/ports/distfiles on /pool/var/poudriere/data/.m/13amd64-server/01/distfiles (nullfs, local, nfsv4acls)
tmpfs on /pool/var/poudriere/data/.m/13amd64-server/07/.p (tmpfs, local)
linprocfs on /pool/var/poudriere/data/.m/13amd64-server/03/compat/linux/proc (linprocfs, local)
linprocfs on /pool/var/poudriere/data/.m/13amd64-server/02/compat/linux/proc (linprocfs, local)
tmpfs on /pool/var/poudriere/data/.m/13amd64-server/07/wrkdirs (tmpfs, local)
/pool/var/poudriere/data/.m/13amd64-server/ref/var/db/ports on /pool/var/poudriere/data/.m/13amd64-server/01/var/db/ports (nullfs, local, read-only)
tmpfs on /pool/var/poudriere/data/.m/13amd64-server/02/.p (tmpfs, local)
tmpfs on /pool/var/poudriere/data/.m/13amd64-server/03/.p (tmpfs, local)
tmpfs on /pool/var/poudriere/data/.m/13amd64-server/02/wrkdirs (tmpfs, local)
tmpfs on /pool/var/poudriere/data/.m/13amd64-server/03/wrkdirs (tmpfs, local)
/pool/var/poudriere/ports/server on /pool/var/poudriere/data/.m/13amd64-server/07/usr/ports (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/var/db/ports on /pool/var/poudriere/data/.m/13amd64-server/04/var/db/ports (nullfs, local, read-only)
/pool/var/poudriere/ports/server on /pool/var/poudriere/data/.m/13amd64-server/02/usr/ports (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/ports/server on /pool/var/poudriere/data/.m/13amd64-server/03/usr/ports (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/packages/13amd64-server/.building on /pool/var/poudriere/data/.m/13amd64-server/07/packages (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/packages/13amd64-server/.building on /pool/var/poudriere/data/.m/13amd64-server/02/packages (nullfs, local, noatime, read-only, nfsv4acls)
/pool/var/poudriere/data/packages/13amd64-server/.building on /pool/var/poudriere/data/.m/13amd64-server/03/packages (nullfs, local, noatime, read-only, nfsv4acls)
/pool/source/FreeBSD/ports/distfiles on /pool/var/poudriere/data/.m/13amd64-server/07/distfiles (nullfs, local, nfsv4acls)
/pool/source/FreeBSD/ports/distfiles on /pool/var/poudriere/data/.m/13amd64-server/02/distfiles (nullfs, local, nfsv4acls)
/pool/source/FreeBSD/ports/distfiles on /pool/var/poudriere/data/.m/13amd64-server/03/distfiles (nullfs, local, nfsv4acls)
/pool/var/poudriere/data/.m/13amd64-server/ref/var/db/ports on /pool/var/poudriere/data/.m/13amd64-server/07/var/db/ports (nullfs, local, read-only)
/pool/var/poudriere/data/.m/13amd64-server/ref/var/db/ports on /pool/var/poudriere/data/.m/13amd64-server/02/var/db/ports (nullfs, local, read-only)
/pool/var/poudriere/data/.m/13amd64-server/ref/var/db/ports on /pool/var/poudriere/data/.m/13amd64-server/03/var/db/ports (nullfs, local, read-only)
/pool/var/poudriere/data/packages/13amd64-server/.building/.npkg/librsvg2-rust-2.52.5 on /pool/var/poudriere/data/.m/13amd64-server/02/.npkg (nullfs, local, noatime, nfsv4acls)
/pool/var/poudriere/data/packages/13amd64-server/.building/.npkg/cargo-c-0.9.6 on /pool/var/poudriere/data/.m/13amd64-server/01/.npkg (nullfs, local, noatime, nfsv4acls)

I attempted the first one, and umount reported the device as busy.

Code:

root@xxx:/home/localuser # umount /pool/var/poudriere/jails/13amd64/rescue
umount: unmount of /pool/var/poudriere/data/.m/13amd64-server/ref/rescue failed: Device busy

I assumed that the others would be like this as well. Since I'm the only person who uses the serivces of this machine, I rebooted the FreeBSD machine via shutdown -r now.

After reboot, there were a lot less mountpoints for poudriere:

Code:

pool/poudriere on /pool/poudriere (zfs, local, nfsv4acls)
pool/poudriere/data on /pool/var/poudriere/data (zfs, local, noatime, nfsv4acls)
pool/poudriere/data/packages on /pool/var/poudriere/data/packages (zfs, local, noatime, nfsv4acls)
pool/poudriere/jails on /pool/poudriere/jails (zfs, local, nfsv4acls)
pool/poudriere/jails/13amd64 on /pool/var/poudriere/jails/13amd64 (zfs, local, noatime, nfsv4acls)
pool/poudriere/ports on /pool/poudriere/ports (zfs, local, nfsv4acls)
pool/poudriere/ports/server on /pool/var/poudriere/ports/server (zfs, local, noatime, nfsv4acls)
pool/var/poudriere-base on /pool/var/poudriere-base (zfs, local, nfsv4acls)

I attempted to remove the 13amd64 jail manually using poudriere jail -d -j 13amd64, but as before this errored out with no such jail.

I then attempted to remove manually using rm:

Code:

root@xxx:~/poudriere # rm -rf /pool/var/poudriere/jails/13amd64
rm: /pool/var/poudriere/jails/13amd64/var/empty: Operation not permitted
rm: /pool/var/poudriere/jails/13amd64/var: Directory not empty
rm: /pool/var/poudriere/jails/13amd64/libexec/ld-elf32.so.1: Operation not permitted
rm: /pool/var/poudriere/jails/13amd64/libexec/ld-elf.so.1: Operation not permitted
rm: /pool/var/poudriere/jails/13amd64/libexec: Directory not empty
rm: /pool/var/poudriere/jails/13amd64/usr/lib/librt.so.1: Operation not permitted
rm: /pool/var/poudriere/jails/13amd64/usr/lib: Directory not empty
rm: /pool/var/poudriere/jails/13amd64/usr/lib32/libcrypt.so.5: Operation not permitted
rm: /pool/var/poudriere/jails/13amd64/usr/lib32/libthr.so.3: Operation not permitted
rm: /pool/var/poudriere/jails/13amd64/usr/lib32/librt.so.1: Operation not permitted
rm: /pool/var/poudriere/jails/13amd64/usr/lib32/libc.so.7: Operation not permitted
rm: /pool/var/poudriere/jails/13amd64/usr/lib32: Directory not empty
rm: /pool/var/poudriere/jails/13amd64/usr/bin/chpass: Operation not permitted
rm: /pool/var/poudriere/jails/13amd64/usr/bin/opiepasswd: Operation not permitted
rm: /pool/var/poudriere/jails/13amd64/usr/bin/opieinfo: Operation not permitted
rm: /pool/var/poudriere/jails/13amd64/usr/bin/crontab: Operation not permitted
rm: /pool/var/poudriere/jails/13amd64/usr/bin/su: Operation not permitted
rm: /pool/var/poudriere/jails/13amd64/usr/bin/passwd: Operation not permitted
rm: /pool/var/poudriere/jails/13amd64/usr/bin/login: Operation not permitted
rm: /pool/var/poudriere/jails/13amd64/usr/bin: Directory not empty
rm: /pool/var/poudriere/jails/13amd64/usr: Directory not empty
rm: /pool/var/poudriere/jails/13amd64/sbin/init: Operation not permitted
rm: /pool/var/poudriere/jails/13amd64/sbin: Directory not empty
rm: /pool/var/poudriere/jails/13amd64/lib/libcrypt.so.5: Operation not permitted
rm: /pool/var/poudriere/jails/13amd64/lib/libc.so.7: Operation not permitted
rm: /pool/var/poudriere/jails/13amd64/lib/libthr.so.3: Operation not permitted
rm: /pool/var/poudriere/jails/13amd64/lib: Directory not empty
rm: /pool/var/poudriere/jails/13amd64: Device busy

Eternal_Noob and Sir Dice made a post to alter the system immutable flag, which I used below:

Code:

root@xxx:~/poudriere # chflags -R noschg /pool/var/poudriere/jails/13amd64

I then got an error that the device was already busy:

Code:

root@xxx:~/poudriere # rm -rf /pool/var/poudriere/jails/13amd64
rm: /pool/var/poudriere/jails/13amd64: Device busy

Figuring it was another mount point, I referred to the previous mount points above and used zfs destroy to remove the underlying strucutre:

Code:

root@xxx:~/poudriere # zfs destroy pool/poudriere/jails/13amd64

I then attempted to create the jail again. Poudriere didn't like there was a directory with that name already present:

Code:

root@xxx:~/poudriere # poudriere jail -c -j 13amd64 -v 13.0-RELEASE
[00:00:00] Error: Directory /pool/var/poudriere/jails/13amd64 already exists

So I removed it:

Code:

root@xxx:~/poudriere # rm -r /pool/var/poudriere/jails/13amd64

Which enabled me to finally create the jail:

Code:

root@xxx:~/poudriere # poudriere jail -c -j 13amd64 -v 13.0-RELEASE
[00:00:00] Creating 13amd64 fs at /pool/var/poudriere/jails/13amd64... done
[00:00:00] Using pre-distributed MANIFEST for FreeBSD 13.0-RELEASE amd64
[00:00:00] Fetching base for FreeBSD 13.0-RELEASE amd64
/pool/var/poudriere/jails/13amd64/fromftp/base         180 MB   12 MBps    14s
[00:00:16] Extracting base... done
[00:00:31] Fetching src for FreeBSD 13.0-RELEASE amd64
/pool/var/poudriere/jails/13amd64/fromftp/src.         153 MB 7345 kBps    22s
[00:00:53] Extracting src... done
[00:01:09] Fetching lib32 for FreeBSD 13.0-RELEASE amd64
/pool/var/poudriere/jails/13amd64/fromftp/lib3          67 MB 9927 kBps    07s
[00:01:16] Extracting lib32... done
[00:01:22] Cleaning up... done
[00:01:22] Recording filesystem state for clean... done
[00:01:23] Upgrading using http
/etc/resolv.conf -> /pool/var/poudriere/jails/13amd64/etc/resolv.conf
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching public key from update1.freebsd.org... done.
Fetching metadata signature for 13.0-RELEASE from update1.freebsd.org... done.
Fetching metadata index... done.
Fetching metadata files... done.
Inspecting system... done.
Preparing to download files... done.
Fetching patches... done.
Applying patches... done.
Fetching files... done.
[...]
13.0-RELEASE-p5
[00:02:00] Recording filesystem state for clean... done
[00:02:00] Jail 13amd64 13.0-RELEASE-p5 amd64 is ready to be used

Now, I'm using tmux() happily now on my remote machine that's building packages, and will test out its resilience by intentionally breaking network connection and seeing what happens.

Poudriere: How to Clean up a Dropped SSH Session During a Build?

SomePoorBastard

zirias@

Erichans

astyle

Alain De Vos

SomePoorBastard