Bash script to automate installing bunch of packages

Sergei_Shablovsky

Sergei_Shablovsky

Dear FreeBSD Gurus!
Please share bash script to automate installing bunch of packages without any user interaction in a process.

Let me write a little bit of details what I need exactly:

pkg_autoinstaller_list.txt - text file that contain name of packages that I would to install. Each name in separate row. Full name of package (“zsh-5.2.4”), or short version (“zsh”), case-sensitive.
Looks like this below:
—
zsh
zsh-highlighting
py38-speedtest-cli
librespeed
fping
bpytop
webmin
...
—

pkg-autoinstaller.sh - bash script that I asking for.

pkg_autoinstaller_mmddyyyy_hhmmss.log - log file, that also contain copy of screen output of each pkg installation process for review later.
Name of the file contain data&time of creation.
Each string of log with time stamp.
Summary with Start time and End time, number of possible errors/warnings during the process at the start of this log file, looks like this:
—
Installations summary:
TOTAL packages to install 54
EXIST IN REPO 54
ABORTED 1
SUCCESSFULLY INSTALLED 53
WARNINGS 2
REBOOT REQUIRED


TOTAL packages to install - calculated from pkg_to_install.txt.
EXIST IN REPO - calculated from pkg_to_install.txt that successfully founded in pkg repo.
ABORTED - calculated if in output words “error”, “aborted”, case-sensitive
SUCCESSFULLY INSTALLED = EXIST IN REPO - ABORTED
WARNINGS - calculated if in output word “warning”, case-sensitive
REBOOT REQUIRED - persist if in output word “reboot”, case-sensitive

Script parameters:
-fpkg #force update pkg system
-f #force reinstall package even installed already
-c #ignore case-sensitivity when search in pkg repo
-e #not open log-file in default system editor at the end of installation
-rst #restart system after installation complete if no errors (mean no “Error”, “Failed” in every package from list)
-s #play short beeps (if possible by system mb beeper) on complete after log-file was opened in editor successfully.

Logic: (you may correct or add something):
- clean pkg system cache
- update pkg system before installation
- check/reserve 300kb space for log file
- read each package name from pkg_to_install.txt, install it, write screen output to pkg_autoinstaller.log log file
- [OPT] open log file in editor on tty0 (or where from script started), play short beeps to get attention
- [OPT] standard restart server

Thanks you all another one time!

P.S. I more that sure that this is trivial script, that may be exist in Your scripts library... ;) I still thinking it very useful for SysAdmins that installing BSD frequently.
 
Last edited:
P.P.S.
I know that another one option would be installing some tool both on source server and replica like “pkg manager”, and using pkg list export/import to automate installation.
Anyway, keeping a set of this pkg_list.txt not so different from keeping a set of files to import to this “pkg manager”.

May be a You suggest me really great useful tool.
 
Don't use bash(1), as you'll have to install it 1st. Perhaps you can base it on this
Bash: 
#!/bin/sh
env ASSUME_ALWAYS_YES=yes pkg-static bootstrap
while read pkgname; do
    env ASSUME_ALWAYS_YES=yes pkg install ${pkgname}
done < pkg_list.txt | tee pkg_autoinstaller_$(date +%Y%m%dT%H%M%S).log

Or maybe even
Bash: 
env ASSUME_ALWAYS_YES=yes pkg-static bootstrap
env ASSUME_ALWAYS_YES=yes xargs pkg install < pkg_list.txt | tee pkg_autoinstaller_$(date +%Y%m%dT%H%M%S).log
 
If this is for (mass) deployment, you should either take a look at the scripting capabilities of bsdinstall(8) to create a custom, automated installer image, or consider using something like sysutlis/ansible where you can define a set of packages + configuration for a given type of host (e.g. a webserver) and run that against multiple hosts.
Depending on your use case both may have their advantages and you could also combine them.
 
ondra_knezour said:
sysutils/firstboot-pkgs may be source of some inspiration
Click to expand...

Thank You for suggestion!

From Description:

When the system first boots, install the pkg(8) tools (if not already installed) and packages listed in the $firstboot_pkgs_list rc.conf variable. If the installed packages added new rc.d scripts, request a reboot.
Obviously, this port is not useful after a system is already running; it is intended to be included as part of the installation or disk image building process.

I start this topic from “script” because needs to easy keep “stack” of packages for each type of server (balancer, public web server, fw, etc...).
Adding extra + one pkg to stack are easy - just add pkg name to list, and run script on each server of that type.

No needs to pull off server from working environment, reinstall, then put it back...
 
Geezer said:
ports-mgmt/synth is used to manage ports, and can be configured to use only packages.
Click to expand...

Thank You for suggestions!

From Description:

Synth is a custom packge repository builder for FreeBSD and DragonFly. It is intended to replace Portmaster, portupgrade, and poudriere for the average user. It is simple to learn (the powerful options are limited in number) and user-friendly, but it is extremely fast due to its parallel building capability. It will "drop-in" on any system as it leverages the stock pkg(8) facilities. All ports are built in a clean environment, so it is finally safe to build ports as needed on a live system. The default profile is the system itself, not a new jail, which can be a valuable feature for some environments. To bring a system up-to-date only requires one command after the ports tree is updated: > synth upgrade-system During the building process, a curses-based display will show the status of all the builders and the entire bulk run process. A dynamic and searchable web-based report is generated simultaneously. Synth is intended to be grasped and utilized by novice users within minutes, but offers most of the same powerful features as Poudriere for the power users. Synth requires no preparation; it works immediately upon installation. WWW: https://github.com/jrmarino/synth

Looks like VERY close to what I need. Needed to read more it's docs...
 
sko said:
If this is for (mass) deployment, you should either take a look at the scripting capabilities of bsdinstall(8) to create a custom, automated installer image, or consider using something like sysutlis/ansible where you can define a set of packages + configuration for a given type of host (e.g. a webserver) and run that against multiple hosts.
Depending on your use case both may have their advantages and you could also combine them.
Click to expand...

Thank You for suggestions so much!

A few questions:

1.
I am a little bit newbie in BSD. Could You (or anyone else here :) be so please to explain in short main difference in pro/cons between Ansible, bsdinstall, synth?

2.
From management point of view no difference between needs to keep bunch of BSD install images (one per each type of server), OR needs to keep bunch of “pkg-lists” + pkg management system.

Right now I see only one disadvantage of bsdinstall: each time when FreeBSD rollout new STABLE, I need to re-creating ALL of custom installer images.

Where am I wrong?

3.
Which ansible2 or ansible I need for FreeBSD 13 ?

4.
From RedHat web I see that ansible is paid software ?
I am a little bit disappointed, is it free (as I see it on freshports) or paid (as I see on RH web) ?

Thank You!
 
Bobi B. said:
Don't use bash(1), as you'll have to install it 1st. Perhaps you can base it on this
Bash: 
#!/bin/sh
env ASSUME_ALWAYS_YES=yes pkg-static bootstrap
while read pkgname; do
    env ASSUME_ALWAYS_YES=yes pkg install ${pkgname}
done < pkg_list.txt | tee pkg_autoinstaller_$(date +%Y%m%dT%H%M%S).log

Or maybe even
Bash: 
env ASSUME_ALWAYS_YES=yes pkg-static bootstrap
env ASSUME_ALWAYS_YES=yes xargs pkg install < pkg_list.txt | tee pkg_autoinstaller_$(date +%Y%m%dT%H%M%S).log
Click to expand...

Thank You so much!!!
2 perfect lines of code! Impressive! ;)

Do I need to put in pkg_list.txt full name of package “py38-speedtest-cli-2.4.7”, or shorted version “py38-speedtest-cli” would be enough ?
 
Bobi B. said:
Write the name the same way you would normally do if you called pkg(8) manually: pkg install py38-speedtest-cli, hence py38-speedtest-cli.
Click to expand...
This is going to pose a problem when the default Python version changes. Use the port origin in this case; pkg install net/py-speedtest-cli. Then it will always be installed, regardless of the Python version at that time.
 
Sergei_Shablovsky said:
Right now I see only one disadvantage of bsdinstall: each time when FreeBSD rollout new STABLE, I need to re-creating ALL of custom installer images.
Click to expand...
No
1. why would you want to run stable on servers? Use -RELEASE and quarterly packages until you have a very good reason not to. It makes EVERYTHING much more sane and easier to maintain if you don't have several updates every other day and can just run identical packet versions on all hosts.
2. The 'custom installer image' pretty much is just the stock image + your bsdinstall script. Everything else is pulled from the repositories and configs or additional scripts can also be pulled from e.g. a git repo.
To add the script you just create an md device from the image using mdconfig(8), mount the appropriate partition/slice, add everything you want and you're done:
Code: 
# mdconfig FreeBSD-12.3-RELEASE-amd64-memstick.img
# # gpart show md0
=>      1  2099728  md0  MBR  (1.0G)
        1     1600    1  efi  (800K)
     1601  2098128    2  freebsd  [active]  (1.0G)
# mount /dev/md0s2a /mnt
# ls /mnt
.cshrc          COPYRIGHT       boot/           etc/            libexec/        mnt/            proc/           root/           tmp/            var/
.profile        bin/            dev/            lib/            media/          net/            rescue/         sbin/           usr/

<add / modify anything you want>

# umount /mnt
# mdconfig -d -u md0

If this still is too much work, it can be very easily scripted. But if you use -RELEASE for your servers (as you should), this task is only necessary every few months or even just once a year.

On top of that you can (and probably should, depending on the number of systems) run an orchestration framework like ansible/chef/puppet. Keep the bsdinstall routine as minimal and generic as possible - e.g. only necessary network config; ssh keys, custom pkg repos and hooks to integrate to your config management. Everything on top of that which might be host-specific is then handled by the config management.
There's absolutely no need to keep dozens of install images and the horrors of keeping them all up to date. With something like ansible you just add a new role to a server, maybe write a new config file template and you're done.


Regarding ansible: just use sysutils/ansible if you deploy a new setup. The older versions only make sense if you have to keep compatibility.
 
Bobi B. said:
Don't use bash(1), as you'll have to install it 1st. Perhaps you can base it on this
Bash: 
#!/bin/sh
env ASSUME_ALWAYS_YES=yes pkg-static bootstrap
while read pkgname; do
    env ASSUME_ALWAYS_YES=yes pkg install ${pkgname}
done < pkg_list.txt | tee pkg_autoinstaller_$(date +%Y%m%dT%H%M%S).log

Or maybe even
Bash: 
env ASSUME_ALWAYS_YES=yes pkg-static bootstrap
env ASSUME_ALWAYS_YES=yes xargs pkg install < pkg_list.txt | tee pkg_autoinstaller_$(date +%Y%m%dT%H%M%S).log
Click to expand...
So, if I need use that SirDice wrote, is this scripts working with string in pkg_list.txt ?
—
net/py38-speedtest-cli
net/fping
...
—

Could You be so please to modify a Your script to be able work with?
Thank You so much!
 
The port is net/py-speedtest-cli, that's called the "origin" in FreeBSD's port/package lingo. Depending on the (default) version of Python the name of the resulting package is py${PYTHON_VERSION}-speedtest-cli. If the Python version changes to 3.9 for example the package would be named py39-speedtest-cli. See how that Python version is reflected in the package name? If you try to install py38-speedtest-cli it will fail, because the package is now called py39-speedtest-cli. If you run pkg install net/py-speedtest-cli, it will always be installed, regardless of what the (default) Python version happens to be.

Code: 
     pkg install [--{automatic,force,no-scripts,ignore-missing}]
                 [--{dry-run,fetch-only,quiet,recursive,no-repo-update,yes}]
                 [--repository reponame]
                 [--{case-sensitive,glob,case-insensitive,regex}]
                 <pkg-origin|pkg-name|pkg-name-version> ...
In this case pkg-origin is net/py-speedtest-cli; pkg-name is py38-speedtest-cli and pkg-name-version is py38-speedtest-cli-2.1.3.
 
bakul said:
Can't you just use pkg install -y $(cat pkg_to_install)? Why make it more complicated? In your pkg list file put category/package-name as SirDice suggests. If you want to clear the cache just do rm -rf /var/cache/pkg once in a while.
Click to expand...
Thank You for suggestion!

Is this code doing the same as Bobi B.'s script doing?

And what about pkg_autoinstaller.log ? ;)
 
SirDice said:
The port is net/py-speedtest-cli, that's called the "origin" in FreeBSD's port/package lingo. Depending on the (default) version of Python the name of the resulting package is py38-speedtest-cli. If the Python version changes to 3.9 for example the package would be named py39-speedtest-cli. See how that Python version is reflected in the package name? If you try to install py38-speedtest-cli it will fail, because the package is now called py39-speedtest-cli. If you run pkg install net/py-speedtest-cli, it will always be installed, regardless of what the (default) Python version happens to be.
Click to expand...
Thank You for detailed explaining!

So, following this logic, the Bobi B.' script must using as input “origin” name.
—
net/py-speedtest-cli
net/fping
net/smokeping
...
—

And this would be best variant?
 
If you look at an existing server, to get a list of packages to install, look at the output from pkg prime-list, or pkg prime-origins. You don't want to install everything, you only want to install the packages you need. Dependencies can change over time and they will automatically be installed, so you don't need to install those yourself.
 
SirDice said:
If you look at an existing server, to get a list of packages to install, look at the output from pkg prime-list, or pkg prime-origins. You don't want to install everything, you only want to install the packages you need. Dependencies can change over time and they will automatically be installed, so you don't need to install those yourself.
Click to expand...
Thank You!

Do I need to add to Bobi. B's script at the end some commands to delete old/unused/unlinked dependencies? Or BSD doing that on regular base (cron) by themselve?
 
bakul said:
sudo pkg install -y $(cat pkg_to_install.txt) >& pkg_autoinstaller_$(date +%m%d%Y_%H%m%S).log? I assume this is only for new installs? For regular updates you'd need something else.
Click to expand...

Yes, for new installs.

bakul said:
In zsh I usually have these aliases for date related stuff:

alias ymd='date +%Y%m%d'
alias ymdhms='date -u +%Y%m%d%H%M%S' # for UTC timestamp
Click to expand...
Thanks!

Which main difference in Your and Bobi. B script in real work? Which is better ?
 
OP should just keep it simple. Have something to practice all those ideas on - not the production server, but something that is OK to mess up and delete. Once the results of practice are acceptable, take down some good notes, reproduce the steps for good measure, and only then touch the production stuff.
 
You must log in or register to reply here.
Back
Top