Solved a puzzle with missing file and cron

[aragats]

An hour I started getting emails with errors reporting by a cron() task:

/usr/local/www/misc/eldora/get_image.sh: convert: not found

I logged in my VPS and found that indeed, the file convert doesn't exist. It's supposed to belong to graphics/ImageMagick7 or to one of its flavors. So, I checked pkg info and found it's not installed. Okay, it could happen during updates/upgrades whatever. But why I have never received those error messages before? I didn't touch anything in this VPS for long time. I checked the security reports – everything looks good.

I'm using Gmail to relay emails, so checked the "Sent" folder and found just those few messages from today.

What could happen? What am I missing? How to track this issue down?
FreeBSD 12.0-RELEASE-p10

Thanks for ideas!

 

[ljboiler]

pkg should log install/update/delete actions to /var/log/messages (or the compressed archived versions messages.X.bz). I'd start there...

grep 'pkg\['

 

[aragats]

pkg should log install/update/delete actions to /var/log/messages

That makes sense, thanks!

% grep -i magic /var/log/messages
Sep 12 21:02:35 eu pkg[53351]: ImageMagick-6.9.9.28_2,1 deinstalled

However, it doesn't explain why no emails were sent since Sept 12 until today (the cron task runs every 5 minutes).

 

[SirDice]

Check /var/log/cron to see if it actually ran or not. Then check /var/log/maillog, maybe the mail got stuck in the queue, I've had this happen a couple times after an upgrade (it messed up my mail configuration).

 

[aragats]

Check /var/log/cron to see if it actually ran or not. Then check /var/log/maillog, maybe the mail got stuck in the queue

Just checked: cron did run, the mail queue is empty, /var/log/maillog* files don't have any references besides those emails which suddenly appeared yesterday after 12 days of the non-existing file...

 

[SirDice]

/var/log/maillog* files don't have any references besides those emails which suddenly appeared yesterday after 12 days of the non-existing file...

Was sendmail running during that time? If there's no MTA there's nowhere for the cron output to go. You should, at the very least, see the daily periodic emails, unless you disabled those.

 

[aragats]

Yes, I receive the daily periodic emails every day, also there are other cron tasks which send me notifications/reminders etc on purpose (not stdout/stderr).
The MTA is mail/ssmtp.

 

[ralphbsz]

Suggestion: Put a known broken command into cron. For example something where the path is wrong (execute /usr/local/bin/foo, and make sure it doesn't exist). Also put something that "works" but fail, for example put this:

#/bin/sh
echo Hallo and goodbye > /dev/stderr
exit 1

and run that from your cron. Make sure you get e-mails each time.

 

[aragats]

Put a known broken command into cron ... Also put something that "works" but fail

I checked both and they worked, then realized it must be a problem in my script. And indeed – I found a condition which has never been triggered before!

That script was pulling screenshots from a web camera at a ski area, and was very useful in winter season. Then they stopped that webcam on one day in April. Now (yesterday) it started working again, but with different parameters which triggered that condition!

No magic – problem solved!

 

[ralphbsz]

Software broken by a snowflake. Seems like a common problem. Right up there with software that breaks because a butterfly flaps it wings.

 

[aragats]

Software broken by a snowflake.

Software is not broken. The issue itself appeared when a package was removed during update process – so it's an operator's omission...
I don't think it's a common practice to check for all executables' existence in shell scripts.

 

[ralphbsz]

No, I agree, and for the shells scripts I write at home I don't do that either.

For the stuff I write at work, we have tests. Both unit tests (where an automated test runs, with fake input data, and we make sure the output is 100% identical to what is expected, and we require that at least 90% of all lines are executed, both in scripts and in programs). And system tests, where human testers set up and run software and systems. They first try to run the system according to the manual, make sure the good case works correctly. Then they try to break it, for example with insane inputs.