Intense I/O: found pkg and find running at night

[lifepillar]

Last night at about 3am I noticed my server doing a lot of I/O. Top showed first pkg then find being run by root (not in a jail). After several minutes they stopped. Is there a periodic task that is supposed to run those processes? All my services run inside jails. Root has no crontab. Running FreeBSD 13.1-RELEASE-p5.

 

[covacat]

security audit searches for setuids, pkg audit

 

[cracauer@]

Also locate(1) database building. Although that should run weekly on the weekend.

 

[bsduck]

Root has no crontab.

So you removed the default /etc/crontab ?

 

[Alain De Vos]

I just disabled the security stuff in /etc/periodic.conf

daily_status_security_enable="NO"            # Security check
monthly_status_security_enable="NO"            # Security check
security_status_logincheck_enable="NO"
weekly_status_security_enable="NO"            # Security check

 

[ralphbsz]

Several other people already hinted at this:

All my services run inside jails. Root has no crontab.

Periodic is not a service, it is run from the crontab in /etc/crontab. It is not vital (the system won't crash without it), but everything in there is a good idea and there for a reason, even the security audits. You might want to think about re-enabling it.

 

[astyle]

So you removed the default /etc/crontab ?

I take the "root has no crontab" to mean that /etc/crontab simply doesn't have entries for tasks to be run by user root. And it probably shouldn't, it's not a good practice to have root run much of anything from crontab. There are special daemon accounts for that.

Having pkg process run by root, esp. not in a jail, in the middle of the night - that's a cause for concern, even if it were done deliberately by OP (and it's definitely not deliberate). Such things should be programmed with intent, and monitored to make sure they don't have an un-intended effect... in short, some paranoia is justifiable here.

 

[lifepillar]

I take the "root has no crontab" to mean that /etc/crontab simply doesn't have entries for tasks to be run by user root.

Yes, that was badly worded. I meant that I haven't added anything to root's crontab. My `/etc/crontab` is pristine.

Thanks for pointing to the security audits—it was probably setuid checks. Where do I find when those tasks are run? Well, I think I can do some homework and find it myself ?.

This community never disappoints!

 

[SirDice]

Where do I find when those tasks are run?

periodic(8):

# Perform daily/weekly/monthly maintenance.
1       3       *       *       *       root    periodic daily
15      4       *       *       6       root    periodic weekly
30      5       1       *       *       root    periodic monthly

If you have a lot of jails running, it'll be a good idea to move the timing around a bit. That way they won't all start at the same time and choke the system.

 

[T-Daemon]

Last night at about 3am I noticed my server doing a lot of I/O. Top showed first pkg then find being run by root (not in a jail). After several minutes they stopped. Is there a periodic task that is supposed to run those processes?

I've notice the same on my system. It was after midnight, early in the morning when I heard the laptops fan begun to spun up. top(1) showed pkg(8) using a lot of WCPU. Thereafter I saw xz(1) doing the same.

grep(1)ing "pkg" and "xz" through /etc/periodic didn't return anything, but /usr/local/etc/periodic got a hit on /usr/local/etc/periodic/daily/411.pkg-backup.

That script backs up the pkg sql database.

/var/backup

-rw-r--r--  1 root  wheel   6.5M Feb 28 03:40 pkg.sql.xz
-rw-r--r--  1 root  wheel   6.5M Feb 24 03:17 pkg.sql.xz.1
-rw-r--r--  1 root  wheel   6.1M Feb 23 03:02 pkg.sql.xz.2
-rw-r--r--  1 root  wheel   6.1M Feb 22 03:02 pkg.sql.xz.3
-rw-r--r--  1 root  wheel   6.1M Feb 21 03:18 pkg.sql.xz.4
-rw-r--r--  1 root  wheel   6.1M Feb 16 03:02 pkg.sql.xz.5
-rw-r--r--  1 root  wheel   6.1M Feb 15 03:29 pkg.sql.xz.6
-rw-r--r--  1 root  wheel   6.1M Feb 10 03:02 pkg.sql.xz.7

Note the time of the timestamps, which matches the periodic daily in /etc/crontab

1       3       *       *       *       root    periodic daily

However, the file size is to small to justify a high CPU usage. Probably all under /usr/local/etc/periodic/security pkg scripts are run as well:

405.pkg-base-audit
410.pkg-audit
460.pkg-checksum

All three have daily periods specified.

 

[SirDice]

So you removed the default /etc/crontab ?

I consider /etc/crontab to be the system's crontab. Each user has it's own crontab; crontab -e. And root is just a user account, with its own individual crontab. Those are stored in /var/cron/tabs/. periodic(8) gets started via the system's crontab; /etc/crontab.

 

[astyle]

I consider /etc/crontab to be the system's crontab. Each user has it's own crontab; crontab -e. And root is just a user account, with its own individual crontab. Those are stored in /var/cron/tabs/. periodic(8) gets started via the system's crontab; /etc/crontab.

Yeah, but wouldn't that "user crontab" only run while the user is actually logged in? (Well, one workaround is to run it in a /usr/local/bin/screen session, which would be useful (but has security implications) if one has a remote VPS )

 

[cracauer@]

No, user crontabs do run when the user is not logged in.

 

[astyle]

No user crontabs run when the user is not logged in.

Sorry, but there are a couple ways this can be interpreted:

1. User crontabs DO NOT run when user is not logged in.
2. What if there's a missing comma in front of the "No" ?

 

[cracauer@]

sorry. Edited. Exceptions apply, for example that insane systemd change where they killed all user processes when the user logged out. Who knows what they did to the crontabs in that version.