Conflicting periodic tasks slowing system

byrnejb

byrnejb

I ran across this while investigating a different problem:
Code: 
[root@vhost06 ~ (master)]# ps -auwx -O jid -d | grep 'USER\|periodic'
USER         PID  %CPU %MEM     VSZ     RSS TT  STAT STARTED         TIME COMMAND            PID JID TT  STAT         TIME COMMAND
root        6623   0.0  0.0   10684    1852  -  IJ   03:03        0:00.00 lockf -t 0 /var/  6623   3  -  IJ        0:00.00 |-- lockf -t 0 /var/run/periodic.daily.lock 
root        6650   0.0  0.0   11712    2548  -  IJ   03:03        0:00.00 /bin/sh /usr/sbi  6650   3  -  IJ        0:00.00 | `-- /bin/sh /usr/sbin/periodic LOCKED dail
root        7028   0.0  0.0   11780    2556  -  IJ   03:03        0:00.01 /bin/sh /usr/sbi  7028   3  -  IJ        0:00.01 |   |-- /bin/sh /usr/sbin/periodic LOCKED da
root       15394   0.0  0.0   11720    2552  -  IJ   03:03        0:00.00 /bin/sh /etc/per 15394   3  -  IJ        0:00.00 |   | `-- /bin/sh /etc/periodic/daily/450.st
root       15562   0.0  0.0   11712    2548  -  IJ   03:03        0:00.00 /bin/sh - /usr/s 15562   3  -  IJ        0:00.00 |   |   `-- /bin/sh - /usr/sbin/periodic sec
root       15780   0.0  0.0   10684    1852  -  IJ   03:03        0:00.00 lockf -t 0 /var/ 15780   3  -  IJ        0:00.00 |   |     `-- lockf -t 0 /var/run/periodic.s
root       15782   0.0  0.0   11712    2548  -  IJ   03:03        0:00.00 /bin/sh /usr/sbi 15782   3  -  IJ        0:00.00 |   |       `-- /bin/sh /usr/sbin/periodic L
root       16355   0.0  0.0   11760    2556  -  IJ   03:03        0:00.01 /bin/sh /usr/sbi 16355   3  -  IJ        0:00.01 |   |         |-- /bin/sh /usr/sbin/periodic
root        7924   0.0  0.0   11720    2592  -  I    03:06        0:00.00 /bin/sh /etc/per  7924   0  -  I         0:00.00 |-- /bin/sh /etc/periodic/daily/450.status-s
root        8167   0.0  0.0   11712    2588  -  I    03:06        0:00.00 /bin/sh - /usr/s  8167   0  -  I         0:00.00 | `-- /bin/sh - /usr/sbin/periodic security
root        8411   0.0  0.0   10684    1892  -  I    03:06        0:00.00 lockf -t 0 /var/  8411   0  -  I         0:00.00 |   `-- lockf -t 0 /var/run/periodic.securit
root        8412   0.0  0.0   11712    2588  -  I    03:06        0:00.00 /bin/sh /usr/sbi  8412   0  -  I         0:00.00 |     `-- /bin/sh /usr/sbin/periodic LOCKED 
root        8864   0.0  0.0   11760    2592  -  I    03:06        0:00.00 /bin/sh /usr/sbi  8864   0  -  I         0:00.00 |       |-- /bin/sh /usr/sbin/periodic LOCKE
root        8973   0.0  0.0   11800    2604  -  I    03:06        0:00.00 /bin/sh - /etc/p  8973   0  -  I         0:00.00 |       | `-- /bin/sh - /etc/periodic/securi
root        9335   0.0  0.0   11800    2600  -  I    03:06        0:00.00 /bin/sh - /etc/p  9335   0  -  I         0:00.00 |       |   `-- /bin/sh - /etc/periodic/secu
root       20573   0.0  0.0   10684    1908  -  IJ   03:04        0:00.00 lockf -t 0 /var/ 20573   5  -  IJ        0:00.00 |-- lockf -t 0 /var/run/periodic.daily.lock 
root       20652   0.0  0.0   11712    2604  -  IJ   03:04        0:00.00 /bin/sh /usr/sbi 20652   5  -  IJ        0:00.00 | `-- /bin/sh /usr/sbin/periodic LOCKED dail
root       21204   0.0  0.0   11780    2616  -  IJ   03:04        0:00.01 /bin/sh /usr/sbi 21204   5  -  IJ        0:00.01 |   |-- /bin/sh /usr/sbin/periodic LOCKED da
root       28975   0.0  0.0   11720    2608  -  IJ   03:04        0:00.00 /bin/sh /etc/per 28975   5  -  IJ        0:00.00 |   | `-- /bin/sh /etc/periodic/daily/450.st
root       29100   0.0  0.0   11712    2604  -  IJ   03:04        0:00.00 /bin/sh - /usr/s 29100   5  -  IJ        0:00.00 |   |   `-- /bin/sh - /usr/sbin/periodic sec
root       29232   0.0  0.0   10684    1908  -  IJ   03:04        0:00.00 lockf -t 0 /var/ 29232   5  -  IJ        0:00.00 |   |     `-- lockf -t 0 /var/run/periodic.s
root       29345   0.0  0.0   11712    2604  -  IJ   03:04        0:00.00 /bin/sh /usr/sbi 29345   5  -  IJ        0:00.00 |   |       `-- /bin/sh /usr/sbin/periodic L
root       29997   0.0  0.0   11760    2612  -  IJ   03:04        0:00.01 /bin/sh /usr/sbi 29997   5  -  IJ        0:00.01 |   |         |-- /bin/sh /usr/sbin/periodic
root       31285   0.0  0.0   10684    1844  -  IJ   03:04        0:00.00 lockf -t 0 /var/ 31285  11  -  IJ        0:00.00 |-- lockf -t 0 /var/run/periodic.daily.lock 
root       31381   0.0  0.0   11712    2552  -  IJ   03:04        0:00.00 /bin/sh /usr/sbi 31381  11  -  IJ        0:00.00 | `-- /bin/sh /usr/sbin/periodic LOCKED dail
root       31699   0.0  0.0   11780    2564  -  IJ   03:04        0:00.01 /bin/sh /usr/sbi 31699  11  -  IJ        0:00.01 |   |-- /bin/sh /usr/sbin/periodic LOCKED da
root       46756   0.0  0.0   11720    2556  -  IJ   03:04        0:00.00 /bin/sh /etc/per 46756  11  -  IJ        0:00.00 |   | `-- /bin/sh /etc/periodic/daily/450.st
root       46998   0.0  0.0   11712    2552  -  IJ   03:04        0:00.00 /bin/sh - /usr/s 46998  11  -  IJ        0:00.00 |   |   `-- /bin/sh - /usr/sbin/periodic sec
root       47177   0.0  0.0   10684    1844  -  IJ   03:04        0:00.00 lockf -t 0 /var/ 47177  11  -  IJ        0:00.00 |   |     `-- lockf -t 0 /var/run/periodic.s
root       47179   0.0  0.0   11712    2552  -  IJ   03:04        0:00.00 /bin/sh /usr/sbi 47179  11  -  IJ        0:00.00 |   |       `-- /bin/sh /usr/sbin/periodic L
root       47747   0.0  0.0   11760    2560  -  IJ   03:04        0:00.01 /bin/sh /usr/sbi 47747  11  -  IJ        0:00.01 |   |         |-- /bin/sh /usr/sbin/periodic
root        5968   0.0  0.0   11288    2368  1  S+   09:21        0:00.00 grep --color=aut  5968   0  1  S+        0:00.00 |       `-- grep --color=auto USER\\|periodi
root       37019   0.0  0.0   10684    1804  -  IJ   03:04        0:00.00 lockf -t 0 /var/ 37019  10  -  IJ        0:00.00 |-- lockf -t 0 /var/run/periodic.daily.lock 
root       37184   0.0  0.0   11712    2500  -  IJ   03:04        0:00.00 /bin/sh /usr/sbi 37184  10  -  IJ        0:00.00 | `-- /bin/sh /usr/sbin/periodic LOCKED dail
root       39742   0.0  0.0   11780    2508  -  IJ   03:04        0:00.01 /bin/sh /usr/sbi 39742  10  -  IJ        0:00.01 |   |-- /bin/sh /usr/sbin/periodic LOCKED da
root       49969   0.0  0.0   11720    2504  -  IJ   03:04        0:00.00 /bin/sh /etc/per 49969  10  -  IJ        0:00.00 |   | `-- /bin/sh /etc/periodic/daily/450.st
root       50090   0.0  0.0   11712    2500  -  IJ   03:04        0:00.00 /bin/sh - /usr/s 50090  10  -  IJ        0:00.00 |   |   `-- /bin/sh - /usr/sbin/periodic sec
root       50257   0.0  0.0   10684    1804  -  IJ   03:04        0:00.00 lockf -t 0 /var/ 50257  10  -  IJ        0:00.00 |   |     `-- lockf -t 0 /var/run/periodic.s
root       50380   0.0  0.0   11712    2500  -  IJ   03:04        0:00.00 /bin/sh /usr/sbi 50380  10  -  IJ        0:00.00 |   |       `-- /bin/sh /usr/sbin/periodic L
root       50911   0.0  0.0   11760    2508  -  IJ   03:04        0:00.01 /bin/sh /usr/sbi 50911  10  -  IJ        0:00.01 |   |         |-- /bin/sh /usr/sbin/periodic
root       58462   0.0  0.0   11712    2520  -  IJ   03:21        0:00.00 /bin/sh /etc/per 58462   2  -  IJ        0:00.00 |-- /bin/sh /etc/periodic/daily/450.status-s
root       58508   0.0  0.0   11704    2516  -  IJ   03:21        0:00.00 /bin/sh - /usr/s 58508   2  -  IJ        0:00.00 | `-- /bin/sh - /usr/sbin/periodic security
root       58696   0.0  0.0   10676    1812  -  IJ   03:21        0:00.00 lockf -t 0 /var/ 58696   2  -  IJ        0:00.00 |   `-- lockf -t 0 /var/run/periodic.securit
root       58710   0.0  0.0   11704    2516  -  IJ   03:21        0:00.00 /bin/sh /usr/sbi 58710   2  -  IJ        0:00.00 |     `-- /bin/sh /usr/sbin/periodic LOCKED 
root       59333   0.0  0.0   11752    2520  -  IJ   03:21        0:00.01 /bin/sh /usr/sbi 59333   2  -  IJ        0:00.01 |       |-- /bin/sh /usr/sbin/periodic LOCKE
root       75892   0.0  0.0   10684    1884  -  IJ   03:05        0:00.00 lockf -t 0 /var/ 75892   9  -  IJ        0:00.00 |-- lockf -t 0 /var/run/periodic.daily.lock 
root       75966   0.0  0.0   11712    2584  -  IJ   03:05        0:00.00 /bin/sh /usr/sbi 75966   9  -  IJ        0:00.00 | `-- /bin/sh /usr/sbin/periodic LOCKED dail
root       76675   0.0  0.0   11780    2596  -  IJ   03:05        0:00.01 /bin/sh /usr/sbi 76675   9  -  IJ        0:00.01 |   |-- /bin/sh /usr/sbin/periodic LOCKED da
root       27840   0.0  0.0   11720    2592  -  IJ   03:11        0:00.00 /bin/sh /etc/per 27840   9  -  IJ        0:00.00 |   | `-- /bin/sh /etc/periodic/daily/450.st
root       28041   0.0  0.0   11712    2588  -  IJ   03:11        0:00.00 /bin/sh - /usr/s 28041   9  -  IJ        0:00.00 |   |   `-- /bin/sh - /usr/sbin/periodic sec
root       28260   0.0  0.0   10684    1884  -  IJ   03:11        0:00.00 lockf -t 0 /var/ 28260   9  -  IJ        0:00.00 |   |     `-- lockf -t 0 /var/run/periodic.s
root       28311   0.0  0.0   11712    2588  -  IJ   03:11        0:00.00 /bin/sh /usr/sbi 28311   9  -  IJ        0:00.00 |   |       `-- /bin/sh /usr/sbin/periodic L
root       28981   0.0  0.0   11760    2596  -  IJ   03:11        0:00.00 /bin/sh /usr/sbi 28981   9  -  IJ        0:00.00 |   |         |-- /bin/sh /usr/sbin/periodic
root       51476   0.0  0.0   11720    2592  -  IJ   07:28        0:00.00 /bin/sh - /etc/p 51476   9  -  IJ        0:00.00 |   |         | `-- /bin/sh - /etc/periodic/

We have about eight jails running on this host. This number has no varied by more that two for the past couple of years. Within the last week we have experienced significant response degradation on jails running on this host. Looking into ps reveals a number of what, to me, looked like abnormal periodic tasks that are LOCKED.

I recently altered the /etc/crontab files the jails to remove a superfluous reference to atrun. I also altered the crontab file to use jot to stagger the start of the periodic tasks by a random value.

My questions are:

Should I kill these tasks?

How is this prevented?

Is there a maximum number of jails that can be hosted on a single system. None of these jails are providing intensive services. They exist primarily to segregate access from public facing services.
 
byrnejb said:
I also altered the crontab file to use jot to stagger the start of the periodic tasks by a random value.
Click to expand...
That's good. You don't really want all your jails to start the periodic tasks all at the same time. Those periodic jobs are quite I/O intense (lots of find(1) being used), so spreading these out is good.

byrnejb said:
Should I kill these tasks?
Click to expand...
Your call. You can let it limp along, eventually it'll finish (but this might take a while). Or kill them now and accept some broken reports.


byrnejb said:
How is this prevented?
Click to expand...
By staggering the jobs as you already did. That will prevent them from trying to run all at once.
 
First step: Find out what these tasks are doing. The printout you have above seems to tell me that they are running daily 450, which on my system is security.

Then look at the various security tasks. Do you really need to run all of them? Quite a few of the periodic security scripts (which you can find in /etc/periodic/security) run a find over the whole file system, for example to find executables that are setuid. But a jail has the same file system as the host. So are they needed to be run in jails at all, since anything they would report is already present in the report of the main host?
 
You must log in or register to reply here.
Back
Top