Inactive memory not reallocated - swapping

R

rev

Hello,

My system does not reallocate inactive memory and is swapping. swapoff -a fails with allocation error (as expected). Inactive memory seems to get to the current level after a few days and stays there.

Machine has 32G RAM. Output of top -n:
Code: 
113 processes: 1 running, 112 sleeping

Mem: 251M Active, 27G Inact, 3593M Wired, 513M Cache, 1660M Buf, 165M Free
Swap: 4096M Total, 712M Used, 3384M Free, 17% Inuse


  PID USERNAME      THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
 1459 vbox           24  21    0  1394M  1106M select  1 431:38   4.30% VBoxHeadless
  809 transmission    3  20    0   250M 42028K select  1  28:31   0.10% transmission-daemon
 1294 mysql          25  20    0   802M   189M uwait   0   7:56   0.00% mysqld
22738 www             1  24    0   241M     0K accept  2   0:52   0.00% <php-fpm>
24649 www             1  24    0   229M     0K accept  4   0:43   0.00% <php-fpm>
  805 vbox           11  20    0   140M  4976K uwait   0   0:31   0.00% VBoxSVC
24636 www             1  24    0   221M     0K accept  5   0:17   0.00% <php-fpm>
  803 vbox            1  20    0 83324K  3596K select  0   0:09   0.00% VBoxXPCOMIPCD
 1015 pgsql           1  20    0  4284M  3724K select  5   0:06   0.00% postgres
 1021 pgsql           1  20    0 70432K  3960K select  7   0:06   0.00% postgres
 1017 pgsql           1  20    0  4284M 31936K select  1   0:04   0.00% postgres
  763 root            1  20    0 25328K  2136K select  7   0:04   0.00% ntpd
 1020 pgsql           1  20    0  4284M  4144K select  1   0:04   0.00% postgres
  819 root            2  20    0 18964K   604K nanslp  1   0:04   0.00% sshguard
 1315 jabber          1  20    0 47036K  5644K select  3   0:03   0.00% s2s
97487 www             1  20    0 32592K  2824K kqread  5   0:02   0.00% nginx
  536 _pflogd         1  20    0 14600K  1028K bpf     2   0:02   0.00% pflogd
  788 vbox           11  20    0   157M  4728K uwait   7   0:02   0.00% vboxwebsrv
uname -a:
Code: 
FreeBSD x.y.z 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:59 UTC 2014     root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64
Daemons running:
  • postgresql
  • mysql
  • transmission
  • dovecot
  • postfix
  • spamassassin
  • virtualbox-headless + web
  • gopherd
  • jabberd2
  • nginx
  • php-fpm
  • sshguard
Output from a freebsd-memory perl script from Ralf S. Engelschall:
Code: 
SYSTEM MEMORY INFORMATION:
mem_wire:        3772944384 (   3598MB) [ 11%] Wired: disabled for paging out
mem_active:  +    267919360 (    255MB) [  0%] Active: recently referenced
mem_inactive:+  28555124736 (  27232MB) [ 85%] Inactive: recently not referenced
mem_cache:   +    533000192 (    508MB) [  1%] Cached: almost avail. for allocation
mem_free:    +    173142016 (    165MB) [  0%] Free: fully available for allocation
mem_gap_vm:  +      1081344 (      1MB) [  0%] Memory gap: UNKNOWN
-------------- ------------ ----------- ------
mem_all:     =  33303212032 (  31760MB) [100%] Total real memory managed
mem_gap_sys: +    888053760 (    846MB)        Memory gap: Kernel?!
-------------- ------------ -----------
mem_phys:    =  34191265792 (  32607MB)        Total real memory available
mem_gap_hw:  +    168472576 (    160MB)        Memory gap: Segment Mappings?!
-------------- ------------ -----------
mem_hw:      =  34359738368 (  32768MB)        Total real memory installed

SYSTEM MEMORY SUMMARY:
mem_used:        5098471424 (   4862MB) [ 14%] Logically used memory
mem_avail:   +  29261266944 (  27905MB) [ 85%] Logically available memory
-------------- ------------ ----------- ------
mem_total:   =  34359738368 (  32768MB) [100%] Logically total memory

I have no idea what can cause this kind of behaviour. Can anyone help?
 
When in top, press 'o' to change the column to order on. Try "res" for resident memory usage, and "size" for memory size. See if that helps bring the memory hog to the top of the list. That should help track down the process using all the RAM. Then it's a matter of figuring out why it's holding onto it all.

I have a feeling it will be Postgres (4 GB per process showing in your original top output, although that could be shared).
 
phoenix said:
When in top, press 'o' to change the column to order on. Try "res" for resident memory usage, and "size" for memory size. See if that helps bring the memory hog to the top of the list. That should help track down the process using all the RAM. Then it's a matter of figuring out why it's holding onto it all.
Click to expand...
There is no process using that much memory.
top -n -o res
Code: 
  PID USERNAME      THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
 1459 vbox           24  21    0  1406M  1118M select  1 547:07   4.39% VBoxHeadless
 1294 mysql          25  21    0   814M   168M uwait   6   9:58   0.00% mysqld
  809 transmission    3  20    0   258M 38420K select  2  36:53   0.10% transmission-daemon
 1017 pgsql           1  20    0  4284M 32488K select  6   0:06   0.00% postgres
93068 root            1  20    0   204M 11416K kqread  2   0:01   0.00% php-fpm
48665 root            1  20    0 86084K  5164K select  3   0:00   0.00% sshd
48667 phoenix         1  20    0 86084K  5160K select  4   0:00   0.00% sshd
  805 vbox           11  20    0   140M  5100K uwait   5   0:39   0.00% VBoxSVC
  788 vbox           11  20    0   157M  4676K uwait   7   0:03   0.00% vboxwebsrv
 1315 jabber          1  20    0 47036K  4524K select  5   0:04   0.00% s2s
 1462 vbox            1  20    0 84416K  4272K VBoxIS  6   0:01   0.00% VBoxNetDHCP
 1314 jabber          1  20    0 47256K  4168K select  7   0:01   0.00% c2s
 1020 pgsql           1  20    0  4284M  4080K select  0   0:05   0.00% postgres
 1021 pgsql           1  20    0 70432K  3960K select  6   0:08   0.00% postgres
 1015 pgsql           1  20    0  4284M  3704K select  0   0:08   0.00% postgres
 1018 pgsql           1  20    0  4284M  3584K select  3   0:01   0.00% postgres
  803 vbox            1  20    0 83324K  3584K select  3   0:11   0.00% VBoxXPCOMIPCD
 1019 pgsql           1  20    0  4284M  3568K select  6   0:01   0.00% postgres

top -n -o size
Code: 
  PID USERNAME      THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
 1353 pgsql           1  20    0  4288M     0K sbwait  3   0:00   0.00% <postgres>
 1352 pgsql           1  20    0  4288M     0K sbwait  2   0:00   0.00% <postgres>
48746 pgsql           1  20    0  4288M     0K sbwait  7   0:00   0.00% <postgres>
 1017 pgsql           1  20    0  4284M 32488K select  6   0:06   0.00% postgres
 1020 pgsql           1  20    0  4284M  4080K select  3   0:05   0.00% postgres
 1015 pgsql           1  20    0  4284M  3704K select  4   0:08   0.00% postgres
 1018 pgsql           1  20    0  4284M  3584K select  0   0:01   0.00% postgres
 1019 pgsql           1  20    0  4284M  3568K select  4   0:01   0.00% postgres
 1459 vbox           24  21    0  1406M  1118M select  1 547:09   4.05% VBoxHeadless
 1294 mysql          25  20    0   814M   168M uwait   4   9:58   0.00% mysqld
  809 transmission    3  20    0   258M 38372K select  3  36:53   0.10% transmission-daemon
47163 www             1  23    0   237M 11760K accept  7   0:16   0.00% php-fpm
47137 www             1  20    0   233M 10188K accept  7   0:08   0.00% php-fpm
47277 www             1  52    0   229M     0K accept  4   0:10   0.00% <php-fpm>
93068 root            1  20    0   204M 11416K kqread  5   0:01   0.00% php-fpm
  788 vbox           11  20    0   157M  4676K uwait   7   0:03   0.00% vboxwebsrv
  805 vbox           11  20    0   140M  5100K uwait   5   0:39   0.00% VBoxSVC
 1313 jabber          1  20    0 96728K  3364K select  0   0:02   0.00% sm
phoenix said:
I have a feeling it will be Postgres (4 GB per process showing in your original top output, although that could be shared).
Click to expand...
Yes, it's shared memory and not allocated per process.
 
Could that be from transmission? Things like network buffers and that?

For a brute force debugging I would suggest to restart all services, one after the other, and check if the memory comes back. But that may be something the users may not want to happen, but then - who wants a system that is turning into a snail day by day?
 
But how can I see if the memory is back? If I shut a daemon down, I of course get more free memory and can run swapoff -a successfully. But that doesn't mean it's using the inactive one.

Or should "inactive" one become "free" again?
 
You must log in or register to reply here.
Back
Top