Possible kernel memory leak FreeBSD 9.0 release

Remme · Aug 17, 2012

Hi,

Please help to solve a kernel memory leak problem.
After few weeks searching forums and googling any help is welcome.

Here is the problem description:
We are installed the latest 9.0 FreeBSD with all recent patches.
Ports tree is also up to date.
Host running a nginx, php-fpm, memcached, mysqld, sphinxsearch.
Two WD HDD is gmirrored (each partition).
Also we are using a small ssd for nginx cache and mysql databases.
8 GB RAM.

We have always growing active memory till kernel panic.
(it's takes about 2 weeks till memory leak kills host)

According to top - active memory is always growing but sum of all processes memory is stable about 3.5 Gb.
So looks like kernel slowly eating all our memory

uname -r
9.0-RELEASE-p3

Code:

vmstat -z
ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               208,      0,      84,       1,      84,   0,   0
UMA Zones:              896,      0,      84,       0,      84,   0,   0
UMA Slabs:              568,      0,    6212,     186, 1121654,   0,   0
UMA RCntSlabs:          568,      0,    2210,     240,  299509,   0,   0
UMA Hash:               256,      0,       0,       0,       3,   0,   0
16 Bucket:              152,      0,      18,     107,     151,   0,   0
32 Bucket:              280,      0,       6,      36,     281,  29,   0
64 Bucket:              536,      0,      24,       4,     600,  71,   0
128 Bucket:            1048,      0,     390,     258,  421119,1231184,   0
VM OBJECT:              216,      0,  185074,    6122,802935160,   0,   0
MAP:                    232,      0,       7,      25,       7,   0,   0
KMAP ENTRY:             120, 278008,      90,     685, 4371033,   0,   0
MAP ENTRY:              120,      0,    9149,    4057,2020509093,   0,   0
fakepg:                 120,      0,       0,     744,   41083,   0,   0
mt_zone:               4112,      0,     302,       9,     302,   0,   0
16:                      16,      0,    2023,     833,304257560,   0,   0
32:                      32,      0,    3627,    1423,190401641,   0,   0
64:                      64,      0,    3183,    1689,266850042,   0,   0
128:                    128,      0,    9053,    9768,200576267,   0,   0
256:                    256,      0,    1608,    4227,68949080,   0,   0
512:                    512,      0,    7555,    3001,56527162,   0,   0
1024:                  1024,      0,      89,     807,66007444,   0,   0
2048:                  2048,      0,     188,     766, 5107006,   0,   0
4096:                  4096,      0,     505,     472,29475994,   0,   0
Files:                   80,      0,    1693,    1502,366697406,   0,   0
TURNSTILE:              136,      0,    1813,     847,    8107,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1160,      0,     126,     831,28414026,   0,   0
THREAD:                1112,      0,    1061,     751, 2959481,   0,   0
SLEEPQUEUE:              80,      0,    1813,     884,    8107,   0,   0
VMSPACE:                392,      0,     102,     878,28413995,   0,   0
cpuset:                  72,      0,       2,      98,       2,   0,   0
audit_record:           960,      0,       0,       0,       0,   0,   0
mbuf_packet:            256,      0,    2103,     728,539089502,   0,   0
mbuf:                   256,      0,     182,    1247,1761199755,   0,   0
mbuf_cluster:          2048,  25600,    2831,     511, 3053398,   0,   0
mbuf_jumbo_page:       4096,  12800,      14,     525,69180199,   0,   0
mbuf_jumbo_9k:         9216,   6400,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384,   3200,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,     164,    1180,123080226,   0,   0
g_bio:                  232,      0,       0,    2240,130363876,   0,   0
ttyinq:                 160,      0,     810,     246,    1980,   0,   0
ttyoutq:                256,      0,     421,     134,    1032,   0,   0
ata_request:            328,      0,       0,     612,50944836,   0,   0
ata_composite:          336,      0,       0,       0,       0,   0,   0
VNODE:                  480,      0,  181313,    4863,123109166,   0,   0
VNODEPOLL:              112,      0,       1,     131,       5,   0,   0
NAMEI:                 1024,      0,       0,     708,490540140,   0,   0
S VFS Cache:            108,      0,  175290,   25317,122314420,   0,   0
L VFS Cache:            328,      0,      69,    1707,  349874,   0,   0
DIRHASH:               1024,      0,   13001,     447,98434805,   0,   0
NCLNODE:                560,      0,       0,       0,       0,   0,   0
Mountpoints:            768,      0,       7,      18,       8,   0,   0
pipe:                   728,      0,      14,     731,31033156,   0,   0
ksiginfo:               112,      0,     949,     833, 1031495,   0,   0
itimer:                 344,      0,       1,      43,       3,   0,   0
KNOTE:                  128,      0,    2406,    1422,569901495,   0,   0
socket:                 680,  25602,    1858,    1700,133143402,   0,   0
unpcb:                  240,  25600,      45,     723,32000136,   0,   0
ipq:                     56,    819,       0,     189,      36,   0,   0
udp_inpcb:              392,  25600,      34,     866,20225275,   0,   0
udpcb:                   16,  25704,      34,     974,20225275,   0,   0
tcp_inpcb:              392,  25600,    8217,    9213,80917970,   0,   0
tcpcb:                  976,  25600,    1752,    1712,80917970,   0,   0
tcptw:                   72,  41000,    6465,    8785,42309523,   0,   0
syncache:               152,  15375,      34,     816,44953728,   0,   0
hostcache:              136,  15372,    6229,    1191,  448702,   0,   0
tcpreass:                40,   1680,       0,     840,  238618,   0,   0
sackhole:                32,      0,       0,     909, 3424643,   0,   0
sctp_ep:               1368,  25600,       0,       0,       0,   0,   0
sctp_asoc:             2280,  40000,       0,       0,       0,   0,   0
sctp_laddr:              48,  80064,       0,     216,      14,   0,   0
sctp_raddr:             704,  80000,       0,       0,       0,   0,   0
sctp_chunk:             136, 400008,       0,       0,       0,   0,   0
sctp_readq:             104, 400032,       0,       0,       0,   0,   0
sctp_stream_msg_out:    112, 400026,       0,       0,       0,   0,   0
sctp_asconf:             40, 400008,       0,       0,       0,   0,   0
sctp_asconf_ack:         48, 400032,       0,       0,       0,   0,   0
ripcb:                  392,  25600,       0,      50,      10,   0,   0
rtentry:                200,      0,      37,      39,      37,   0,   0
selfd:                   56,      0,    2270,     880,551066611,   0,   0
SWAPMETA:               288, 116519,     133,     907,110049288,   0,   0
FFS inode:              168,      0,  181259,   21427,123107512,   0,   0
FFS1 dinode:            128,      0,       0,       0,       0,   0,   0
FFS2 dinode:            256,      0,  181259,   15046,123106981,   0,   0

top:

Code:

last pid: 66247;  load averages:  0.50,  0.54,  0.59                                                                     up 4+02:14:13  12:50:42
98 processes:  1 running, 95 sleeping, 2 zombie
CPU:  2.6% user,  0.2% nice,  3.6% system,  0.0% interrupt, 93.6% idle
Mem: 3520M Active, 2910M Inact, 1080M Wired, 370M Cache, 827M Buf, 36M Free
Swap: 2048M Total, 2800K Used, 2045M Free

sysctl.conf

Code:

kern.maxfiles=25000
kern.maxfilesperproc=24000
net.inet.tcp.maxtcptw=40960
kern.ipc.somaxconn=16384
# do reboot after panic
debug.debugger_on_panic=0

mount

Code:

/dev/mirror/p2 on / (ufs, local, soft-updates)
devfs on /dev (devfs, local, multilabel)
/dev/mirror/p4 on /tmp (ufs, local, noatime, soft-updates)
/dev/mirror/p5 on /var (ufs, local, noatime, soft-updates)
/dev/mirror/p6 on /usr (ufs, local, noatime, soft-updates)
/dev/ada1 on /ssd (ufs, local, noatime, soft-updates)
devfs on /var/named/dev (devfs, local, multilabel)

Code:

gmirror status
     Name    Status  Components
mirror/p1  COMPLETE  ada0p1 (ACTIVE)
                     ada2p1 (ACTIVE)
mirror/p2  COMPLETE  ada0p2 (ACTIVE)
                     ada2p2 (ACTIVE)
mirror/p3  COMPLETE  ada0p3 (ACTIVE)
                     ada2p3 (ACTIVE)
mirror/p4  COMPLETE  ada0p4 (ACTIVE)
                     ada2p4 (ACTIVE)
mirror/p5  COMPLETE  ada0p5 (ACTIVE)
                     ada2p5 (ACTIVE)
mirror/p6  COMPLETE  ada0p6 (ACTIVE)
                     ada2p6 (ACTIVE)

Where we should digg to catch a memory leak ?
It's not a userland process. TOP RES sum memory usage is stable.

rob34 · Aug 18, 2012

The output from top shows 2 zombie processes. I wonder if that is contributing to your memory issue.

Remme · Aug 18, 2012

rob34 said:
The output from top shows 2 zombie processes. I wonder if that is contributing to your memory issue.

No it's not an issue.
Here is the some additional info:

top:

Code:

last pid:  4154;  load averages:  0.10,  0.30,  0.49                                                                      up 4+16:50:00  03:26:29
92 processes:  1 running, 91 sleeping
CPU:  5.4% user,  0.0% nice, 11.5% system,  0.8% interrupt, 82.3% idle
Mem: 3530M Active, 2904M Inact, 1106M Wired, 323M Cache, 827M Buf, 53M Free
Swap: 2048M Total, 3876K Used, 2044M Free

ps aux | awk '{memory +=$4}; END {print memory }'
36.6
ps aux | awk '{memory +=$6}; END {print memory }'
3173588
sysctl vm.stats.vm.v_active_count
vm.stats.vm.v_active_count: 893551
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this value directly showed in top multiply by memory page size. (4k)

Code:

SYSTEM MEMORY INFORMATION:
mem_wire:        1162616832 (   1108MB) [ 14%] Wired: disabled for paging out
mem_active:  +   3660181504 (   3490MB) [ 44%] Active: recently referenced
mem_inactive:+   2558533632 (   2440MB) [ 30%] Inactive: recently not referenced
mem_cache:   +    238456832 (    227MB) [  2%] Cached: almost avail. for allocation
mem_free:    +    680710144 (    649MB) [  8%] Free: fully available for allocation
mem_gap_vm:  +       536576 (      0MB) [  0%] Memory gap: UNKNOWN
-------------- ------------ ----------- ------
mem_all:     =   8301035520 (   7916MB) [100%] Total real memory managed
mem_gap_sys: +    268271616 (    255MB)        Memory gap: Kernel?!
-------------- ------------ -----------
mem_phys:    =   8569307136 (   8172MB)        Total real memory available
mem_gap_hw:  +     20627456 (     19MB)        Memory gap: Segment Mappings?!
-------------- ------------ -----------
mem_hw:      =   8589934592 (   8192MB)        Total real memory installed

SYSTEM MEMORY SUMMARY:
mem_used:        5112233984 (   4875MB) [ 59%] Logically used memory
mem_avail:   +   3477700608 (   3316MB) [ 40%] Logically available memory
-------------- ------------ ----------- ------
mem_total:   =   8589934592 (   8192MB) [100%] Logically total memory

t1066 · Aug 19, 2012

If you are willing to try DTrace, you could run the following d script.

Code:

#!/usr/sbin/dtrace -s

#pragma D option quiet

dtmalloc:::malloc
{
        @malloc[probefunc] = sum(args[3]);
        @total[probefunc] = sum(args[3]);
}


dtmalloc:::free
{
        @free[probefunc] = sum(args[3]);
        @total[probefunc] = sum(-args[3]);
}

tick-1h
{
        printf("%-20s %20s %20s %20s\n", "FUNCTION", "ALLOCATED", "FREE", "TOTAL");
        printa("%-20.20s %20@d %20@d %20@d\n", @malloc, @free, @total);
        printf("-------------------------------------------------------------\n");
}

tick-12h
{
        exit(0);
}

After 12 hours, the final output is

Code:

FUNCTION                        ALLOCATED                 FREE                TOTAL
audit_data                            128                  128                    0
bus                                   960                  960                    0
osd                                  1072                 1104                  -32
DEVFS3                               1280                 1280                    0
lltable                              1792                 1792                    0
raid_data                            2400                 2400                    0
xform                                2688                 2688                    0
DEVFS1                               4096                 4096                    0
feeder                               7680                 7680                    0
loginclass                          15488                15488                    0
uidinfo                             27776                27776                    0
session                             31616                32000                 -384
pgrp                                33408                33792                 -384
cl_savebuf                          50752                50752                    0
scsi_cd                             73136                73136                    0
kobj                               122880               122880                    0
umtx                               176640               143616                33024
shm                                196608               212992               -16384
devbuf                             216704               216704                    0
select                             334976               333696                 1280
sbuf                               571280               571280                    0
sglist                             799936               799936                    0
plimit                             809728               809728                    0
GEOM                              2184208              2184208                    0
USBdev                            2248192              2248192                    0
DEVFSP                            2252928              2252928                    0
proc-args                         2295216              2295440                 -224
fadvise                           2644832              2644832                    0
sysctl                            4712048              4712048                    0
filedesc                          5258432              5263552                -5120
sysctltmp                         7209888              7209888                    0
taskqueue                         7524864              7524864                    0
lockf                             8445504              8445824                 -320
pcb                              10647136             10647584                 -448
routetbl                         11061248             11061248                    0
acl                              11321344             11321344                    0
Unitno                           11661056             11661664                 -608
inpcbpolicy                      12600480             12603328                -2848
vnodemarker                      29259776             29259776                    0
subproc                          41707008             41723392               -16384
CAM XPT                          43845632             43845632                    0
kqueue                           43990784             43990784                    0
iprtheap                         46111424             46111424                    0
kdtrace                          87377792             87343232                34560
ipsecpolicy                     231722496            231768064               -45568
nvidia                          263010656            263010656                    0
cred                            286044160            286044800                 -640
soname                          433570784            433571296                 -512
eli data                        800746240            800746240                    0
mbuf_tag                       4639548224           4639548224                    0
iov                            7089069344           7089069344                    0
temp                          11221337248          11221337248                    0
ioctlops                      22986257488          22986257504                  -16
solaris                      157538510768         157678934048           -140423280
-------------------------------------------------------------

Maybe this would help to pinpoint memory leak if there is any.

Remme · Aug 19, 2012

Thank you t1066 for good point.
Just recompiled a new kernel with dtrace support.
Script running and I am waiting for results.
One more additional info: we have a lot of small files on the disk
located in subdirectory tree something like /dir/aa/bb/aabbfile.txt
So I think it's somehow related to total number of inodes.
Addiltionally a nginx storing own cache on /ssd - so there a lot of small files for cache too. The number of files on the disk is not growing.

Remme · Aug 19, 2012

Here is the stats for a last 2 hours:

Code:

-------------------------------------------------------------
FUNCTION                        ALLOCATED                 FREE                TOTAL
sigio                                   0                   64                  -64
DEVFS2                                256                    0                  256
pts                                   512                    0                  512
DEVFS1                               1024                    0                 1024
stack                                2048                 2048                    0
newdirblk                            5504                 5504                    0
solaris                              6272                 6272                    0
DEVFS3                               6912                    0                 6912
UMAHash                              9728                 4608                 5120
ip6opt                              25856                25856                    0
tty                                 26624                24576                 2048
select                              40960                    0                40960
loginclass                          80832                80832                    0
uidinfo                             91776                91648                  128
session                            164352               163840                  512
pgrp                               174592               173824                  768
sysctltmp                          176224               176224                    0
umtx                               187392                11520               175872
CAM XPT                            294912               294912                    0
lltable                            381184               368640                12544
acpica                             490496               490496                    0
GEOM                               500736               500736                    0
cl_savebuf                         798080               798080                    0
bus                               1343680              1343680                    0
routetbl                          1576960              1576960                    0
sysctl                            2647968              2647968                    0
plimit                            4201216              4200192                 1024
vnodemarker                       5223424              5223424                    0
freefrag                         10432128             10432256                 -128
bmsafemap                        11517952             11509760                 8192
freefile                         14678144             14678144                    0
proc-args                        18847504             18847072                  432
pagedep                          22355200             22385152               -29952
freeblks                         30709760             30708736                 1024
freework                         36584320             36583296                 1024
savedino                         40652544             40652544                    0
indirdep                         47369984             47369856                  128
ioctlops                         52882592             52882592                    0
diradd                           54966528             54989312               -22784
dirrem                           55073536             55065088                 8448
sbuf                            122068224            122068224                    0
pcb                             143172928            143161408                11520
kdtrace                         224737536            224651776                85760
inodedep                        243672576            243727360               -54784
newblk                          281683456            281663488                19968
Unitno                          360457408            360457280                  128
soname                          718592400            718592416                  -16
ufs_dirhash                     764659632            761882736              2776896
kqueue                          807155456            807129088                26368
iov                            1116127264           1116127264                    0
cred                           1412654720           1412649280                 5440
lockf                          1588966592           1588966272                  320
temp                           2684924720           2684924720                    0
filedesc                       3787133920           3787020272               113648
subproc                       10365613568          10365386752               226816
-------------------------------------------------------------
FUNCTION                        ALLOCATED                 FREE                TOTAL
sigio                                   0                   64                  -64
DEVFS2                                256                    0                  256
pts                                   512                    0                  512
DEVFS1                               1024                    0                 1024
stack                                2048                 2048                    0
newdirblk                            6144                 6144                    0
solaris                              6272                 6272                    0
DEVFS3                               6912                    0                 6912
UMAHash                              9728                 4608                 5120
ip6opt                              26112                26112                    0
tty                                 26624                24576                 2048
select                              40960                    0                40960
loginclass                          91008                91008                    0
uidinfo                            106240               106112                  128
session                            185088               184576                  512
umtx                               187392                14592               172800
sysctltmp                          192864               192864                    0
pgrp                               195584               194816                  768
CAM XPT                            331776               331776                    0
lltable                            446464               428544                17920
acpica                             490496               490496                    0
GEOM                               562176               562176                    0
cl_savebuf                         813696               813696                    0
bus                               1343680              1343680                    0
routetbl                          1773056              1773056                    0
sysctl                            2967792              2967792                    0
plimit                            4730368              4729344                 1024
vnodemarker                       5871104              5871104                    0
freefrag                         10595072             10595328                 -256
bmsafemap                        12927744             12922880                 4864
freefile                         16374080             16358016                16064
proc-args                        20421264             20420832                  432
pagedep                          25062912             25075200               -12288
freeblks                         34144256             34111872                32384
freework                         40081024             40048640                32384
savedino                         45648128             45648128                    0
indirdep                         52606720             52606720                    0
ioctlops                         58518288             58518288                    0
diradd                           61503616             61507840                -4224
dirrem                           61583744             61573120                10624
sbuf                            137103744            137103744                    0
pcb                             161549280            161543872                 5408
kdtrace                         251861888            251776000                85888
inodedep                        272563712            272381952               181760
newblk                          310849792            310784256                65536
Unitno                          403731456            403731392                   64
soname                          803913920            803913664                  256
ufs_dirhash                     856197200            853278608              2918592
kqueue                          898596608            898570240                26368
iov                            1255426432           1255426432                    0
cred                           1542662080           1542656640                 5440
lockf                          1780357952           1780357632                  320
temp                           2999472736           2999472736                    0
filedesc                       4249912800           4249797616               115184
subproc                       11619829248          11619598336               230912
-------------------------------------------------------------

Remme · Aug 19, 2012

ufs_dirhash is growing every hour.
Probably it's normal due high number of files at the HDD.

sysctl sysctl vfs.ufs.
vfs.ufs.dirhash_reclaimage: 5
vfs.ufs.dirhash_lowmemcount: 25
vfs.ufs.dirhash_docheck: 0
vfs.ufs.dirhash_mem: 13438625
vfs.ufs.dirhash_maxmem: 13549568
vfs.ufs.dirhash_minsize: 2560
vfs.ufs.rename_restarts: 6

t1066 · Aug 20, 2012

Since vfs.ufs.dirhash_lowmemcount is already at 25, you should consider increase vfs.ufs.dirhash_maxmem.

You may try to dig deeper into each function, by using the following modified d script.

Code:

#!/usr/sbin/dtrace -s

#pragma D option quiet

BEGIN
{
        printf("%-20s %20s %20s %20s\n", "FUNCTION", "ALLOCATED", "FREE", "TOTAL");
}

dtmalloc::$1:malloc
{
        @malloc[probefunc] = sum(args[3]);
        @total[probefunc] = sum(args[3]);
}

dtmalloc::$1:free
{
        @free[probefunc] = sum(args[3]);
        @total[probefunc] = sum(-args[3]);
}

tick-1m
{
        printa("%-20.20s %20@d %20@d %20@d\n", @malloc, @free, @total);
}

tick-60m
{
        exit(0);
}

If you save the above script as malloc.d and you are interested in ufs_dirhash, you can run

# ./malloc.d ufs_dirhash

This will give a much detail picture of how memory is used by ufs_dirhash.

Remme · Aug 20, 2012

On last reboot I am removed noatime options in /etc/fstab.
Regarding the ufs_dirhash:
Pushed it to 128Mb limit. But sometimes vfs.ufs.dirhash_lowmemcount is
still growing a little bit.

sysctl sysctl vfs.ufs.
vfs.ufs.dirhash_reclaimage: 5
vfs.ufs.dirhash_lowmemcount: 112
vfs.ufs.dirhash_docheck: 0
vfs.ufs.dirhash_mem: 83265493
vfs.ufs.dirhash_maxmem: 134217728
vfs.ufs.dirhash_minsize: 2560
vfs.ufs.rename_restarts: 12

Here is the malloc.d script output:

Code:

./malloc.d ufs_dirhash
FUNCTION                        ALLOCATED                 FREE                TOTAL
ufs_dirhash                         86880                    0                86880
ufs_dirhash                        134848                    0               134848
ufs_dirhash                        202976                    0               202976
ufs_dirhash                        295808                    0               295808
ufs_dirhash                        394400                    0               394400
ufs_dirhash                        463840                    0               463840
ufs_dirhash                        530848                    0               530848
ufs_dirhash                        614880                    0               614880
ufs_dirhash                        677280                    0               677280
ufs_dirhash                        743936                53280               690656
ufs_dirhash                        807296                53280               754016
ufs_dirhash                        851424                53280               798144
ufs_dirhash                        925408                53280               872128

Also I am looked to subproc (it's probably related to userspace process memory)

Code:

./malloc.d subproc
FUNCTION                        ALLOCATED                 FREE                TOTAL
subproc                          20316160             20316160                    0
subproc                          42786816             42795008                -8192
subproc                          65003520             65028096               -24576
subproc                          82042880             82051072                -8192
subproc                         104824832            104828928                -4096
subproc                         123293696            123301888                -8192
subproc                         142020608            142036992               -16384
subproc                         159727616            159748096               -20480
subproc                         176181248            176152576                28672
subproc                         192933888            192897024                36864
subproc                         211079168            211042304                36864
subproc                         232284160            232255488                28672
subproc                         247283712            247250944                32768
subproc                         268849152            268816384                32768
subproc                         285163520            285138944                24576
subproc                         306548736            306515968                32768
subproc                         328122368            328052736                69632
subproc                         348663808            348622848                40960
subproc                         366395392            366403584                -8192
subproc                         390660096            390676480               -16384
subproc                         414109696            414126080               -16384
subproc                         433430528            433451008               -20480
subproc                         449609728            449626112               -16384
subproc                         466006016            466018304               -12288
subproc                         480702464            480718848               -16384
subproc                         500494336            500498432                -4096
subproc                         515293184            515321856               -28672
subproc                         531828736            531841024               -12288
subproc                         548081664            548093952               -12288
subproc                         564645888            564658176               -12288
subproc                         581836800            581849088               -12288
subproc                         602492928            602501120                -8192
subproc                         621928448            621944832               -16384
subproc                         640720896            640712704                 8192
subproc                         672854016            672833536                20480
subproc                         697917440            697876480                40960
subproc                         725282816            725258240                24576
subproc                         750436352            750424064                12288
subproc                         772464640            772464640                    0
subproc                         795967488            795971584                -4096
subproc                         819851264            819851264                    0
subproc                         839016448            839016448                    0
subproc                         867217408            867217408                    0
subproc                         892387328            892391424                -4096
subproc                         915288064            915292160                -4096
subproc                         934658048            934653952                 4096
subproc                         954683392            954687488                -4096
subproc                         974516224            974516224                    0
subproc                         999133184            999112704                20480
subproc                        1023672320           1023684608               -12288
subproc                        1046654976           1046683648               -28672
subproc                        1069256704           1069273088               -16384
subproc                        1087078400           1087090688               -12288
subproc                        1104482304           1104502784               -20480
subproc                        1126522880           1126543360               -20480
subproc                        1144356864           1144377344               -20480
subproc                        1165557760           1165578240               -20480
subproc                        1188519936           1188540416               -20480
subproc                        1211129856           1211154432               -24576
subproc                        1232977920           1233002496               -24576

Now I am just looking how the things going.
Really hard to catch small leaks and reasons for them

It must be somehow related to number of inodes/files/directories on the disk.

Remme · Aug 20, 2012

Got some free calls in dirhash.

Code:

./malloc.d ufs_dirhash
FUNCTION                        ALLOCATED                 FREE                TOTAL
ufs_dirhash                         37952                    0                37952
ufs_dirhash                         83904                 6144                77760
ufs_dirhash                        123024                 9232               113792
ufs_dirhash                        156560                 9232               147328
ufs_dirhash                        201776                 9232               192544
ufs_dirhash                        251024                 9232               241792
ufs_dirhash                        290416                24080               266336
ufs_dirhash                        311344                24080               287264
ufs_dirhash                        350992                24080               326912
ufs_dirhash                        396464                24080               372384
ufs_dirhash                        429008                24080               404928
ufs_dirhash                        451760                24080               427680
ufs_dirhash                        496224               220688               275536
ufs_dirhash                        535968               220688               315280
ufs_dirhash                        569632               220688               348944
ufs_dirhash                        594944               220688               374256
ufs_dirhash                        616448               220688               395760
ufs_dirhash                        637312               220688               416624
ufs_dirhash                        673984               220688               453296
ufs_dirhash                        692096               739344               -47248
ufs_dirhash                        739808               739344                  464
ufs_dirhash                        772768               739344                33424
ufs_dirhash                        805504               739344                66160
ufs_dirhash                        826208               739344                86864
ufs_dirhash                        845600               739344               106256
ufs_dirhash                        873184               822288                50896
ufs_dirhash                        911040               822288                88752
ufs_dirhash                        935840               822288               113552
ufs_dirhash                        990976               822288               168688
ufs_dirhash                       1065664               822288               243376
ufs_dirhash                       1119152               822288               296864
ufs_dirhash                       1178448               825872               352576
ufs_dirhash                       1214496               828896               385600
ufs_dirhash                       2205792              3318416             -1112624
ufs_dirhash                       2295776              3318416             -1022640
ufs_dirhash                       2368512              3318416              -949904
ufs_dirhash                       2434368              3318416              -884048
ufs_dirhash                       2487392              3322000              -834608
ufs_dirhash                       2535200              3322000              -786800
ufs_dirhash                       2574912              3322000              -747088
ufs_dirhash                       2636352              3322000              -685648

Remme · Aug 20, 2012

Still growing active memory image.

vmstat -m

Code:

vmstat -m
         Type InUse MemUse HighUse Requests  Size(s)
      CAM SIM     4     1K       -        4  256
      scsi_cd     0     0K       -        5  16
         cdev     7     2K       -        7  256
   CAM periph    10     3K       -       26  16,32,64,128,256
      CAM XPT    55    25K       -      634  32,64,128,1024,2048
        sigio    10     1K       -       11  64
     filedesc   203   313K       - 16416622  16,32,64,128,512,1024,2048,4096
      kdtrace  1123   255K       -  9142370  64,256
         kenv   111    13K       -      131  16,32,64,128
       kqueue    32    82K       -  2257777  256,512,2048,4096
    proc-args   113     7K       -   687929  16,32,64,128,256
        hhook     2     1K       -        2  128
      ithread    81    14K       -       81  32,128,256
       KTRACE   100    13K       -      100  128
       linker   414   591K       -      563  16,32,64,128,256,512,1024,2048,4096
        lockf    46     6K       - 44195973  64,128
   loginclass     2     1K       -     4221  64
       ip6ndp    13     2K       -       16  64,128
       ip6opt     0     0K       -      173  256
         temp    41    26K       -  8142725  16,32,64,128,256,512,1024,2048,4096
       devbuf  2751 10140K       -     2862  16,32,64,128,256,512,1024,2048,4096
       module   432    54K       -      432  128
     mtx_pool     2    16K       -        2  
         UART     6     4K       -        6  16,512,1024
      subproc  1019   993K       -  8355394  512,4096
         proc     2    16K       -        2  
      session    45     6K       -     4354  128
         pgrp    53     7K       -     5016  128
         cred   166    26K       - 26936440  64,256
      uidinfo    11     4K       -     2595  128,2048
       plimit    34     9K       -    55013  256
      ata_pci     2     1K       -        2  64
    sysctltmp     0     0K       -    16651  16,32,64,128,4096
    sysctloid  3718   185K       -     3900  16,32,64,128
       sysctl     0     0K       -   541989  16,32,64
      tidhash     1    16K       -        1  
      callout     3  1536K       -        3  
         umtx  3510   439K       -     4278  128
     p1003.1b     1     1K       -        1  16
         SWAP     2   277K       -        2  64
       bus-sc    64   264K       -     2583  16,32,64,128,256,512,1024,2048,4096
          bus  1061    89K       -     7875  16,32,64,128,256,512,1024
      devstat    14    29K       -       14  32,4096
 eventhandler    84     7K       -       84  64,128
         kobj   292  1168K       -      394  4096
      Per-cpu     1     1K       -        1  32
       USBdev    29     8K       -       29  64,128,512,1024
          USB    45    35K       -       47  16,32,64,128,256,2048,4096
         rman   228    27K       -      482  16,32,128
         sbuf     1     1K       -   562740  16,32,64,128,256,512,1024,2048,4096
        stack     0     0K       -       20  256
    taskqueue    19     2K       -       19  16,32,128
       Unitno    19     2K       - 37033238  32,64
          iov     3     1K       - 73970735  16,32,64,128,256,512
       select   844   106K       -      844  128
     ioctlops     0     0K       -  5589524  16,32,64,128,256,512,1024,2048
          msg     4    30K       -        4  2048,4096
          sem     4   106K       -        4  2048,4096
          shm     1    20K       -        1  
          tty    28    28K       -       48  1024,2048,4096
          pts     9     3K       -        9  256
     mbuf_tag     0     0K       -       23  32
        shmfd     1     8K       -        1  
          pcb  1735   211K       - 14821089  16,32,128,1024,2048,4096
       soname    64     8K       - 54282733  16,32,128
          acl     0     0K       -     1976  4096
     vfscache     1  2048K       -        1  
   cl_savebuf     0     0K       -    36134  64
     vfs_hash     1  1024K       -        1  
      acpidev    35     3K       -       35  64
       vnodes     2     1K       -        2  256
  vnodemarker     1     1K       -    33592  512
        mount    89     5K       -      159  16,32,64,128,256
          BPF     9     2K       -        9  128
  ether_multi    66     4K       -       76  16,32,64
       ifaddr    90    26K       -       90  32,64,128,256,512,4096
        ifnet    10    19K       -       10  128,2048
        clone     6    24K       -        8  16,128,4096
       arpcom     2     1K       -        2  16
      lltable    85    26K       -     5119  256,512
          tun     1     1K       -        1  256
     pci_link    16     2K       -       16  32,128
     routetbl    62     8K       -    20579  32,64,128,256,512
         igmp     9     3K       -        9  256
     acpiintr     1     1K       -        1  64
     in_multi     4     1K       -        4  256
    sctp_iter     0     0K       -       14  256
     sctp_ifn     4     1K       -        4  128
     sctp_ifa    15     2K       -       15  128
     sctp_vrf     1     1K       -        1  64
    sctp_a_it     0     0K       -       14  16
    hostcache     1    28K       -        1  
     syncache     1    96K       -        1  
       DEVFS1   149    75K       -      157  512
       DEVFS3   348    87K       -      367  256
       DEVFS2   149     3K       -      149  16
    in6_multi    36     5K       -       36  32,256
   DEVFS_RULE    54    26K       -       54  64,512
          mld     9     2K       -        9  128
          rpc     2     1K       -        2  256
audit_evclass   179     6K       -      218  32
     savedino     0     0K       -   463853  256
     freework  1584   198K       -   862659  16,128
    newdirblk     0     0K       -      279  64
       dirrem  1140   143K       -  1337094  128
        mkdir     0     0K       -       12  128
       diradd   364    46K       -  1342838  128
     freefile   348    22K       -   705709  64
     freeblks  1326   166K       -   739285  128
     freefrag     1     1K       -   228674  128
     indirdep    13     2K       -    11954  128
       newblk 21934  5740K       -  3494509  256
    bmsafemap    46    20K       -   151173  256
     inodedep  1959  2003K       -  1488295  512
      pagedep   211   309K       -   286908  256
  ufs_dirhash  8257  4629K       -  4005668  16,32,64,128,256,512,1024,2048,4096
    ufs_mount    15    71K       -       15  512,4096
    vm_pgdata     2   129K       -        2  128
      UMAHash     3   266K       -       18  512,1024,2048,4096
        DEVFS    62     2K       -       63  16,128
       DEVFSP     3     1K       -       26  64
      memdesc     1     4K       -        1  4096
     atkbddev     2     1K       -        2  64
    pfs_nodes    21     6K       -       21  256
       acpica  1482   144K       -    56726  16,32,64,128,256,512,1024
       apmdev     1     1K       -        1  128
         GEOM   180    33K       -     7389  16,32,64,128,256,512,1024,2048
     acpitask     1     2K       -        1  2048
CAM dev queue     4     1K       -        4  128
    CAM queue    20     1K       -       68  16,32
      acpisem    20     3K       -       20  128
       kbdmux     7    18K       -        7  16,512,1024,2048
      io_apic     1     2K       -        1  2048
      entropy  1024    64K       -     1024  64
          msi     2     1K       -        2  128
     nexusdev     3     1K       -        3  16
       isadev     4     1K       -        4  128
          LED     4     1K       -        4  16,128
  mirror_data    18     5K       -       54  64,128,512
      solaris 194677 210428K       -   198905  16,32,64,128,256,512,1024,2048,4096
       cyclic    16     2K       -       26  16,64,128,256
          fbt 46322  6047K       -    46322  128

t1066 · Aug 21, 2012

What is the value of your kern.openfiles? Is it increasing ever slightly?

Remme · Aug 21, 2012

sysctl kern.openfiles

kern.openfiles: 1849

it's stable - here is the gpaph:

lme@ · Aug 21, 2012

Maybe you should point the readers of hackers@ to this thread?

Remme · Aug 21, 2012

lme@ said:
Maybe you should point the readers of hackers@ to this thread?

Sorry but how to do this ?
Subscribe to the list ? and post right ?

Remme · Aug 21, 2012

lme@ said:
Maybe you should point the readers of hackers@ to this thread?

Done - maybe someone will take a look here to help.
Thanks.

ivoras@ · Aug 22, 2012

Remme said:
Hi,
We have always growing active memory till kernel panic.
(it's takes about 2 weeks till memory leak kills host)

According to top - active memory is always growing but sum of all processes memory is stable about 3.5 Gb.
So looks like kernel slowly eating all our memory

What exactly is the panic message?

The vfs.ufs.dirhash_lowmemcount discussion is a bit of a red herring because it's not caused when vfs.ufs.dirhash_maxmem is too low, but when the kernel needs to allocate memory and finds there is not enough of it, so it asks dirhash to free some (i.e. it indicates a "kernel memory low" event count). This event happens relatively often, even on systems which run perfectly with years of uptime, so it doesn't indicate a memory leak by itself.

I'm curious about you running nginx (because I don't run it, so I might be wrong...): does it use some "advanced" optimizations? If it uses mmap() and/or sendfile() calls, can you configure it to not use them, just for testing? Is there anything other it does differently than e.g. apache?

I know you said that the sum of userland processes memory usage is approximately constant, but have you ever tried to simply stop your entire userland web stack (nginx, mysql, others) when the leak becomes noticable and compare the memory usage?

Remme · Aug 22, 2012

ivoras@ said:
What exactly is the panic message?

The vfs.ufs.dirhash_lowmemcount discussion is a bit of a red herring because it's not caused when vfs.ufs.dirhash_maxmem is too low, but when the kernel needs to allocate memory and finds there is not enough of it, so it asks dirhash to free some (i.e. it indicates a "kernel memory low" event count). This event happens relatively often, even on systems which run perfectly with years of uptime, so it doesn't indicate a memory leak by itself.

I'm curious about you running nginx (because I don't run it, so I might be wrong...): does it use some "advanced" optimizations? If it uses mmap() and/or sendfile() calls, can you configure it to not use them, just for testing? Is there anything other it does differently than e.g. apache?

I know you said that the sum of userland processes memory usage is approximately constant, but have you ever tried to simply stop your entire userland web stack (nginx, mysql, others) when the leak becomes noticable and compare the memory usage?

Here is the part of dmesg with kernel panic message:

Code:

Aug  2 15:33:13 zeta kernel: 
Aug  2 15:33:13 zeta kernel: Fatal trap 12: page fault while in kernel mode
Aug  2 15:33:13 zeta kernel: cpuid = 2; apic id = 02
Aug  2 15:33:13 zeta kernel: fault virtual address        = 0x18
Aug  2 15:33:13 zeta kernel: fault code           = supervisor read data, page not present
Aug  2 15:33:13 zeta kernel: instruction pointer  = 0x20:0xffffffff8087861a
Aug  2 15:33:13 zeta kernel: stack pointer                = 0x28:0xffffff82344509a0
Aug  2 15:33:13 zeta kernel: frame pointer                = 0x28:0xffffff82344509d0
Aug  2 15:33:13 zeta kernel: code segment         = base 0x0, limit 0xfffff, type 0x1b
Aug  2 15:33:13 zeta kernel: = DPL 0, pres 1, long 1, def32 0, gran 1
Aug  2 15:33:13 zeta kernel: processor eflags     = interrupt enabled, resume, IOPL = 0
Aug  2 15:33:13 zeta kernel: current process              = 12865 (searchd)
Aug  2 15:33:13 zeta kernel: trap number          = 12
Aug  2 15:58:22 zeta syslogd: kernel boot file is /boot/kernel/kernel

About nginx - we are using it for many years on many hosts.
Yes it's using sendfile/kqueue.
Only two new things in the config: started to use mod_geoip and mod_proxy for storing
cache on the SSD (a lot of small files).

We started to investigate this memory leak
from restarting all userland daemons (exept probably sendmail,sshd,openvpn,smartd and bsnmpd)
We are removed even varnishd (web caching proxy) for testing.
You may see this on the memory usage graph (small drops in active memory usage).
We can't stop all permanently due it's production server.
php-fpm, nginx, memcached, mysqld, searchd(sphinxsearch) - all of them was restarted (not just reloaded configs).

I understand your point ivoras about removing part by part and finally catch the reason.
We are tryed even stop an stats script what is collecting RRD system data - but no success.
Probably we should migrate critical services to another host and this way to pinpoint problem part.
Since FreeBSD 2.x.x - no memory leak problems at all for many years - so it's sound really unusual to get such problem with kernel now.

Remme · Aug 22, 2012

I have some suspicions regarding SSD TRIM support (it's added recently in 8.3 and 9.0).
Is this possible somehow to track memory usage by this part of the kernel ?

ivoras@ · Aug 23, 2012

Remme said:

Here is the part of dmesg with kernel panic message:

Code:

Aug  2 15:33:13 zeta kernel: 
Aug  2 15:33:13 zeta kernel: Fatal trap 12: page fault while in kernel mode

Aug  2 15:33:13 zeta kernel: current process              = 12865 (searchd)
Aug  2 15:58:22 zeta syslogd: kernel boot file is /boot/kernel/kernel

Note that this type of panic, as-is, is not directly related to kernel memory leaks. A leak might expose a bug which could lead to this kind of situation, but it is unlikely.

If you are running a recent FreeBSD and enabled crashdumps, you should have a "text dump" description of the system state when the panic happens together with your kernel dump (usually in /var/crash). I think the best bet right now is for you to look at this file, find a kernel stack trace of the panic, and post it here.

Also, does the crash regularly happen in searchd or is it randomly spread across processes?

php-fpm, nginx, memcached, mysqld, searchd(sphinxsearch) - all of them was restarted (not just reloaded configs).

Yes, that was what I was aiming at.

Regarding the TRIM support: I'm not an expert in that part of the code but I can only see one explicit malloc() call in this code path, and it should be accounted for under the "temp" label if you issue a "vmstat -m". This is also probably a red herring because it looks like this memory should be properly freed. Do a "vmstat -m" and see if there is anything strange.

Remme · Aug 23, 2012

Thank you ivoras.
Unfortunately kernel dumps was disabled till now.
I think we are one step forward to find a workaround.
We got a second kernel panic (same searchd) - but memory was not totally filled in.
On last reboot we are moved all data files what is used by searchd to /var/db/sphinxsearch from /ssd/sphinxsearch. Now looks like active memory is not growing in previous rate. (Total active memory must grow day or two due memcached and others not filled totally with data).

So only one difference for now - databases was moved from SSD to RAID1 (WD).

ada1: <SSDSA2SH032G1GN INTEL 045C8860> ATA-7 SATA 2.x device
WARNING: /ssd: TRIM flag on fs but disk does not support TRIM
(filesystem on SSD was by mistake created with -t flag (TRIM support))
Filesystem on SSD was created with command:
newfs -t -S 4096 -b 32768 -f 4096 -O 2 -U -m 8 -o space -L cache /dev/ada1

We must wait for now and see memory usage, and pray to do not get a kernel panic again

vmstat -m

Code:

         Type InUse MemUse HighUse Requests  Size(s)
      CAM SIM     4     1K       -        4  256
      scsi_cd     0     0K       -        5  16
         cdev     7     2K       -        7  256
   CAM periph    10     3K       -       26  16,32,64,128,256
      CAM XPT    55    25K       -      488  32,64,128,1024,2048
        sigio    10     1K       -       20  64
     filedesc   173   320K       -  9945389  16,32,64,128,512,1024,2048,4096
      kdtrace  1031   238K       -  5602866  64,256
         kenv   111    13K       -      131  16,32,64,128
       kqueue    32    90K       -  1512879  256,512,2048,4096
    proc-args    84     5K       -   447598  16,32,64,128,256
        hhook     2     1K       -        2  128
      ithread    81    14K       -       81  32,128,256
       KTRACE   100    13K       -      100  128
       linker   148    28K       -      158  16,32,64,128,256,512,2048,4096
        lockf    46     6K       - 27757696  64,128
   loginclass     2     1K       -     2927  64
       ip6ndp    13     2K       -       16  64,128
       ip6opt     0     0K       -      140  256
         temp    41    26K       -  5252871  16,32,64,128,256,512,1024,2048,4096
       devbuf  2751 10140K       -     6812  16,32,64,128,256,512,1024,2048,4096
       module   416    52K       -      416  128
     mtx_pool     2    16K       -        2  
         UART     6     4K       -        6  16,512,1024
      subproc   914   835K       -  5063232  512,4096
         proc     2    16K       -        2  
      session    35     5K       -     3061  128
         pgrp    39     5K       -     3619  128
         cred   140    22K       - 16422830  64,256
      uidinfo     9     3K       -     1839  128,2048
       plimit    21     6K       -    38089  256
      ata_pci     2     1K       -        2  64
    sysctltmp     0     0K       -    13562  16,32,64,128,4096
    sysctloid  3718   185K       -     3900  16,32,64,128
       sysctl     0     0K       -   641307  16,32,64
      tidhash     1    16K       -        1  
      callout     3  1536K       -        3  
         umtx  3384   423K       -     4128  128
     p1003.1b     1     1K       -        1  16
         SWAP     2   277K       -        2  64
       bus-sc    64   264K       -     2583  16,32,64,128,256,512,1024,2048,4096
          bus  1061    89K       -     9849  16,32,64,128,256,512,1024
      devstat    14    29K       -       14  32,4096
 eventhandler    84     7K       -       84  64,128
         kobj   292  1168K       -      373  4096
      Per-cpu     1     1K       -        1  32
       USBdev    29     8K       -       29  64,128,512,1024
          USB    45    35K       -       47  16,32,64,128,256,2048,4096
         rman   228    27K       -      482  16,32,128
         sbuf     1     1K       -   390699  16,32,64,128,256,512,1024,2048,4096
        stack     0     0K       -       32  256
    taskqueue    19     2K       -       19  16,32,128
       Unitno    17     1K       - 22668943  32,64
          iov     0     0K       - 42778861  16,32,64,128,256,512
       select   763    96K       -      763  128
     ioctlops     0     0K       -  2810418  16,32,64,128,256,512,1024,2048
          msg     4    30K       -        4  2048,4096
          sem     4   106K       -        4  2048,4096
          shm     1    20K       -        1  
          tty    23    23K       -       67  1024,2048,4096
          pts     4     1K       -       16  256
     mbuf_tag     0     0K       -       23  32
        shmfd     1     8K       -        1  
          pcb  1422   201K       -  9632439  16,32,128,1024,2048,4096
       soname    12     1K       - 35163141  16,32,128
          acl     0     0K       -     1976  4096
     vfscache     1  2048K       -        1  
   cl_savebuf     0     0K       -    14746  64
     vfs_hash     1  1024K       -        1  
      acpidev    35     3K       -       35  64
       vnodes     2     1K       -        2  256
  vnodemarker     0     0K       -    23083  512
        mount    97     5K       -      332  16,32,64,128,256
          BPF     9     2K       -        9  128
  ether_multi    66     4K       -       76  16,32,64
       ifaddr    90    26K       -       90  32,64,128,256,512,4096
        ifnet    10    19K       -       10  128,2048
        clone     6    24K       -        8  16,128,4096
       arpcom     2     1K       -        2  16
      lltable    80    25K       -     3651  256,512
          tun     1     1K       -        1  256
     pci_link    16     2K       -       16  32,128
     routetbl    62     8K       -    14099  32,64,128,256,512
         igmp     9     3K       -        9  256
     acpiintr     1     1K       -        1  64
     in_multi     4     1K       -        4  256
    sctp_iter     0     0K       -       14  256
     sctp_ifn     4     1K       -        4  128
     sctp_ifa    15     2K       -       15  128
     sctp_vrf     1     1K       -        1  64
    sctp_a_it     0     0K       -       14  16
    hostcache     1    28K       -        1  
     syncache     1    96K       -        1  
       DEVFS1   134    67K       -      151  512
       DEVFS3   305    77K       -      345  256
       DEVFS2   133     3K       -      145  16
    in6_multi    36     5K       -       36  32,256
   DEVFS_RULE    54    26K       -       54  64,512
          mld     9     2K       -        9  128
          rpc     2     1K       -        2  256
audit_evclass   179     6K       -      218  32
     savedino     0     0K       -   322223  256
     freework   130    17K       -   542825  16,128
    newdirblk     0     0K       -      203  64
       dirrem    64     8K       -   900960  128
        mkdir     0     0K       -       24  128
       diradd   257    33K       -   899194  128
     freefile   281    18K       -   476400  64
     freeblks   129    17K       -   490770  128
     freefrag     3     1K       -    87798  128
     indirdep     3     1K       -     8429  128
       newblk   310   334K       -  2150498  256
    bmsafemap    26    15K       -    98409  256
     inodedep   688  1368K       -  1004022  512
      pagedep   156   295K       -   181629  256
  ufs_dirhash 19166  6696K       -   437119  16,32,64,128,256,512,1024,2048,4096
    ufs_mount    16    71K       -       32  128,512,1024,4096
    vm_pgdata     2   129K       -        2  128
      UMAHash     3   266K       -       18  512,1024,2048,4096
        DEVFS    62     2K       -       63  16,128
      memdesc     1     4K       -        1  4096
     atkbddev     2     1K       -        2  64
    pfs_nodes    21     6K       -       21  256
       acpica  1482   144K       -    67502  16,32,64,128,256,512,1024
       apmdev     1     1K       -        1  128
         GEOM   180    33K       -     5229  16,32,64,128,256,512,1024,2048
     acpitask     1     2K       -        1  2048
CAM dev queue     4     1K       -        4  128
    CAM queue    20     1K       -       68  16,32
      acpisem    20     3K       -       20  128
       kbdmux     7    18K       -        7  16,512,1024,2048
      io_apic     1     2K       -        1  2048
      entropy  1024    64K       -     1024  64
          msi     2     1K       -        2  128
     nexusdev     3     1K       -        3  16
       isadev     4     1K       -        4  128
          LED     4     1K       -        4  16,128
  mirror_data    18     5K       -       54  64,128,512

vmstat -m | grep temp

Code:

         temp    41    26K       -  5254383  16,32,64,128,256,512,1024,2048,4096

ivoras@ · Aug 24, 2012

From what you said, I think that the problem is not SSD or TRIM, or a memory leak, but that it is very likely that searchd exposes some kind of a kernel bug. You should collect crash dumps.

Remme · Aug 24, 2012

Active memory was still increasing. (but more slowly)
The searchd is not the state of the art - but userland processes suppose to do not have a possibility to crash the system.
At this moment I am moved all from ssd to hdd and unmounted /ssd.
Waiting for now kernel panic or stable memory usage.

atrull · Aug 24, 2012

I've got a similar issue with a big jail box of mine running 8.3.

http://web.internationalconspiracy..../potjie.internationalconspiracy.org/kmem.html

Initialy I thought it was ZFS-related, but I've disabled ZFS and it just keeps growing.

The temporary solution is a reboot, or wait for the system to swap so much that it is unusable THEN reboot!.

This memory is not being held by any actual processes, there is something clearly eating its way through kmem at a steady rate and not releasing it.

Remme · Aug 25, 2012

atrull said:
I've got a similar issue with a big jail box of mine running 8.3.

http://web.internationalconspiracy..../potjie.internationalconspiracy.org/kmem.html

Initialy I thought it was ZFS-related, but I've disabled ZFS and it just keeps growing.

The temporary solution is a reboot, or wait for the system to swap so much that it is unusable THEN reboot!.

This memory is not being held by any actual processes, there is something clearly eating its way through kmem at a steady rate and not releasing it.

I can't find how do you count kmem usage via munin
(http://munin-monitoring.org/browser/branches/1.4-stable/plugins)
is this some kind of custom plugin ?

BTW, after moving from SSD to HDD last 6 hours looks like memory usage is stable.
I will wait till tomorrow and go further with testing SSD config.

Possible kernel memory leak FreeBSD 9.0 release

Administrator