Mbuf leak detection tool

M

my_freebsd

Hi All,

Do you know of a tool to detect mbuf leaks which one could leverage in identifying
mbuf leaks (with possible stack traces of allocs and frees) ?

I understand we might be able to use the vmstat -z for possible statistics of the various
mbuf zones, but it helps only to certain point.
 
I think only you can truly know. You don't specify anything about your system, what network connectivity you have, what network devices, what the perceived issue is etc etc etc...

One generalised way: start up in single user mode, see if it leaks. If it doesn't then disable everything in rc.conf that's network related, manually starting them. Test/observe at each stage.
 
Thank you for the reply, mark_j.

I was primarily looking for a tool for accounting for mbuf allocations and frees across various zones. Today, we have a bunch of m_get API(s) to alloc mbuf and only one m_free() API. It seems little confusing with the mbuf, mbuf_pack, mbuf_cluster etc zones to keep an account of the allocations and frees for the mbuf. May be we already have a tool to account for the mbuf allocations. Query is applicable across any FBSD release.

Appreciate your help and insights in this regard.
 
Well, again, you're quite vague in what you want or what your problem actually is. You're now talking about kernel IPCs?

"A bunch of m_get()... one m_free()"? Well dah! There's malloc/alloc/calloc in userland, but guess what, there's only free().

What are you tinkering in the IPC memory for?

Honestly, you need to either elucidate what you're trying to achieve or just write a program to solve it, because I can't even hazard a guess what you're up to. It's all too cryptic for me. o_O

Good luck.

Oh, just thought, netstat with the -x option is a tool of great importance.
 
Thank you, mark_j.

I am relatively new to FreeBSD IPC, was suspecting a leak lurking around in FreeBSD latest code, due to the continuous growth in mbuf memory usage.

I am writing a program to identify the plausible leak.

netstat -m seems to be a good starting point, thank you for the pointer.
 
I also see an mbuf 'leak' on my system since FreeBSD 13.0. I had to greatly increase the mbuf limits on this home server (nothing fancy), in order to survive the first few minutes after booting without completely locking network traffic.

I only have a 'bge' network adapter connecting to my home network (1 GBit). The only thing interesting on my network is that it also transports multicast TV streams on a separate vlan.
Code: 
$ vmstat -z
ITEM                   SIZE  LIMIT     USED     FREE      REQ     FAILSLEEP XDOMAIN
kstack_cache:         16384,      0,     348,       4,    3939,   0,   0,   0
buf free cache:         272,      0,    2233,      29,  279709,   0,   0,   0
vm pgcache:            4096,      0,   87447,       1,34863912, 352,   0,   0
debugnet_mbuf:          256,      0,       0,       0,   33024,   0,   0,   0
debugnet_mbuf_cluster:   2048,      0,       0,       0,   32768,   0,   0,   0
debugnet_mbuf_packet:   2048,      0,       0,       0,       0,   0,   0,   0
UMA Kegs:               256,      0,      86,       7,      86,   0,   0,   0
UMA Zones:              320,      0,      99,       7,      99,   0,   0,   0
UMA Slabs 0:             56,      0,   17571,     354,   19035,   0,   0,   0
UMA Slabs 1:             88,      0,       5,      40,       5,   0,   0,   0
UMA Hash:               128,      0,       0,       0,       0,   0,   0,   0
2 Bucket:                20,      0,      31,     171, 4395650,   0,   0,   0
4 Bucket:                28,      0,       1,     144, 1462685,   0,   0,   0
8 Bucket:                44,      0,       2,      90, 2089685,   0,   0,   0
16 Bucket:               76,      0,      20,      86,  611339,   0,   0,   0
32 Bucket:              128,      0,      35,     523,  399831,  32,   0,   0
64 Bucket:              256,      0,      29,     241,   22922,   0,   0,   0
128 Bucket:             512,      0,      17,     142,   22522,   0,   0,   0
256 Bucket:            1024,      0,      61,      83,   88304, 122,   0,   0
SMR SHARED:              16,      0,       2,      29,       2,   0,   0,   0
SMR CPU:                 20,      0,       2,      29,       2,   0,   0,   0
vmem:                   960,      0,       1,       1,       1,   0,   0,   0
vmem btag:               28,      0,   35651,    2157,   47527, 520,   0,   0
VM OBJECT:              164,      0,   56794,    1238, 7428393,   0,   0,   0
RADIX NODE:              44, 117208,   18464,    2692,10961005,   0,   0,   0
KMAP ENTRY:              56,      0,       3,      66,       3,   0,   0,   0
MAP ENTRY:               56,      0,   93065,    8455,25222449,   0,   0,   0
VMSPACE:                244,      0,      89,      55,  472153,   0,   0,   0
fakepg:                  68,      0,       0,       0,       0,   0,   0,   0
pcpu-4:                   4,      0,       0,       0,       0,   0,   0,   0
pcpu-8:                   8,      0,    2613,     459,    2620,   0,   0,   0
pcpu-16:                 16,      0,       9,     247,       9,   0,   0,   0
pcpu-32:                 32,      0,       0,       0,       0,   0,   0,   0
pcpu-64:                 64,      0,     411,      37,     411,   0,   0,   0
malloc-16:               16,      0,    9198,     669, 1929679,   0,   0,   0
malloc-32:               32,      0,    4435,    5598,10218855,   0,   0,   0
malloc-64:               64,      0,    9760,    2966, 4345962,   0,   0,   0
malloc-128:             128,      0,   24296,    1837,  305130,   0,   0,   0
malloc-256:             256,      0,    1196,     409,  228759,   0,   0,   0
malloc-384:             384,      0,     410,     140,  529229,   0,   0,   0
malloc-512:             512,      0,     123,      13,     441,   0,   0,   0
malloc-1024:           1024,      0,      69,      11,  164975,   0,   0,   0
malloc-2048:           2048,      0,     364,       6,  510696,   0,   0,   0
malloc-4096:           4096,      0,     184,      47, 1740254,   0,   0,   0
malloc-8192:           8192,      0,      16,       1,     379,   0,   0,   0
malloc-16384:         16384,      0,      17,       3,   42377,   0,   0,   0
malloc-32768:         32768,      0,       7,       1,     204,   0,   0,   0
malloc-65536:         65536,      0,       2,       1,    9086,   0,   0,   0
SLEEPQUEUE:              44,      0,     352,      32,     352,   0,   0,   0
kenv:                   258,      0,       6,       9,    3036,   0,   0,   0
Files:                   60,      0,     606,     198, 4228303,   0,   0,   0
filedesc0:              868,      0,     112,      50,  472175,   0,   0,   0
PWD:                     16,      0,      31,     475,  236866,   0,   0,   0
rangeset pctrie nodes:     44,      0,       0,       0,       0,   0,   0,   0
rl_entry:                28,      0,     201,      89,     264,   0,   0,   0
TURNSTILE:               72,      0,     352,       0,     352,   0,   0,   0
umtx pi:                 52,      0,       0,       0,       0,   0,   0,   0
umtx_shm:                52,      0,   21215,       1,   21217,   0,   0,   0
swrbuf:                 340,    128,       0,      29,    1526,   0,   0,   0
swwbuf:                 340,     64,       0,      16,   26300,   0,   0,   0
mdpbuf:                 340,     25,       0,       0,       0,   0,   0,   0
nfspbuf:                340,    128,       0,       0,       0,   0,   0,   0
vnpbuf:                 340,    128,       0,       0,       0,   0,   0,   0
clpbuf:                 340,    128,       0,      32,    6921,   0,   0,   0
pbuf:                   340,     16,       0,     553,       0,   0,   0,   0
MAC labels:              20,      0,       0,       0,       0,   0,   0,   0
PROC:                   840,      0,     112,      50,  472174,   0,   0,   0
PGRP:                    48,      0,      50,      34,   10082,   0,   0,   0
THREAD:                 896,      0,     348,       3,    3939,   0,   0,   0
cpuset:                  44,      0,       4,      60,       6,   0,   0,   0
domainset:               20,      0,       0,       0,       0,   0,   0,   0
audit_record:          1136,      0,       0,       0,       0,   0,   0,   0
mbuf_packet:            256,      0,     512,     163, 1480821,   0,   0,   0
mbuf:                   256, 189496,   33128,     307, 2205061,   0,   0,   0
mbuf_cluster:          2048,  98304,   33443,       1,   33443,   0,   0,   0
mbuf_jumbo_page:       4096,  14804,       0,     143,   38940,   0,   0,   0
mbuf_jumbo_9k:         9216,   4386,       0,       0,       0,   0,   0,   0
mbuf_jumbo_16k:       16384,   2467,       0,       0,       0,   0,   0,   0
epoch_record pcpu:      192,      0,       4,      17,       4,   0,   0,   0
ttyinq:                 152,      0,     540,       6,    1050,   0,   0,   0
ttyoutq:                256,      0,     281,       4,     552,   0,   0,   0
DEVCTL:                1024,   1000,       0,     100,     203,   0,   0,   0
g_bio:                  264,      0,       0,     540, 2039032,   0,   0,   0
cryptop:                212,      0,       0,       0,       0,   0,   0,   0
vtnet_tx_hdr:            16,      0,       0,     253,       0,   0,   0,   0
FPU_save_area:          512,      0,       1,       7,       1,   0,   0,   0
VNODE:                  300,      0,   33679,     108, 1282433,   0,   0,   0
BUF TRIE:                44,      0,     388,    3384,  220510,   0,   0,   0
S VFS Cache:             72,      0,   32223,   14033, 1083945,   0,   0,   0
STS VFS Cache:           96,      0,       0,       0,       0,   0,   0,   0
L VFS Cache:            288,      0,     305,     143,  232526,   0,   0,   0
LTS VFS Cache:          312,      0,       0,       0,       0,   0,   0,   0
NAMEI:                 1024,      0,       0,       8, 8610397,   0,   0,   0
NCLNODE:                416,      0,       0,       0,       0,   0,   0,   0
TMPFS node:             144,      0,      18,      38,      55,   0,   0,   0
DIRHASH:               1024,      0,     178,     278,   13161,   0,   0,   0
pipe:                   416,      0,      68,      40,  390851,   0,   0,   0
Mountpoints:           2624,      0,       8,       1,       8,   0,   0,   0
AIO:                    116,      0,       0,       0,       0,   0,   0,   0
AIOP:                    16,      0,       0,       0,       0,   0,   0,   0
AIOCB:                  336,      0,       0,       0,       0,   0,   0,   0
AIOLIO:                 168,      0,       0,       0,       0,   0,   0,   0
ksiginfo:                80,      0,     150,     100,  117855,   0,   0,   0
itimer:                 232,      0,       0,       0,       0,   0,   0,   0
KNOTE:                  116,      0,     105,     140,  169778,   0,   0,   0
socket:                 572,  15299,     197,      62,  235032,   0,   0,   0
IPsec SA lft_c:          16,      0,       0,       0,       0,   0,   0,   0
ipq:                     36,   3072,       0,       0,       0,   0,   0,   0
udp_inpcb:              316,  15299,      29,      43,  129407,   0,   0,   0
udpcb:                   20,  15299,      29,     173,  129407,   0,   0,   0
tcp_inpcb:              316,  15299,      68,      76,   42922,   0,   0,   0
tcpcb:                  800,  15299,      59,      46,   42922,   0,   0,   0
tcptw:                   60,   3059,       9,     125,   15396,   0,   0,   0
syncache:               128,  15360,       0,      31,    6520,   0,   0,   0
hostcache:               76,  15360,      69,      37,     690,   0,   0,   0
sackhole:                20,      0,       0,     202,     206,   0,   0,   0
tcpreass:                32,   6144,       0,     127,    7408,   0,   0,   0
udplite_inpcb:          316,  15299,       0,       0,       0,   0,   0,   0
ripcb:                  316,  15299,       5,       7,     901,   0,   0,   0
unpcb:                  192,  15299,      97,      50,   61787,   0,   0,   0
routing nhops:          256,      0,      11,       4,      19,   0,   0,   0
rtentry:                104,      0,      16,      23,      19,   0,   0,   0
swpctrie:                44, 753756,     827,      93,    8848,   0,   0,   0
swblk:                   72, 753704,   26064,     256,   54326,   0,   0,   0
FFS inode:              116,      0,   33535,     310, 1282145,   0,   0,   0
FFS1 dinode:            128,      0,       0,       0,       0,   0,   0,   0
FFS2 dinode:            256,      0,   33535,     170, 1282145,   0,   0,   0

After booting the mbuf count remains pretty stable, but I have to allocate 72 MB just for the MBufs if I read things correctly
 
You must log in or register to reply here.
Back
Top