how to find out memory leak on 14.1-RELEASE-p4

R

RPUgCZUqR4qZ

We have few servers with 14.1-RELEASE-p4, all of them have significant memory leak
Please, help us find out what exactly leaking and fix it.
Intersting observation is 14.1-RELEASE-p3 don't have the issue, with same set of services/processes/aplications.

here is top header
Code: 
Mem: 135G Active, 31G Inact, 60G Laundry, 32G Wired, 56K Buf, 6099M Free
ARC: 18G Total, 5280M MFU, 10G MRU, 11M Anon, 274M Header, 2380M Other
     14G Compressed, 26G Uncompressed, 1.90:1 Ratio
Swap: 66G Total, 20G Used, 46G Free, 29% Inuse, 104K In
mem line is 135+31+60+32+6 = 264 not 272Gb total installed RAM

Code: 
sysctl vfs.zfs.arc_min; sysctl vfs.zfs.arc_max
vfs.zfs.arc_min: 17179869184
vfs.zfs.arc_max: 34359738368


sysctl
Code: 
hw.physmem: 291599888384
hw.realmem: 294191628288

example memory gap, it can be up to 50%
Code: 
SYSTEM MEMORY INFORMATION:
mem_wire:       28933750784 (  27593MB) [ 10%] Wired: disabled for paging out
mem_active:  + 125202481152 ( 119402MB) [ 44%] Active: recently referenced
mem_inactive:+  12180725760 (  11616MB) [  4%] Inactive: recently not referenced
mem_cache:   +            0 (      0MB) [  0%] Cached: almost avail. for allocation
mem_free:    +  10700419072 (  10204MB) [  3%] Free: fully available for allocation
mem_gap_vm:  + 107080982528 ( 102120MB) [ 37%] Memory gap: UNKNOWN                 <<<<<<<<<<<<<<<<<<<=====================
-------------- ------------ ----------- ------
mem_all:     = 284098359296 ( 270937MB) [100%] Total real memory managed
mem_gap_sys: +   7501529088 (   7154MB)        Memory gap: Kernel?!
-------------- ------------ -----------
mem_phys:    = 291599888384 ( 278091MB)        Total real memory available
mem_gap_hw:  +  51997495296 (  49588MB)        Memory gap: Segment Mappings?!
-------------- ------------ -----------
mem_hw:      = 343597383680 ( 327680MB)        Total real memory installed

SYSTEM MEMORY SUMMARY:
mem_used:      320716238848 ( 305858MB) [ 93%] Logically used memory
mem_avail:   +  22881144832 (  21821MB) [  6%] Logically available memory
-------------- ------------ ----------- ------
mem_total:   = 343597383680 ( 327680MB) [100%] Logically total memory

vmstat -z and vmstat -m attached
 

Attachments

I don't know FreeBSD enough but on Linux if there are objects in memory that are allocated but not used they are "parked" in the swap and the behaviour seen is that the swap usage increases but there's no disk activity on it.

At work to debug a massive memory leak we use a script (that I'm not allowed to post here for obvious reasons even if I'm the one who wrote it) that prints out the list of the running programs that are leaking memory in the swap. It's not exacly the same but this page may help you (of course it must be "translated" to be used with FreeBSD ).
 
cracauer@ said:
Do you still have a comparable p3 machine running?
Click to expand...
Yes
We have one server with p3 and two with p4.
All of them have
~ zfs version
zfs-2.2.4-FreeBSD_g256659204
zfs-kmod-2.2.4-FreeBSD_g256659204
~ sha256sum /boot/kernel/zfs.ko
4e944f4cdd86d0ad0d1ac5f6c7eaf98f7e63db4c65c43a3f31e022c967162546 /boot/kernel/zfs.ko
Click to expand...

just for reference
We have 14.0-RELEASE with significant memory leak, but it's OK. We know about ZFS memory leak issue, we will upgrade on this weekend.
 
p4

vfs.zfs.arc_min: 17179869184
vfs.zfs.arc_max: 34359738368
hw.physmem: 343125901312
hw.realmem: 345731760128
SYSTEM MEMORY INFORMATION:
mem_wire: 32380182528 ( 30880MB) [ 9%] Wired: disabled for paging out
mem_active: + 40740634624 ( 38853MB) [ 12%] Active: recently referenced
mem_inactive:+ 246216675328 ( 234810MB) [ 73%] Inactive: recently not referenced
mem_cache: + 0 ( 0MB) [ 0%] Cached: almost avail. for allocation
mem_free: + 13573840896 ( 12945MB) [ 4%] Free: fully available for allocation
mem_gap_vm: + 1398902784 ( 1334MB) [ 0%] Memory gap: UNKNOWN
-------------- ------------ ----------- ------
mem_all: = 334310236160 ( 318823MB) [100%] Total real memory managed
mem_gap_sys: + 8815665152 ( 8407MB) Memory gap: Kernel?!
-------------- ------------ -----------
mem_phys: = 343125901312 ( 327230MB) Total real memory available
mem_gap_hw: + 471482368 ( 449MB) Memory gap: Segment Mappings?!
-------------- ------------ -----------
mem_hw: = 343597383680 ( 327680MB) Total real memory installed

SYSTEM MEMORY SUMMARY:
mem_used: 83806867456 ( 79924MB) [ 24%] Logically used memory
mem_avail: + 259790516224 ( 247755MB) [ 75%] Logically available memory
-------------- ------------ ----------- ------
mem_total: = 343597383680 ( 327680MB) [100%] Logically total memory
Click to expand...

p3

vfs.zfs.arc_min: 17179869184
vfs.zfs.arc_max: 34359738368
hw.physmem: 411860537344
hw.realmem: 414450188288
SYSTEM MEMORY INFORMATION:
mem_wire: 60497170432 ( 57694MB) [ 15%] Wired: disabled for paging out
mem_active: + 21439823872 ( 20446MB) [ 5%] Active: recently referenced
mem_inactive:+ 303705014272 ( 289635MB) [ 75%] Inactive: recently not referenced
mem_cache: + 0 ( 0MB) [ 0%] Cached: almost avail. for allocation
mem_free: + 14997446656 ( 14302MB) [ 3%] Free: fully available for allocation
mem_gap_vm: + 653242368 ( 622MB) [ 0%] Memory gap: UNKNOWN
-------------- ------------ ----------- ------
mem_all: = 401292697600 ( 382702MB) [100%] Total real memory managed
mem_gap_sys: + 10567839744 ( 10078MB) Memory gap: Kernel?!
-------------- ------------ -----------
mem_phys: = 411860537344 ( 392780MB) Total real memory available
mem_gap_hw: + 456323072 ( 435MB) Memory gap: Segment Mappings?!
-------------- ------------ -----------
mem_hw: = 412316860416 ( 393216MB) Total real memory installed

SYSTEM MEMORY SUMMARY:
mem_used: 93614399488 ( 89277MB) [ 22%] Logically used memory
mem_avail: + 318702460928 ( 303938MB) [ 77%] Logically available memory
-------------- ------------ ----------- ------
mem_total: = 412316860416 ( 393216MB) [100%] Logically total memory
Click to expand...
 

Attachments

Hm. I see the p3 as having more RAM and using it. The major difference seems to be that the p3 machine has a lot more allocated in kernel malloc().

Are you sure that it isn't simply a userland process being bigger on p4?
 
Both servers have same load profile.
mysql/nginx/perl app/jail and few others

Main annoying thing right now is p4 won't free laundry memory even under pressure.
In same time p3 server don't have this problem at all.

sysctl variables almost same

Right now we I have only one workaround to free laundry memory to reduce risk of outage - restart mysql and jails with perl app.

Biggest user land process on both servers is mysql 88Gb on p4 and 100Gb on p4.

As I see under FreeBSD we don't have a good(simple) tool to investigate laundry memory.
 
But you never posted the laundry value (top header) for both machines. Could you post that with the processes visible?

There is no commit in releng/14.1 that could be related to the amount of laundry kept around. Or any touch of the VM system in general.

I continue to think that it is weird to have 60 GB laundry when so much swapspace is in use and the machine is probably currently under memory pressure. But how that could be related to p3 vs. p4 I can't see.
 
But you never posted the laundry value (top header) for both machines. Could you post that with the processes visible?
Click to expand...
sorry, I provided more detailed information from a perl script from sysctl variables

p4 top

last pid: 7749; load averages: 0.92, 1.68, 2.21 up 5+02:53:54 01:15:52
1734 processes:3 running, 1058 sleeping, 18 zombie, 655 waiting
CPU: 0.1% user, 0.5% nice, 0.9% system, 0.0% interrupt, 98.4% idle
Mem: 69G Active, 171G Laundry, 28G Wired, 1485M Free
ARC: 16G Total, 5336M MFU, 5571M MRU, 32M Anon, 267M Header, 5193M Other
7731M Compressed, 31G Uncompressed, 4.11:1 Ratio
Swap: 64G Total, 192M Used, 64G Free, 11M Out
Click to expand...
p4

Wed Sep 25 00:38:46 UTC 2024
SYSTEM MEMORY INFORMATION:
mem_wire: 33429495808 ( 31880MB) [ 9%] Wired: disabled for paging out
mem_active: + 54226014208 ( 51713MB) [ 16%] Active: recently referenced
mem_inactive:+ 10987433984 ( 10478MB) [ 3%] Inactive: recently not referenced
mem_laundry + 221791313920 ( 211516MB) [ 66%] Loundry: Pages eligible for laundering
mem_cache: + 0 ( 0MB) [ 0%] Cached: almost avail. for allocation
mem_free: + 12653129728 ( 12066MB) [ 3%] Free: fully available for allocation
mem_gap_vm: + 1222848512 ( 1166MB) [ 0%] Memory gap: UNKNOWN
-------------- ------------ ----------- ------
mem_all: = 334310236160 ( 318823MB) [100%] Total real memory managed
mem_gap_sys: + 8815665152 ( 8407MB) Memory gap: Kernel?!
-------------- ------------ -----------
mem_phys: = 343125901312 ( 327230MB) Total real memory available
mem_gap_hw: + 471482368 ( 449MB) Memory gap: Segment Mappings?!
-------------- ------------ -----------
mem_hw: = 343597383680 ( 327680MB) Total real memory installed

SYSTEM MEMORY SUMMARY:
mem_used: 319956819968 ( 305134MB) [ 93%] Logically used memory
mem_avail: + 23640563712 ( 22545MB) [ 6%] Logically available memory
-------------- ------------ ----------- ------
mem_total: = 343597383680 ( 327680MB) [100%] Logically total memory
Click to expand...

p3 top
last pid: 54330; load averages: 1.02, 1.50, 1.68 up 36+17:21:21 01:16:34
1821 processes:1775 sleeping, 46 zombie
CPU: 0.0% user, 1.1% nice, 0.5% system, 0.0% interrupt, 98.3% idle
Mem: 17G Active, 281G Inact, 120M Laundry, 51G Wired, 24G Free
ARC: 25G Total, 10G MFU, 10G MRU, 19M Anon, 606M Header, 5081M Other
17G Compressed, 43G Uncompressed, 2.55:1 Ratio
Swap: 2048M Total, 2048M Free
Click to expand...

p3
SYSTEM MEMORY INFORMATION:
mem_wire: 54790111232 ( 52251MB) [ 13%] Wired: disabled for paging out
mem_active: + 22099128320 ( 21075MB) [ 5%] Active: recently referenced
mem_inactive:+ 301627289600 ( 287654MB) [ 75%] Inactive: recently not referenced
mem_laundry + 126693376 ( 120MB) [ 0%] Loundry: Pages eligible for laundering
mem_cache: + 0 ( 0MB) [ 0%] Cached: almost avail. for allocation
mem_free: + 22488764416 ( 21446MB) [ 5%] Free: fully available for allocation
mem_gap_vm: + 160710656 ( 153MB) [ 0%] Memory gap: UNKNOWN
-------------- ------------ ----------- ------
mem_all: = 401292697600 ( 382702MB) [100%] Total real memory managed
mem_gap_sys: + 10567839744 ( 10078MB) Memory gap: Kernel?!
-------------- ------------ -----------
mem_phys: = 411860537344 ( 392780MB) Total real memory available
mem_gap_hw: + 456323072 ( 435MB) Memory gap: Segment Mappings?!
-------------- ------------ -----------
mem_hw: = 412316860416 ( 393216MB) Total real memory installed

SYSTEM MEMORY SUMMARY:
mem_used: 88200806400 ( 84114MB) [ 21%] Logically used memory
mem_avail: + 324116054016 ( 309101MB) [ 78%] Logically available memory
-------------- ------------ ----------- ------
mem_total: = 412316860416 ( 393216MB) [100%] Logically total memory
Click to expand...
 
Of course it is only a guess that the situation is related to “laundry memory” and only differs on these machines.
I've probably checked everything I can think of and I'm out of options.

Basically, I understand that this behavior is normal for freebsd, it's just that it's very confusing to see such a large amount of “laundry memory” on machines with p4, while it doesn't happen on a machine with p3.

p4: top 10 processes, RAM usage
~ ps axl | sort -n -k 8 | tail -n 10
0 97612 68595 8 21 1 1080364 620944 kqread SNJ - 1:30.34 /script/k_mojo (perl)
0 34058 92785 58 21 1 1975344 638004 kqread SNJ - 2:14.12 /script/k_mojo (perl)
0 37600 92785 0 21 1 2065456 647312 kqread SNJ - 5:02.32 /script/k_mojo (perl)
0 13890 92785 61 21 1 2272304 651616 kqread SNJ - 5:12.26 /script/k_mojo (perl)
0 72523 48197 6 21 1 1475604 898708 kqread SNJ - 3:34.03 /script/k_mojo (perl)
0 49246 1 43 68 1 5931460 1695824 futex INJ - 1023:17.74 /usr/bin/mysqlrouter -c /var/db/mysqlrouter/mysqlrouter.conf (RtM:bootstrap_r)
88 64715 90752 60 21 1 27924896 4417012 select INJ - 0:30.28 /usr/local/libexec/mysqld --defaults-extra-file=/usr/local/etc/mysql/my.cnf --basedir=/usr/local --datadir=/var/db/mysql --plugin-dir=/usr/local/lib/mysql/plugi
88 42021 14049 61 23 0 15890480 7246680 select SJ - 151:23.44 /usr/local/libexec/mysqld --defaults-extra-file=/usr/local/etc/mysql/my.cnf --basedir=/usr/local --datadir=/var/db/mysql/mysql --plugin-dir=/usr/local/lib/mysql
0 85776 1 46 21 1 91205004 89271312 select INJ - 2161:20.06 /usr/sbin/mysqld --defaults-extra-file=/etc/mysql/my.cnf --basedir=/usr --pid-file=/var/run/mysqld/mysqld.pid --user=root --daemonize (connection)
Click to expand...


p3: top 10 processes, RAM usage

~ ps axl | sort -n -k 8 | tail -n 10
0 88810 44097 3 21 1 943004 787444 select SNJ - 0:09.77 /script/k_mojo (perl)
0 44932 44578 5 21 1 1979292 963316 select SNJ - 6:27.64 /script/k_mojo (perl)
0 4740 44097 13 21 1 1459100 1142584 select SNJ - 0:11.10 /script/k_mojo (perl)
0 63781 1 14 21 1 1683300 1343316 select SNJ - 7:02.10 chrome: --headless --remote-debugging-port=9222 --disable-gpu --disable-translate --disable-extensions --disable-background-networking --safebrowsing-disable-a
0 63785 63781 28 68 1 1549464 1364552 uwait INJ - 0:45.36 chrome: --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US --service-sandbox-type=network --no-sandbox --use-angle=swiftshader-webgl -
0 42967 1 24 68 1 4719108 1555008 futex INJ - 434:50.67 /usr/bin/mysqlrouter -c /var/db/mysqlrouter/mysqlrouter.conf (io-13)
88 83881 81852 13 21 1 12425096 1910292 select SNJ - 261:18.76 /usr/local/libexec/mysqld --defaults-extra-file=/usr/local/etc/mysql/my.cnf --basedir=/usr/local --datadir=/var/db/mysql/mysql --plugin-dir=/usr/local/lib/mysq
0 63809 63808 1 68 1 5859520 2260032 uwait INJ - 40:05.05 /usr/local/bin/service_utility_go
0 82412 44097 0 4 1 4440988 3698596 select SNJ - 0:17.60 /script/k_mojo (perl)
0 18004 1 13 21 1 105360652 101960348 select SNJ - 5032:28.60 /usr/sbin/mysqld --defaults-extra-file=/etc/mysql/my.cnf --basedir=/usr --pid-file=/var/run/mysqld/mysqld.pid --user=root --daemonize (connection)
Click to expand...
 
cracauer@ said:
But how that could be related to p3 vs. p4 I can't see.
Click to expand...
This makes me think of the issues I had with MySQL:

forums.freebsd.org

Solved - FreeBSD 12.x and MySQL 5.7 and importing file with lots of small lines exhaust RAM and swap

Thought I'd try out the new 12.1 beta and RC releases but have got side-tracked down a MySQL 5.7 rabbit hole. When I try and import a 17Gb database dump file from mysqldump my machines run out of RAM then they swap, then they run out of that, then the OOM killer kicks in with sad results...
forums.freebsd.org forums.freebsd.org

But as you say, p3 to p4 isn't a lot of change so ... can't be related.
 
I had a server outage 40 min ago, server rebooted via sysctl kernel panic

Observations:
load average below 1
788 waiting processes
172Gb laundry RAM
top is running, but not able execute anything else, after exit from any command terminal just freeze
by last received data from iostat, zpool iostat - look like system disk too overloaded

last difference between p3 and p4 is compression algorithm zstd on p3 and lz4 on p4
 
Code: 
CPU:  0.8% user,  0.0% nice,  0.5% system,  0.0% interrupt, 98.7% idle
Mem: 12G Active, 2545M Inact, 427M Laundry, 44G Wired, 711M Buf, 2885M Free
ARC: 33G Total, 7652M MFU, 23G MRU, 12M Anon, 552M Header, 2182M Other
     15G Compressed, 43G Uncompressed, 2.90:1 Ratio
Swap: 8192M Total, 3477M Used, 4715M Free, 42% Inuse, 8192B In
 
It runs a single process ( part of MooseFS Distributed FS ), but this ran rock solid on 12.X before upgrading straight to 14.1.

Not sure why wired size is so big ? Right now it's definitely swapping but the size of the moose fs process stays the same, so it's not from there for sure
 
cracauer@ said:
Hm.

What kind of workload do you run on there?

Is the system currently swapping (vmstat) or is the allocated swapspace just sitting there?
Click to expand...

Code: 
CPU:  0.5% user,  0.0% nice,  0.7% system,  0.0% interrupt, 98.8% idle
Mem: 5095M Active, 1194M Inact, 2078M Laundry, 45G Wired, 531M Buf, 9547M Free
ARC: 33G Total, 8662M MFU, 22G MRU, 7583K Anon, 626M Header, 2253M Other
     16G Compressed, 42G Uncompressed, 2.69:1 Ratio
Swap: 3979M Total, 706M Used, 3274M Free, 17% Inuse, 56K In

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
76125 mfs          39  20  -19  9606M  8382M select  22  85:30  26.76% mfschunkserver

mfs process size will stay the same but Wired goes up and Swap usage increases. I have this happening on at least 7 servers. Swap will go from 0 usage to 50% in a few hours
 
To me it looks like bad decision making on part of the kernel with regards to keeping laundry and building I/O cache even in the face of memory pressure.

I did not follow the FreeBSD-12 to -13 development closely enough to know whether this area had a major change.

Is MooseFS using mmap() to connect to its on-disk storage? Do you know?
 
cracauer@ said:
To me it looks like bad decision making on part of the kernel with regards to keeping laundry and building I/O cache even in the face of memory pressure.

I did not follow the FreeBSD-12 to -13 development closely enough to know whether this area had a major change.

Is MooseFS using mmap() to connect to its on-disk storage? Do you know?
Click to expand...

Yes it does use mmap
 
You must log in or register to reply here.
Back
Top