out of swap space

tarkhil

tarkhil

Code: 
Apr 11 21:00:41 bipgame kernel: pid 10470 (mysqld), jid 2, uid 88, was killed: out of swap space
Apr 11 21:00:46 bipgame kernel: pid 31041 (bhyve), jid 0, uid 0, was killed: out of swap space
Apr 11 21:00:46 bipgame kernel: pid 42311 (node), jid 7, uid 0, was killed: out of swap space
Apr 11 21:00:46 bipgame kernel: pid 61686 (php), jid 20, uid 1001, was killed: out of swap space
this disaster hits me from time to time, and I'm asking for advice.

  1. I do have lots of swap space. Monitoring does not show growing memory usage.
  2. It hits sometimes between 00:30 and 01:50 of an hour, about 2-3 times a week. No, nothing suspicious in crontabs.
What can I check and/or tune? FreeBSD 13.0-RC5 (which is almost 13.0-RELEASE)
 
I recall from several threads in regards to mysql frequently consuming a lot of memory, more specifically from when it goes and backups the database(s) and or database dumps. By any chance you have any jobs that is scheduled to backup the system and/or the database?
 
It will be a bit sad if it is MySQL. My brief testing with one of the 13.0 BETA/RC releases seemed to handle the MySQL import a lot better.

I've also noticed long-running rsync processes dipping into swap.

As for advice - I have this running every minute to try and nail down what processes are doing what - to try and find a pattern:
Code: 
@every_minute date >> /tmp/swap_usage.txt && top -n 100 >> /tmp/swap_usage.txt && pstat -sh >> /tmp/swap_usage.txt

Keep an eye on the tmp file and if you are lucky you may find a pattern.

A minute is eons in terms of what can happen on a modern machine, but hopefully it might give you some things that are worth investigating further.
 
Are you running ZFS in combination with bhyve and MySQL? Then make sure you limit vfs.zfs.arc_max. Also check with a tool like databases/mysqltuner that your MySQL server isn't using too much memory. And off course double check if ARC + bhyve memory + MySQL memory usages don't use a lot more than your machine actually has.
 
SirDice said:
Are you running ZFS in combination with bhyve and MySQL? Then make sure you limit vfs.zfs.arc_max. Also check with a tool like databases/mysqltuner that your MySQL server isn't using too much memory. And off course double check if ARC + bhyve memory + MySQL memory usages don't use a lot more than your machine actually has.
Click to expand...
zfs, and bhyve (but mysql is in jail). Mysqltuner claims that
[OK] Maximum possible memory usage: 4.6G (7.22% of installed RAM)

so it cannot request THAT much memory.

Okay, I'll think about limiting ARC, but it was not a problem for quite a time.
 
Definitely try to limit ARC. Apparently it can get a bit lazy and not release memory quick enough, so you get a battle between MySQL, bhyve and ARC all trying to grab the same memory. That usually doesn't end well. Tweaking ARC usually helps.
 
I'll do
SirDice said:
Definitely try to limit ARC. Apparently it can get a bit lazy and not release memory quick enough, so you get a battle between MySQL, bhyve and ARC all trying to grab the same memory. That usually doesn't end well. Tweaking ARC usually helps.
Click to expand...

but isn't laundry too big?
Code: 
Mem: 17G Active, 2347M Inact, 21G Laundry, 15G Wired, 2340M Free
ARC: 7896M Total, 1061M MFU, 4808M MRU, 54M Anon, 202M Header, 1717M Other
     4579M Compressed, 6112M Uncompressed, 1,33:1 Ratio
Swap: 128G Total, 13G Used, 115G Free, 9% Inuse, 108K In
 
Agree, there was at some point an application that used up all memory, it's usually not the ones that get killed, those are typically innocent bystanders. Figure out which application or process got hammered. Maybe a run-away web process? Besides ZFS, MySQL, bhyve, node and PHP what else is running on that machine?
 
SirDice said:
Definitely try to limit ARC. Apparently it can get a bit lazy and not release memory quick enough
Click to expand...
This is definitely much better on 13, though – I've seen ARC drop by several GB from one second to the next. Maybe there are still situations that require limiting it, but I'd observe this a bit first.
 
Did you try my cron suggestion? Leave it running for 24-48 hours and see if you can see any pattern.

MySQL is my problem child (on 11.x and 12.x certainly) - I've had to make the %age of RAM smaller and smaller and smaller to make it play nicely with Apache and PHP. I don't use ZFS so ARC not an issue in my case. I think 4-5GB max allocated to MySQL on a 32GB RAM machine - even though plenty of recommendations say 25% or 50% RAM (obviously those depend on what else is running on the same machine).
 
richardtoohey2 said:
I think 4-5GB max allocated to MySQL on a 32GB RAM machine
Click to expand...
You should look at that size of the data. Adding too much memory to MySQL is bad for its performance, it'll spend more time managing caches and buffers than actually using it. Ideally you want to assign a little more than the data to allow for some growth. But don't overdo it.
 
You must log in or register to reply here.
Back
Top