ZFS ZFS settings/tuning for 16GB RAM

H

henning

Hi,

I'm using FreeBSD 11.2-RELEASE-p4 as Desktop OS on my Lenovo X230 with 16GB RAM and ZFS as file system. This is my first FreeBSD install. So far, I haven't had any issues with ZFS, but I have some slightly concerning boot messages and was wondering if I should change any ZFS-related configurations. In particular, I have two questions:

1) What is the recommended kmem_size for a machine with 16GB RAM?

I've had the following warning:

Code: 
[henning@localhost ~]$ dmesg | grep -A 2 'ZFS'
...
ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable behavior.
            Consider tuning vm.kmem_size and vm.kmem_size_max in /boot/loader.conf.
...

In /boot/loader.conf, I then set vm.kmem_size="512M" and vm.kmem_size_max="512M".

Result:
Code: 
[henning@localhost ~]$ sysctl -h vm.kmem_size
vm.kmem_size: 536.870.912

The above boot message disappeared as expected. However, I have no clue whether these settings are optimal or even make sense. What is a good value for 16GB RAM? Is there anything else to consider besides memory size?

2. How to correctly set kstack_pages (if at all)?

During boot, I get:

Code: 
[henning@localhost ~]$ dmesg | grep -A 2 'ZFS'
...
ZFS NOTICE: KSTACK_PAGES is 2 which could result in stack overflow panic!
Please consider adding 'options KSTACK_PAGES=4' to your kernel config
...

I tried to fix this by adding kern.kstack_pages=4 to boot/loader.conf. This seems to work, since after a reboot, I get:

Code: 
[henning@localhost ~]$ sysctl kern.kstack_pages
kern.kstack_pages: 4

However, the boot message above persist. How come? Is this something to worry about?

Thank you!
 
Last edited:
Unless this is an i386 machine you shouldn't touch any of those settings.
 
There was a bug a while ago, and it was something like this: on an i386, even if you increase kern.kstack_pages to 4, the warning message is still printed, but the parameter is set correctly. Or some variation of that. I vaguely remember that being in version 11.X (and I don't remember which value of X). Do a search here on the forum, and you should find some information about that.
 
Since you are on a 64-bit machine with sufficient memory, I have no personal experience, and it seems to me that SirDice's advice is likely to be correct.
 
henning said:
You're right, thanks. The thread is here.
Do you agree with SirDice that these settings should not be touched at all?
Click to expand...
This is what people usually recommend - don't touch ZFS tuning unless you need to (some problem). The system is designed to work well in most situations and have sensible default behavior.
 
I do recommend upgrading though, not so much for the messages but because 11.2 will be End-of-Life soon (at the end of this month).
 
henning said:
However, the boot message above persist. How come? Is this something to worry about?
Click to expand...
Please post the output of # uname -a. These messages should only come out if you are running an i386-architecture image. If you are running the recommended amd64-architecture image, you shouldn't see them unless you have made some unusual changes to the system configuration files. Note: There isn't any supported way to upgrade in-place from i386 to amd64 - it needs a clean install.
 
Not strictly related to kmem_size but even on 11.2 / 11.3 there are problems with ZFS eating too much of available RAM under heavy load (rsync backup etc.) and then OOM killer will terminate some processes.
We have always limited arc_max to leave enough space for other processes. The size or arc_max depends on what other services are running on machine in question.
kmem_size left on autotuning defaults:


Code: 
vm.kmem_map_free: 22151868416
vm.kmem_map_size: 11076800512
vm.kmem_size_scale: 1
vm.kmem_size_max: 1319413950874
vm.kmem_size_min: 0
vm.kmem_zmax: 65536
vm.kmem_size: 33228668928

on machine with 32GB RAM
 
Terry_Kennedy said:
Please post the output of # uname -a. These messages should only come out if you are running an i386-architecture image. If you are running the recommended amd64-architecture image, you shouldn't see them unless you have made some unusual changes to the system configuration files.
Click to expand...

Code: 
[henning@localhost ~]$ uname -a
FreeBSD localhost 11.2-RELEASE-p4 FreeBSD 11.2-RELEASE-p4 #0: Thu Sep 27 07:46:01 UTC 2018     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  i386

Looks like I am running an i386-architecture image?

Note: There isn't any supported way to upgrade in-place from i386 to amd64 - it needs a clean install.
Click to expand...

This would explain why I broke my system when I tried to update. (At least I had a zfs snapshot for the meantime, but I'm really not looking forward to doing a clean install....)
 
henning said:
Looks like I am running an i386-architecture image?
Click to expand...
Correct. Which actually means you're limited to 4GB, even if the machine has 16GB of memory. You can use PAE but I really don't recommend it, switch to a 64 bit FreeBSD. And no, you can't "upgrade" 32 bit to 64 bit. Although you might be able to pull it off with a source install, it's not a supported path.
 
SirDice said:
Correct. Which actually means you're limited to 4GB... And no, you can't "upgrade" 32 bit to 64 bit...
Click to expand...

Dang. Stupid beginner's mistake (amd64 sounds so much like "amd, not intel")! Thanks so much, guys, for pointing this out. Explains a lot.
 
henning said:
but I'm really not looking forward to doing a clean install....
Click to expand...

Even if it is not officially supported you can upgrade your 32bit system to 64bit. I did it in the past on the remote machines through SSH. From 8.4-i386 to 10.3-amd64.

We are using centralized build server where all updates are prepared by make buildworld, make buildkernel and mounted through NFS to target machines. On target machine we mount it read only a usually run just make installkernel and installworld. This cannot be used to install 64 bit system on 32 bit running system.
The actual workaround was to run make installkernel DESTDIR=/vol0/10.3_release and make installworld DESTDIR=/vol0/10.3_release on the build machine and then pack it by cd /vol0/10.3_release && tar czvf /vol0/10.3_release.tgz ./.

The dist set was not necessary but is useful
make distribution DESTDIR=/vol0/10.3_release
cd /vol0/10.3_release && tar czvf /vol0/10.3_release.dist.tgz ./

Both archives were transferred to target machine by scp

I think it will be possible with base.txz and karnel.txz from official FTP ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.3-RELEASE/ but you need to exclude /etc/ when untaring the archive. Otherwise it will overwrite your customizations in /etc/ like users in master.passwd, settings in rc.conf and so on.

I also do some additional steps before unpacking the tar archive. Stop cron and all unnecessary services or reboot to single user mode if you have local access to your machine
mv /rescue /resc32
rm -r /usr/share/openssl/man/cat{1,3}
sysctl kern.geom.debugflags=16

Unpack some unmodified default files to /etc/
cd / && tar xvf /home/quip/10.3_release.dist.tgz ./etc/services ./etc/devfs.conf ./etc/devd.conf ./etc/termcap ./etc/termcap.small ./etc/network.subr ./etc/netstart ./etc/rc.subr ./etc/gettytab ./etc/mtree/ ./etc/rc.d/ ./etc/defaults/ ./etc/devd/ ./etc/periodic/

Unpack you new 64 bit system
cd / && tar xvf /home/quip/10.3_release.tgz

Write new boot code (it will be different device with ZFS etc.)
/resc32/gpart bootcode -b /boot/boot mirror/gm0s1

/resc32/reboot

If I did not forget something the machine should boot up with fresh new 64 bit system but as it was not proper upgrade we did run usual make installkernel && make installworld && mergemaster at this point to be sure everything is in the right place.

You can delete 32 bit rescue now rm -r /resc32/

Then all packages must be reinstalled. ( first run pkg-static install -f pkg to update pkg it-self.

In our case with upgrade from 8.4 to 10.3 I must manually delete some files
rm /sbin/sconfig
rm /usr/sbin/apmd /usr/sbin/kgzip /usr/sbin/pnpinfo /usr/sbin/wlconfig

I hope it helps you a bit. If anything goes wrong, you can go back with ZFS BE ( beadm), restore from backup or boot from Live media USB flashdisk and try to unpacky / install system files from Live media, then restore your changes in /etc/ files from backup.
 
Quip said:
Even if it is not officially supported you can upgrade your 32bit system to 64bit.
Click to expand...

Thank you so much! I really didn't expect such a detailed response. This is beyond helpful. I'll try this as soon as I have some spare time and post any follow up questions in a new thread to make it easier for other users to find this information.
 
You must log in or register to reply here.
Back
Top