ZFS Pool Performance When Using Geli

[fr33bsd]

Hi people,

on one of my storage servers (12 spin disks), I run geli with AES CBC encryption due to prevent physical data hijacking by evil data center personel .

As far as I know AES CBC is slower than AES XTC. How ever, I would like to ask you about your performance experience regarding using geli.

My raidz3 12 sata disk D1521 (2,4GHz) 10gb nic server with broadcom 9400-16i hba is performing about 220mb/s in writing (seq.w. only) over the network using netcat.

I know lots of mirrored vdevs perform better in raidz3 and there are sed disks, but this is not the point. The focus here might be on geli encryption.

 

[Cath O'Deray]

Low-end perspective (single HDD, HP EliteBook 850p): as far as I'm aware, no significant impact on performance.

 

[Eric A. Borisch]

You can use hotkernel from sysutils/dtrace-toolkit when writing to see where the kernel is busy. If you load aesni (and your processor supports it) you can push the overhead even lower: https://rubenerd.com/checking-if-freebsd-geli-is-using-aes-ni/

 

[SirDice]

Note that on FreeBSD 13.0 geli(8) will report "accelerated software" instead of "hardware".

geli(8) now reports the use of accelerated software cryptography (such as AES-NI on x86 CPUs) as "accelerated software" rather than "hardware". This is purely a change in naming, and does not imply reduced performance or support. a3d565a1188f (Sponsored by Chelsio Communications)

FreeBSD 13.0-RELEASE Release Notes

FreeBSD is an operating system used to power modern servers, desktops, and embedded platforms.

www.freebsd.org

 

[fr33bsd]

You can use hotkernel from sysutils/dtrace-toolkit when writing to see where the kernel is busy. If you load aesni (and your processor supports it) you can push the overhead even lower: https://rubenerd.com/checking-if-freebsd-geli-is-using-aes-ni/

Thanks Eric

 

[fr33bsd]

Note that on FreeBSD 13.0 geli(8) will report "accelerated software" instead of "hardware".

FreeBSD 13.0-RELEASE Release Notes

FreeBSD is an operating system used to power modern servers, desktops, and embedded platforms.

www.freebsd.org

In fact, it is hardware accelerated, but cbc (essiv) is slower than xtc, and - as far as I know - cbc is computed by cpu without usage of embedded instruction set like aes-ni, mmx and so on ... meaning aes-xts uses aes-ni only, but aes-cbc-essiv uses aes-ni and it additionally computes sha256 (as far as I remember correctly) 'in software mode' ... in new cpus there is sha part of the IS, so I think it cbc could be faster on those cpus.

Regarding the Benchmark: It is no file encryption benchmark, so (geli and openssl aes-xtc/cbc) both actually differ here

$ openssl speed -evp aes-128-xts
Doing aes-128-xts for 3s on 16 size blocks: 144322531 aes-128-xts's in 3.00s
Doing aes-128-xts for 3s on 64 size blocks: 107568538 aes-128-xts's in 3.00s
Doing aes-128-xts for 3s on 256 size blocks: 56227297 aes-128-xts's in 3.00s
Doing aes-128-xts for 3s on 1024 size blocks: 18042450 aes-128-xts's in 3.00s
Doing aes-128-xts for 3s on 8192 size blocks: 2437685 aes-128-xts's in 3.00s
LibreSSL 2.8.3
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) blowfish(idx) 
compiler: information not available
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-xts     770352.88k  2295995.52k  4798117.06k  6158651.78k  6654812.63k
$ openssl speed -evp aes-128-cbc
Doing aes-128-cbc for 3s on 16 size blocks: 262272544 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 64 size blocks: 71872008 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 256 size blocks: 18681349 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 1024 size blocks: 4697442 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 8192 size blocks: 587447 aes-128-cbc's in 3.00s
LibreSSL 2.8.3
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) blowfish(idx) 
compiler: information not available
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc    1399612.67k  1533686.67k  1594167.29k  1601351.81k  1604237.98k