Nginx-FreeBSD slow download!

S

shahzaib

Hi,

We've recently shifted to FreeBSD 10.1-RELEASE due to its robust asynchronous performance for big storage based on .mp4 files. Here is the server specs :

2 x Intel Xeon X5690
96GB DDR3 Memory
12 x 3TB SATA Raid-10 (HBA LSI-9211)
ZFS FileSystem with 18TB usable space
2 x 1Gbps LACP (2Gbps Throughput)

Things are working quite well, no high I/O due to Big Ram cache and AIO performance but once network port started to go over 1Gbps, performance begins to lag, download speed started to stuck around 60-100Kbps on a 4Mbps connection (using wget) which works quite efficient under 800Mbps port (450kbps on 4Mbps). We first thought it could be network issue or LACP issue but doesn't looks like it is, we tested with download/upolading from this server on 2Gbps throughput. We also checked that if requests are in queue using command netstat -Lan but it was '0' .We also checked with Disk Busy rate using gstat(8) which was quite stable as well.

So it looks like either the sysctl(1) values need to tweak or NGINX configurations are not optimized.

Following is the link with all configs performed till now :

http://pastebin.com/EHUHV7SH
====================================================

Please iI need guidance to handle with this problem, iI am sure that some value needs to tweak.

Thanks in advance !!
 
Oko said:
Did you create ZFS pool on the top of the hardware RAID:confused:?
Click to expand...
No, we created using mirroring of each pair (2 x Sata) followed by rest of the drives i.e.

zpool create storage mirror da0 da1 mirror da2 da3 mirror da4 da5 mirror da6 da7 mirror da8 da9 mirror da10 da11
 
I've also observed lots of retransmitted packets in netstat -s output. Here is the status :

http://prntscr.com/9xa6z2

When iI try to download file using wget on 4Mbps . This is what the behavior of download is:

- Download starts --> speed goes up --> download stucks for 2-3 seconds (much like a pause, no progress in download) --> process resume and speed goes up --> stucks again --> resume etc.
 
kpa said:
I would first disable all TSO options on the interfaces and see if it makes a difference.
Click to expand...
Could you please let me know if there would be downtime for disabling TSO? If you can provide me command to do that, it'd be very kind.

Do iI need to disable TSO in all Active interfaces ? i.e Igb0, Igb2, lagg0.
 
Last edited by a moderator:
It should be done on the physical interfaces (igb0,igb1) and shouldn't create downtime.

ifconfig igb0 -tso
 
SirDice said:
It should be done on the physical interfaces (igb0,igb1) and shouldn't create downtime.

ifconfig igb0 -tso
Click to expand...
Thanks, II've disabled tso on two servers to see how it goes. Will keep you updated. Thanks !!
 
Last edited by a moderator:
You could ask on the freebsd-net mailing list, many of the NIC driver developers are there including Jack Vogel who (as I understand) is the main contact at Intel for FreeBSD driver issues.
 
You must log in or register to reply here.
Back
Top