sbwait is causing problem

[psyc]

I have a FreeBSD 9 server and I run it for a Drupal site, which uses Nginx and PHP-FPM and all those setting related with Drupal like Memcached, APC, MySQL.

The problem is now suddenly when the PHP-FPM is running, it calls the sbwait, and it takes too long time to load the page, I get lots of timeouts error in log. At first I thought it was a PHP issue, but it is not. I cannot get what sbwait does, I guess it means socket buffer in waiting position but why is it overkilling my application? Even when I restart the PHP-FPM process, and load the Drupal page via browser, I can see the sbwait opens by itself and queues the PHP-FPM process. It takes a few seconds to process the first page after a restart, then again the page execution time increases and causes timeouts.

What could possibly be wrong here? Why does calling the PHP-FPM process go directly in the sbwait queue and not being served by CPU or accept like other processes?

I have enough memory allocation for everything, PHP has memory of 512 MB, APC 1024 MB, Memcached 1024 MB, PHP-FPM config has pm.dynamic and pm.max-child 40 with pm.start-server 12.

 

[Terry_Kennedy]

The problem is now suddenly when the PHP-FPM is running, it calls the sbwait, and it takes too long time to load the page, I get lots of timeouts error in log.

The first thing to check when something stops working is "what changed?".

Was the server rebooted? Any new software installed, or packages updated? Has the amount of web traffic increased? Any possibility of a Denial-of-Service attack? Any unusual messages from # dmesg -a or in /var/log/messages?

Here is an (older) description of what sbwait means. It is normal for some processes to sit in that state. For example, rwhod(8) is normally in that state while it is waiting to receive information from other systems (the second item in the description I linked to).

Normally an application is written so that if it blocks on waiting for sending / receiving data, that will only affect the thread / child process handling that specific request, and it won't affect other requests to the server. I'm not familiar with the way the programs you're using handle connections, so I can't say if this is the case for them, or if the memory sizes you've set are appropriate.

 

[psyc]

The server was not rebooted, no new software installed, it's in development server so, no increase in traffic too. Maybe little bit increment in page load due to more files, but that should not be the case, as this server needs to be production ready and handle lots of traffic.

Message log doesn't show much info, and the most irritating thing is even after the page load sbwait keeps on running for a long time. which causes little bit CPU overhead. According to this link, sbwait is explained as

The sbwait wchan is present when a thread has invoked the in-kernel
sbwait() function to wait for a socket event. It's used in a number of
situations, but the main ones are:

- The thread is trying to send on a blocking socket, but there's
insufficient socket buffer space, so it must wait for space. This might
occur if it has managead to max out the bandwidth available to a TCP
connection, or flow control is in use and the receiver does not wish to
receive more data yet.

- The thread is trying to receive on a blocking socket, but there's not
enough data to satisfy the read request, so it must wait for data to be
received. It might be waiting for a remote TCP sender to have data
available, or for in-flight data to arrive.

But it's hard to understand why there is insufficient space for socket buffer, as I have only a few applications running, and the memory usage is not even 10% and the CPU is almost idle.

 

[Terry_Kennedy]

but its hard to understand why insufficient space for socket buffer, as I have only few applications running, and the memory usage is not even 10% and CPU is almost idle.

I think you missed the point of the message I quoted - the problem may not be on your system. If the remote system isn't receiving as fast as your system is sending, you're going to end up waiting. Similarly, if your system is waiting for some data from a remote system, there's nothing you can do on your end (except possibly configuring a shorter timeout before your application gives up).

 

[psyc]

Ah! Ok, yes, that must be the problem though, as I cannot reproduce the same issue on my local system. I even have FreeBSD in virtualbox, still didn't see that sbwait in top -H.