Apache config for Worker MPM vs Prefork MPM

Some installations of Apache will have been compiled to work with either Prefork or Worker MPM, but most by default are compiled only to work with Prefork. Changing to Worker MPM may allow your server to handle much higher traffic, more user sessions, with less RAM use. Your site or server will not scale well if it is running Prefork MPM. Worker is a bit more difficult to set up properly, and has some restrictions regarding functionality of PHP scripts.

To determine which MPM is currently running 

/usr/sbin/apachectl -l

If you see worker.c in the list of loaded modules, your Apache is running Worker MPM.

 If prefork.c, it is running prefork.

High system calls in Apache?

In apache web server the system calls are very high  (find using strace tool)

Sol: In httpd.conf disable the Extended Status
ExtendedStatus off

"Reached max clients" error in web server?

Verify the error in err_log file, it shows the "Reached max clients" error.
 If you found this error verify how many http process are running
                                # ps -ef|grep -c httpd|wc -l
                                #17
Open the httpd.conf file and verify the the maxClients is 16.

So web server reached max clients , for that it thrown the "Reached max clients" error in web server?

Solution:
Step 1:Find out the how much memory on avg each http process is using

                       #top -b -c -n 1|grep httpd|awk '{print $6}'
                       #18m
                         18m
                         18m
                         20m
                         18m
                          15m
                 (or) use below script 

 ps -ylC httpd | awk '{x += $8;y += 1} END {print "Apache Memory Usage (MB): "x/1024; print "Average Proccess Size (MB): "x/((y-1)*1024)}'

Avg: 18m

Step 2 : Stop Apache , determine the amount of Free memory available (without running apache)

                     #free -m
                     #total        used    free
                       432        26        405
Step 3: It is good to have at least 20mb of free memory , if apache was running at it max.

                        = free memory - 20mb
                        =385
Step 4:
 
                            free Memory after stopping Apache - 20 mb
     maxClient =  __________________________________
                                   avg http consuming memory

                maxClients = 385/18
                                    =21
                                    =round (21)
                                    =20

So maxClients in httpd.conf is 20

Why my web server memory utilization is very high?

Sol 1:
Use sttaic IP's instead of wildcard IP's(*).

virtualhostIP.conf (in this file don't use wildcard character)

uses less memory 
<virtualHost 10.62.55.11:80>
<virtualHost 10.62.55.11:443>

Uses More memory

<virtualHost *:80>
<virtualHost *:443>


listenIPs.conf (in this file don't use wildcard character)


uses less memory 
<listen 10.62.55.11:80>
<listen 10.62.55.11:443>

Uses More memory

<listen *:80>
<listen *:443>

Sol 2:

Comment unnecessary modules in you httpd.conf file

 if not using perl comment it
if not using CGI comment it

Application is throwing HTTP Errors/connection timed out errors?

Below are the conditions for throwing Connection timed timed out or HTTP errors.

Sol 1: This problem can be raised because of there is no connection timed out specified in LOAD BALANCER or Application server.

Do the below configurations in you configurations:

--> In Jboss  "server.xml" file specify timed out entry
                        <connectionTimedout="600000" ....> [10 min]


---> In worker.property file specify the connection poll timed out entry
                              worker.node1.connection_pool_timedout=600  [10 min]


Still you are facing the connection timed out error do the below configuration

Sol 2:
 In Jboss application server not mentioned the minSpare or maxSpare connection / minSpare or maxSpare values are not accurate but in web server these values are described ,see below

For Jboss , in server.xml file <connection Timedout="600000" maxThreads="500">
For Web server, in httpd.conf file minSpare 50
                                                   maxSpare 250

due to above configuration, connections are established at web server but the application server is not accepting the connection sent by the web server ,  so you are able to see "HTTP errors"


Do the below configurations in you configurations:

For Jboss , server.xml file
<connection Timedout="600000" maxThreads="500" minSpare="50" maxSpare="250"> 

Here for minSpare and maxSpare , web server contained 50 for minSpare and 250 for maxSpare, same values we need to define in server.xml file or bit high is recommended  But after changing these values run the script and see the results and connection timed out errors, if still facing increase or decrease by 5 (50-5,250-5) and perform the test , check the results. Do the same (increasing / decreasing ) till you get the optimal value.

Sol 3:If still facing the connection timed out error , specify the acceptCount in server.xml file

<connection Timedout="600000" maxThreads="500" minSpare="50" maxSpare="250" acceptCount="100"> 

Accept Count: Number of requests in queue

                

Why Application server (Jboss) utilization is very High?

Verify the number of "Max Threads" in server.xml file(deploy/Jboss-web-deployer/server.xml). If this value is not accurate then , it creates more threads, then CPU utilization increases.

Rule of thumb for "Max Threads" = 250*no. of processors

= 250 *2 (2 core system)

= 500

Now change the value in your server.xml , shown below

Too many files opened issue

How to solve too many files issue?

This issue occur in two areas
1. If Web server, App server and Data base servers residing in One machine, then you m ayget this error
2. If socket timed out not mentioned then this issue may this issue.

Using ulimit -a command, we will find how many sockets had that machine. In any machine 1024 is the maximum sockets.

Bottleneck identification:

Using netstat -a|grep WAIT|wc -l command we will find how many sockets opened by the application. If 1024 connection are utilized then we will get too many open files.

If you want know which server opened more socket, just use lsof -p <Process ID> for finding the sockets opened by that process.

Internal Process  of a socket: All HTTP requests internally used TCP connection. One TCP connection is one Socket in computer science language. So if you send n number of requests to the server , the server opens the socket to handle your requests but we need to close the sockets once your request got over other wise socket won't close, for this we need to mention the socket timed out.

Solutions:

For issue 1: Just increase the sockets using ulimit -Hn <Parameter> (Ex ulimit -Hn 2048)command, but it is not recommended , you have to discuss with your architect on this.

For issue 2: Just provide the socket time out in your server configuration file. In web server change the value in  uri_worker.property file