Print

[samv18]

PostPosted: Thu 18 Jun '15 19:55    Post subject: Busy Servers reaches Maxclients.   Reply with quote
Hello all,

I am new member of apachelounge and hope that some one can help me out with the issues I am facing since a month with apache.

I am using apache 2.2 webserver and tomcat 6 as app server.
I have two unix boxes (let say A and B) where apache is installed for load balancing purpose.

The issue is now and then I see that on both the server reaches to 250 busy servers which makes my site very slow and after some time the site is unaccessible.

When I see this I restart apache on both unix boxes and also restart my app server.

But that does not help. As soon as I start apache the httpd process ramps up to 12 (ps -ef | grep httpd) within a minute and the busy servers still remains at 250.

The only I have to do is wait and watch till the busy servers goes down to 250 and then site is back to normal.

Some times it takes hours for busy servers to go down below 250.

I dont understand that why even restarting apache and tomcat doesnt help. why the busy servers are still at 250. even after I restart.

This is what I have in httpd.conf

<IfModule worker.c>
ServerLimit 80
StartServers 2
MaxClients 250
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>

I tried to increase MaxClients to 300, which also didnt help and apache or httpd process shuts down by itself.

Any help is much appreciated.

Thanks

[mario]

Are you sure that worker mpm is used?

You cab check that with

sudo /etc/init.d/apachectl -V

or

sudo /etc/init.d/apache2ctl -V

depending on your distro

[samv18]

I tried both command and it said that "Command not found"

I have below configuration in httpd.conf so I am assuming that MPM is getting used.


<IfModule !mpm_winnt.c>
<IfModule !mpm_netware.c>
        #LockFile logs/accept.lock
</IfModule>
</IfModule>


<IfModule !mpm_netware.c>
<IfModule !perchild.c>
        #ScoreBoardFile logs/apache_runtime_status
</IfModule>
</IfModule>

<IfModule !mpm_winnt.c>
<IfModule !mpm_netware.c>
        #
        # If you wish httpd to run as a different user or group, you must run
        # httpd as root initially and it will switch.
        #
        # User/Group: The name (or #number) of the user/group to run httpd as.
        #  . On SCO (ODT 3) use "User nouser" and "Group nogroup".
        #  . On HPUX you may not be able to use shared memory as nobody, and the
        #    suggested workaround is to create a user www and use that user.
        #  NOTE that some kernels refuse to setgid(Group) or semctl(IPC_SET)
        #  when the value of (unsigned)Group is above 60000;
        #  don't use Group #-1 on these systems!
        #
        User ecprd
        Group ecgrp
        #<IfModule mod_suexec.c>
                #SuexecUserGroup TheUser TheUsersGroup
        #</IfModule>
</IfModule>
</IfModule>

[mario]

How do you start / restart apache? Which distro do you use?

[samv18]

sudo /opt/hpws/apache/bin/httpd -f /prdbvn_ws/apache/conf/httpd.conf -D SSL -k (start|stop)

I use HP distro.

Below is the server status at non-peak time where not many users using the website.

Server Version: Apache
Server Built: Mar 1 2005 05:13:27
Current Time: Sunday, 21-Jun-2015 21:59:19 EDT
Restart Time: Sunday, 21-Jun-2015 08:45:08 EDT
Parent Server Generation: 0
Server uptime: 13 hours 14 minutes 10 seconds
Total accesses: 127898 - Total Traffic: 1.5 GB
CPU Usage: u213.44 s70.37 cu0 cs0 - .596% CPU load
2.68 requests/sec - 33.6 kB/second - 12.5 kB/request
242 requests currently being processed, 8 idle workers
RRRRRRRRRRRRRRRRRRRRRRRRW.......................................
RRRRRRRRRRRRRRRRRRRRRRRRR.......................................
RRR_RRRRRRRRRRR_RRRRRRRRR.......................................
RRRRRRRRRRRRRRRRRRRRRRRRR.......................................
RRRRRRRRRRRRRRRRRRRRRRRRR.......................................
RRRRRRR_RRRR_RRRRRRRRRRRR.......................................
RRRRRRRRRRRRRRRRRRRRRRRRR.......................................
_R_RRRRRRRRRRRRRRRRRRRRRR.......................................
RRRRRRRRRRRR_RRRRRR_RRRRR.......................................
RRRRRRRRRRRRRRRRRRRRRRRRR.......................................

[mario]

Ok then you can do

Code: [Select]

sudo /opt/hpws/apache/bin/httpd -f /prdbvn_ws/apache/conf/httpd.conf -D SSL -V
The server maybe an older version

Server Built: Mar 1 2005 05:13:27

[samv18]

Thank you!
When I executed that commad, here what it says.
Server version: Apache/2.0.53  HP-UX_Apache-based_Web_Server
Server built:   Mar  1 2005 05:13:27
Server's Module Magic Number: 20020903:9
Architecture:   32-bit
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/worker"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PROC_PTHREAD_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D APR_PROCESS_LOCK_IS_GLOBAL
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D HTTPD_ROOT="/opt/hpws/apache"
 -D DEFAULT_CHROOT_DIR=""
 -D SUEXEC_BIN="/opt/hpws/apache/bin/suexec"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

[mario]

That is apache 2.0.53 a bit very old 

I don't know what HP changed to the original apache web server.

However I would try

Code: [Select]

    StartServers             3
    MaxClients             320
    MinSpareThreads         20
    MaxSpareThreads         40
    ThreadsPerChild         20
    MaxConnectionsPerChild   0

[samv18]

Thanks for the suggestion.

I tried to increase MaxClients to 300, which also didnt help and apache or httpd process shuts down by itself( may be because OS configuration ie. RAM, memory etc.).

One thing which I dont understand is why  apache restart is not helping. Even after I start the busy server stays at 250.
Initially I thought its because of the App server/ Coding in the application. But App server(Tomcat) restart also didnt help.

[mario]

Do you use -k restart or -k stop and -k start?

[samv18]

Yes.
Do you see anything wrong with this below stat(CPU usage) from server-status.

CPU Usage: u213.44 s70.37 cu0 cs0 - .596% CPU load

[mario]

I've no clue about that numbers. If I have to work with linux I use htop to see the memory and cpu usage.

[samv18]

Ok thanks. when I do TOP things looks normal to me.

I havent made any changes to httpd.conf but today apache died by itself and I see following in the logs.
Jun 26 09:30:03 2015] [crit] (22)Invalid argument: ap_queue_pop failed
[Fri Jun 26 09:30:03 2015] [crit] (22)Invalid argument: ap_queue_pop failed
[Fri Jun 26 09:30:03 2015] [crit] (22)Invalid argument: ap_queue_pop failed
[Fri Jun 26 09:30:03 2015] [crit] (22)Invalid argument: ap_queue_pop failed
[Fri Jun 26 09:30:03 2015] [emerg] (28)No space left on device: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.
[Fri Jun 26 09:30:04 2015] [emerg] (28)No space left on device: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.
[Fri Jun 26 09:30:06 2015] [emerg] (28)No space left on device: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.
[Fri Jun 26 09:31:28 2015] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Fri Jun 26 09:31:39 2015] [alert] Child 17330 returned a Fatal error...\nApache is exiting!
[Fri Jun 26 09:31:39 2015] [warn] child process 17173 still did not exit, sending a SIGTERM
[Fri Jun 26 09:31:39 2015] [warn] child process 17054 still did not exit, sending a SIGTERM
[Fri Jun 26 09:31:39 2015] [warn] child process 17310 still did not exit, sending a SIGTERM
[Fri Jun 26 09:31:39 2015] [warn] child process 17328 still did not exit, sending a SIGTERM
[Fri Jun 26 10:04:46 2015] [warn] pid file //prdbvn_ws/apache/logs/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Fri Jun 26 10:04:47 2015] [notice] Apache configured -- resuming normal operations



I have to start it again.

[Gregg]

What 3rd party modules are you using?

The reason I ask is that I scoured the 2.0 source
/** no space left on device */
#define APR_STATUS_IS_ENOSPC(s)         ((s) == APR_ENOSPC)

The only thing in 2.0 that uses that code is protocol.c and mod_dav_fs, neither of which return the actual error message "No space left on device:"

So this has to be some other module barfing, obviously unable to write a filename.lock file because there's no space left to write to. This could be memory or file storage and seeing this is HP-UX, permissons?

[mario]

what does disk free show?

Code: [Select]

df -h