The Apache Haus Forum

Advanced search  

News:

Welcome to Apache Haus Distribution Forum

Pages: [1]   Go Down

Author Topic: Busy Servers reaches Maxclients.  (Read 7051 times)

samv18

  • Newbie
  • *
  • Offline Offline
  • Posts: 7
Busy Servers reaches Maxclients.
« on: June 19, 2015, 03:43:09 PM »

PostPosted: Thu 18 Jun '15 19:55    Post subject: Busy Servers reaches Maxclients.   Reply with quote
Hello all,

I am new member of apachelounge and hope that some one can help me out with the issues I am facing since a month with apache.

I am using apache 2.2 webserver and tomcat 6 as app server.
I have two unix boxes (let say A and B) where apache is installed for load balancing purpose.

The issue is now and then I see that on both the server reaches to 250 busy servers which makes my site very slow and after some time the site is unaccessible.

When I see this I restart apache on both unix boxes and also restart my app server.

But that does not help. As soon as I start apache the httpd process ramps up to 12 (ps -ef | grep httpd) within a minute and the busy servers still remains at 250.

The only I have to do is wait and watch till the busy servers goes down to 250 and then site is back to normal.

Some times it takes hours for busy servers to go down below 250.

I dont understand that why even restarting apache and tomcat doesnt help. why the busy servers are still at 250. even after I restart.

This is what I have in httpd.conf

<IfModule worker.c>
ServerLimit 80
StartServers 2
MaxClients 250
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>

I tried to increase MaxClients to 300, which also didnt help and apache or httpd process shuts down by itself.

Any help is much appreciated.

Thanks
Logged

mario

  • Administrator
  • Member Elite
  • *****
  • Offline Offline
  • Posts: 574
Re: Busy Servers reaches Maxclients.
« Reply #1 on: June 19, 2015, 05:01:24 PM »

Are you sure that worker mpm is used?

You cab check that with

sudo /etc/init.d/apachectl -V

or

sudo /etc/init.d/apache2ctl -V

depending on your distro
Logged

samv18

  • Newbie
  • *
  • Offline Offline
  • Posts: 7
Re: Busy Servers reaches Maxclients.
« Reply #2 on: June 19, 2015, 06:31:24 PM »

I tried both command and it said that "Command not found"

I have below configuration in httpd.conf so I am assuming that MPM is getting used.


<IfModule !mpm_winnt.c>
<IfModule !mpm_netware.c>
        #LockFile logs/accept.lock
</IfModule>
</IfModule>


<IfModule !mpm_netware.c>
<IfModule !perchild.c>
        #ScoreBoardFile logs/apache_runtime_status
</IfModule>
</IfModule>

<IfModule !mpm_winnt.c>
<IfModule !mpm_netware.c>
        #
        # If you wish httpd to run as a different user or group, you must run
        # httpd as root initially and it will switch.
        #
        # User/Group: The name (or #number) of the user/group to run httpd as.
        #  . On SCO (ODT 3) use "User nouser" and "Group nogroup".
        #  . On HPUX you may not be able to use shared memory as nobody, and the
        #    suggested workaround is to create a user www and use that user.
        #  NOTE that some kernels refuse to setgid(Group) or semctl(IPC_SET)
        #  when the value of (unsigned)Group is above 60000;
        #  don't use Group #-1 on these systems!
        #
        User ecprd
        Group ecgrp
        #<IfModule mod_suexec.c>
                #SuexecUserGroup TheUser TheUsersGroup
        #</IfModule>
</IfModule>
</IfModule>
Logged

mario

  • Administrator
  • Member Elite
  • *****
  • Offline Offline
  • Posts: 574
Re: Busy Servers reaches Maxclients.
« Reply #3 on: June 21, 2015, 07:12:58 PM »

How do you start / restart apache? Which distro do you use?
Logged

samv18

  • Newbie
  • *
  • Offline Offline
  • Posts: 7
Re: Busy Servers reaches Maxclients.
« Reply #4 on: June 22, 2015, 04:00:31 AM »

sudo /opt/hpws/apache/bin/httpd -f /prdbvn_ws/apache/conf/httpd.conf -D SSL -k (start|stop)

I use HP distro.

Below is the server status at non-peak time where not many users using the website.

Server Version: Apache
Server Built: Mar 1 2005 05:13:27
Current Time: Sunday, 21-Jun-2015 21:59:19 EDT
Restart Time: Sunday, 21-Jun-2015 08:45:08 EDT
Parent Server Generation: 0
Server uptime: 13 hours 14 minutes 10 seconds
Total accesses: 127898 - Total Traffic: 1.5 GB
CPU Usage: u213.44 s70.37 cu0 cs0 - .596% CPU load
2.68 requests/sec - 33.6 kB/second - 12.5 kB/request
242 requests currently being processed, 8 idle workers
RRRRRRRRRRRRRRRRRRRRRRRRW.......................................
RRRRRRRRRRRRRRRRRRRRRRRRR.......................................
RRR_RRRRRRRRRRR_RRRRRRRRR.......................................
RRRRRRRRRRRRRRRRRRRRRRRRR.......................................
RRRRRRRRRRRRRRRRRRRRRRRRR.......................................
RRRRRRR_RRRR_RRRRRRRRRRRR.......................................
RRRRRRRRRRRRRRRRRRRRRRRRR.......................................
_R_RRRRRRRRRRRRRRRRRRRRRR.......................................
RRRRRRRRRRRR_RRRRRR_RRRRR.......................................
RRRRRRRRRRRRRRRRRRRRRRRRR.......................................
Logged

mario

  • Administrator
  • Member Elite
  • *****
  • Offline Offline
  • Posts: 574
Re: Busy Servers reaches Maxclients.
« Reply #5 on: June 22, 2015, 10:10:03 AM »

Ok then you can do

Code: [Select]
sudo /opt/hpws/apache/bin/httpd -f /prdbvn_ws/apache/conf/httpd.conf -D SSL -V
The server maybe an older version

Quote
Server Built: Mar 1 2005 05:13:27
Logged

samv18

  • Newbie
  • *
  • Offline Offline
  • Posts: 7
Re: Busy Servers reaches Maxclients.
« Reply #6 on: June 22, 2015, 03:28:20 PM »

Thank you!
When I executed that commad, here what it says.
Server version: Apache/2.0.53  HP-UX_Apache-based_Web_Server
Server built:   Mar  1 2005 05:13:27
Server's Module Magic Number: 20020903:9
Architecture:   32-bit
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/worker"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PROC_PTHREAD_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D APR_PROCESS_LOCK_IS_GLOBAL
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D HTTPD_ROOT="/opt/hpws/apache"
 -D DEFAULT_CHROOT_DIR=""
 -D SUEXEC_BIN="/opt/hpws/apache/bin/suexec"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"


Logged

mario

  • Administrator
  • Member Elite
  • *****
  • Offline Offline
  • Posts: 574
Re: Busy Servers reaches Maxclients.
« Reply #7 on: June 22, 2015, 04:00:36 PM »

That is apache 2.0.53 a bit very old  ;)

I don't know what HP changed to the original apache web server.

However I would try

Code: [Select]
    StartServers             3
    MaxClients             320
    MinSpareThreads         20
    MaxSpareThreads         40
    ThreadsPerChild         20
    MaxConnectionsPerChild   0
Logged

samv18

  • Newbie
  • *
  • Offline Offline
  • Posts: 7
Re: Busy Servers reaches Maxclients.
« Reply #8 on: June 22, 2015, 04:07:25 PM »

Thanks for the suggestion.

I tried to increase MaxClients to 300, which also didnt help and apache or httpd process shuts down by itself( may be because OS configuration ie. RAM, memory etc.).

One thing which I dont understand is why  apache restart is not helping. Even after I start the busy server stays at 250.
Initially I thought its because of the App server/ Coding in the application. But App server(Tomcat) restart also didnt help.

Logged

mario

  • Administrator
  • Member Elite
  • *****
  • Offline Offline
  • Posts: 574
Re: Busy Servers reaches Maxclients.
« Reply #9 on: June 22, 2015, 04:52:57 PM »

Do you use -k restart or -k stop and -k start?
Logged

samv18

  • Newbie
  • *
  • Offline Offline
  • Posts: 7
Re: Busy Servers reaches Maxclients.
« Reply #10 on: June 22, 2015, 04:56:37 PM »

Yes.
Do you see anything wrong with this below stat(CPU usage) from server-status.

CPU Usage: u213.44 s70.37 cu0 cs0 - .596% CPU load
Logged

mario

  • Administrator
  • Member Elite
  • *****
  • Offline Offline
  • Posts: 574
Re: Busy Servers reaches Maxclients.
« Reply #11 on: June 24, 2015, 12:16:52 PM »

I've no clue about that numbers. If I have to work with linux I use htop to see the memory and cpu usage.
Logged

samv18

  • Newbie
  • *
  • Offline Offline
  • Posts: 7
Re: Busy Servers reaches Maxclients.
« Reply #12 on: June 26, 2015, 04:33:30 PM »

Ok thanks. when I do TOP things looks normal to me.

I havent made any changes to httpd.conf but today apache died by itself and I see following in the logs.
Jun 26 09:30:03 2015] [crit] (22)Invalid argument: ap_queue_pop failed
[Fri Jun 26 09:30:03 2015] [crit] (22)Invalid argument: ap_queue_pop failed
[Fri Jun 26 09:30:03 2015] [crit] (22)Invalid argument: ap_queue_pop failed
[Fri Jun 26 09:30:03 2015] [crit] (22)Invalid argument: ap_queue_pop failed
[Fri Jun 26 09:30:03 2015] [emerg] (28)No space left on device: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.
[Fri Jun 26 09:30:04 2015] [emerg] (28)No space left on device: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.
[Fri Jun 26 09:30:06 2015] [emerg] (28)No space left on device: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.
[Fri Jun 26 09:31:28 2015] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Fri Jun 26 09:31:39 2015] [alert] Child 17330 returned a Fatal error...\nApache is exiting!
[Fri Jun 26 09:31:39 2015] [warn] child process 17173 still did not exit, sending a SIGTERM
[Fri Jun 26 09:31:39 2015] [warn] child process 17054 still did not exit, sending a SIGTERM
[Fri Jun 26 09:31:39 2015] [warn] child process 17310 still did not exit, sending a SIGTERM
[Fri Jun 26 09:31:39 2015] [warn] child process 17328 still did not exit, sending a SIGTERM
[Fri Jun 26 10:04:46 2015] [warn] pid file //prdbvn_ws/apache/logs/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Fri Jun 26 10:04:47 2015] [notice] Apache configured -- resuming normal operations



I have to start it again.
Logged

Gregg

  • Administrator
  • Member Elite
  • *****
  • Offline Offline
  • Posts: 828
Re: Busy Servers reaches Maxclients.
« Reply #13 on: June 26, 2015, 07:58:30 PM »

What 3rd party modules are you using?

The reason I ask is that I scoured the 2.0 source
/** no space left on device */
#define APR_STATUS_IS_ENOSPC(s)         ((s) == APR_ENOSPC)

The only thing in 2.0 that uses that code is protocol.c and mod_dav_fs, neither of which return the actual error message "No space left on device:"

So this has to be some other module barfing, obviously unable to write a filename.lock file because there's no space left to write to. This could be memory or file storage and seeing this is HP-UX, permissons?
Logged

mario

  • Administrator
  • Member Elite
  • *****
  • Offline Offline
  • Posts: 574
Re: Busy Servers reaches Maxclients.
« Reply #14 on: July 02, 2015, 10:48:30 AM »

what does disk free show?

Code: [Select]
df -h
Logged
Pages: [1]   Go Up
 

Sitemap 1 2 3 4 5 6 7 8 9 10 11 12 13