Pages

[AJPRO2021]

Since we've upgraded Tomcat Web Services from 2.4.46 to 2.4.47/48 is when we've started to have Gateway Timeout problem.

Here are the version of the Tomcat running on the windows 2016 64bit server.

Apache Tomcat/9.0.46 --> (apache-tomcat-9.0.46-windows-x64)
Apache/2.4.48 (Win32) OpenSSL/1.1.1k --> (httpd-2.4.48-o111k-x86-vc15)

for the longest, we have had enabled the SSL in httpd.conf and configured the httpd-ssl.conf environment to run with our own internal certificate.

==================================
httpd.conf
==================================

Code Select Expand

# Secure (SSL/TLS) connections
# Note: The following must must be present to support
#       starting without SSL on platforms with no /dev/random equivalent
#       but a statically compiled-in mod_ssl.
#
<IfModule ssl_module>
Include conf/extra/httpd-ssl.conf
#Include conf/extra/httpd-ahssl.conf
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin
</IfModule>
<IfModule http2_module>
    ProtocolsHonorOrder On
    Protocols h2 h2c http/1.1
</IfModule>

=====================================
httpd-ssl.conf
=====================================

Listen 443

SSLPassPhraseDialog  builtin

SSLSessionCache        "shmcb:${SRVROOT}/logs/ssl_scache(512000)"
SSLStaplingCache       "shmcb:${SRVROOT}/logs/ssl_stapling_data(512000)"
SSLSessionCacheTimeout  600

<VirtualHost _default_:443>
DocumentRoot "${SRVROOT}/htdocs"
ServerName ecsma.epssdri.com:443
ServerAdmin
Redirect permanent /index.html /ostcs/csas.exe

ErrorLog "${SRVROOT}/logs/error.log"
TransferLog "${SRVROOT}/logs/access.log"

SSLEngine on
SSLProtocol TLSv1.2
SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384
SSLHonorCipherOrder on
SSLInsecureRenegotiation on

SSLCompression off

SSLCertificateFile "${SRVROOT}/conf/ssl/escm.cer"
SSLCertificateKeyFile "${SRVROOT}/conf/ssl/escm.cer"


<FilesMatch "\.(cgi|shtml|phtml|php)$">
    SSLOptions +StdEnvVars
</FilesMatch>

<Directory "${SRVROOT}/cgi-bin">
    SSLOptions +StdEnvVars
</Directory>

BrowserMatch ".*MSIE.*" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0

CustomLog "${SRVROOT}/logs/ssl_request.log" \
          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

</VirtualHost>                                 

=====================================

All major Apache Tomcat/9.0.46 and Apache Web Service's timeouts have been raised beyond the limit.

Our Application stops communicating at exactly 01:06:21 (min) and due to lack of responds Tomcat closes the session at 03:02:01 (min) into the process.

We have ran Wireshark capture on both client and server site and we have found no abnormality with communication between the client/host.

we had no choice but to revert back to Apache 2.4.46 to stabilize our environment.

I believe the major change is with OpenSSL 1.1.1k vs 1.1.1g which was packaged with 2.4.46 release; Not sure if this what is causing the failure!

Any feed back from anyone will be greatly appreciated.

Thanks.

[mario]

How do you connect to Tomcat? mod_jk or mod_proxy_ajp or mod_proxy_http ?

[AJPRO2021]

These are the only modules that has been enabled.

Code Select Expand


LoadModule access_compat_module modules/mod_access_compat.so
LoadModule actions_module modules/mod_actions.so
LoadModule alias_module modules/mod_alias.so
LoadModule allowmethods_module modules/mod_allowmethods.so
LoadModule asis_module modules/mod_asis.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule authn_core_module modules/mod_authn_core.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule dir_module modules/mod_dir.so
LoadModule env_module modules/mod_env.so
LoadModule include_module modules/mod_include.so
LoadModule info_module modules/mod_info.so
LoadModule isapi_module modules/mod_isapi.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule mime_module modules/mod_mime.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule socache_shmcb_module modules/mod_socache_shmcb.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule status_module modules/mod_status.so


[mario]

I don't see there anything how you connect tot the tomcat server nor any proxy settings.

[AJPRO2021]

Thanks for your reply...

We have loaded the WAR files and the only configuration change we made to the Apache tomcat server.xml is as follow:
The only part added to the httpd.conf file is the application connectivity entry:

The only change made to the server.xml

Code Select Expand


    <!-- connectionTimeout & connectionUploadTimeout is set to 15 minutes-->
    <Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="-1"
               connectionUploadTimeout="900000"
               disableUploadTimeout="false"
               redirectPort="8443" />







Here is the content of httpd.conf with all remarks removed

Code Select Expand



Define SRVROOT "/Apache/Apache24"
ServerRoot "${SRVROOT}"

TraceEnable extended
TimeOut 600
KeepAliveTimeout 45
Define ENABLE_TLS13 "No"
Listen 80

LoadModule access_compat_module modules/mod_access_compat.so
LoadModule actions_module modules/mod_actions.so
LoadModule alias_module modules/mod_alias.so
LoadModule allowmethods_module modules/mod_allowmethods.so
LoadModule asis_module modules/mod_asis.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule authn_core_module modules/mod_authn_core.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule dir_module modules/mod_dir.so
LoadModule env_module modules/mod_env.so
LoadModule include_module modules/mod_include.so
LoadModule info_module modules/mod_info.so
LoadModule isapi_module modules/mod_isapi.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule mime_module modules/mod_mime.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule socache_shmcb_module modules/mod_socache_shmcb.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule status_module modules/mod_status.so


<IfModule unixd_module>
User daemon
Group daemon
</IfModule>


ServerAdmin
ServerName xxx.xxx.xxxx.xx:80

<Directory />
    AllowOverride none
    Require all denied
</Directory>

Alias /img "D:/OPENTEXT/support"
<Directory "D:/OPENTEXT/support">
AllowOverride None
Order allow,deny
Allow from all
Require all granted
</Directory>

ScriptAlias /otcs "D:/OPENTEXT/cgi"
<Directory "D:/OPENTEXT/cgi">
AllowOverride None
Options ExecCGI
Order allow,deny
Allow from all
Require all granted
</Directory>

<Directory "D:/OPENTEXT/appData/supportasset/">
Options Indexes FollowSymlinks MultiViews
AllowOverride None
Require all granted
</Directory>


DocumentRoot "${SRVROOT}/htdocs"
<Directory "${SRVROOT}/htdocs">
    Options Indexes FollowSymLinks

    AllowOverride None

    Require all granted
</Directory>

<IfModule dir_module>
    DirectoryIndex index.html
</IfModule>

<Files ".ht*">
    Require all denied
</Files>

ErrorLog "logs/error.log"

LogLevel warn

<IfModule log_config_module>
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common

    <IfModule logio_module>
      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>

    CustomLog "logs/access.log" common

</IfModule>

<IfModule alias_module>
    Alias /appimg "D:/OPENTEXT/appData/supportasset"
    ScriptAlias /cgi-bin/ "${SRVROOT}/cgi-bin/"
</IfModule>

<IfModule cgid_module>
</IfModule>

<Directory "${SRVROOT}/cgi-bin">
    AllowOverride None
    Options None
    Require all granted
</Directory>

<IfModule mime_module>
    TypesConfig conf/mime.types
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz
    AddHandler cgi-script .cgi .pl
    AddHandler type-map var
</IfModule>


Include conf/extra/httpd-autoindex.conf
Include conf/extra/httpd-info.conf
Include conf/extra/httpd-vhosts.conf
Include conf/extra/httpd-default.conf

<IfModule proxy_html_module>
Include conf/extra/httpd-proxy-html.conf
</IfModule>

<IfModule ssl_module>
Include conf/extra/httpd-ssl.conf
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin
</IfModule>
<IfModule http2_module>
    ProtocolsHonorOrder On
    Protocols h2 h2c http/1.1
</IfModule>






Here is the content of httpd-ssl.conf

Code Select Expand


#SSLRandomSeed startup file:/dev/random  512
#SSLRandomSeed startup file:/dev/urandom 512
#SSLRandomSeed connect file:/dev/random  512
#SSLRandomSeed connect file:/dev/urandom 512

Listen 443

SSLPassPhraseDialog  builtin

#SSLSessionCache       "dbm:${SRVROOT}/logs/ssl_scache"
SSLSessionCache        "shmcb:${SRVROOT}/logs/ssl_scache(512000)"
SSLStaplingCache       "shmcb:${SRVROOT}/logs/ssl_stapling_data(512000)"
SSLSessionCacheTimeout  600

<VirtualHost _default_:443>
DocumentRoot "${SRVROOT}/htdocs"
ServerName ecmaa.epri.com:443
ServerAdmin
Redirect permanent /index.html /otcs/cs.exe

ErrorLog "${SRVROOT}/logs/error.log"
TransferLog "${SRVROOT}/logs/access.log"

SSLEngine on
SSLProtocol TLSv1.2
#SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL
#SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384
SSLHonorCipherOrder on
SSLInsecureRenegotiation on


SSLCompression off

# Note: ECM PKCS12 file has multiple items in it (e.g. a certificate and private key),
# the PEM file that has been created contains all of the items in it.
SSLCertificateFile "${SRVROOT}/conf/ssl/ecma.cer"
SSLCertificateKeyFile "${SRVROOT}/conf/ssl/ecma.cer"

#SSLVerifyClient require
#SSLVerifyDepth  10


<FilesMatch "\.(cgi|shtml|phtml|php)$">
    SSLOptions +StdEnvVars
</FilesMatch>

<Directory "${SRVROOT}/cgi-bin">
    SSLOptions +StdEnvVars
</Directory>


BrowserMatch ".*MSIE.*" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0

CustomLog "${SRVROOT}/logs/ssl_request.log" \
          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

</VirtualHost>                                 






Finally here is the content of the httpd-vhosts.conf file

Code Select Expand



<VirtualHost _default_:80>
DocumentRoot "${SRVROOT}/htdocs"
ServerAdmin
ServerName ecmaa.epri.com:80

Timeout 600


Redirect permanent /index.html https://ecmaa.epri.com/otcs/cs.exe
Redirect permanent / https://ecmaa.epri.com/

ErrorLog "logs/ecmaa.epri.com-error.log"
CustomLog "logs/ecmaa.epri.com-access.log" common
</VirtualHost>


Please let me know if you need any additional information.

Thank you.

[mario]

The two important parts (at least I think)

are

port 80 vhost
Redirect permanent /index.html https://ecmaa.epri.com/otcs/cs.exe
Redirect permanent / https://ecmaa.epri.com/

and

port 443 vhost
Redirect permanent /index.html /otcs/cs.exe

So in the very end of very request to httpd Apache web server is to call /otcs/cs.exe. I've no clue what that is for. It is not a part of the orig. apache.

How your httpd apache is connected to Tomcat apache I wonder about that, if it is at all.

[AJPRO2021]

Thank you and I do appreciate your feedback.

Somehow, under current setting (Configuration), Tomcat is serving the sites and Apache is doing the redirect!

To your point, I have started looking into configuring Tomcat to work with Apache using the mod_jk module.
Available documents are self-explanatory and I have been able to make the appropriate changes.
Although tomcat and Apache starts up with no issues, but then the Apache Web Server is complaining about "jk_map_to_storage::mod_jk.c (3816): no match for /otcs/cs.exe found"

Here are the version of the Tomcat running on the windows 2016 64bit server.

Apache Tomcat/9.0.46 --> (apache-tomcat-9.0.46-windows-x64)
Apache/2.4.48 (Win64) OpenSSL/1.1.1k mod_jk/1.2.40 --> (httpd-2.4.48-o111k-x64-vc15)
Apache-tomcat connector module " tomcat-connectors-1.2.40-windows-x86_64-httpd-2.4.x"

As you have already noticed our application runs its processes through "cs.exe" That is mapped to httpd.conf; Physical location "D:\OPENTEXT\cgi\"

# Content Server support directory mapping
Alias /img "D:/OPENTEXT/support"
<Directory "D:/OPENTEXT/support">
AllowOverride all
Order allow,deny
Allow from all
Deny from none
Require all granted
</Directory>

# Content Server CGI directory mapping
ScriptAlias /otcs "D:/OPENTEXT/cgi"
<Directory "D:/OPENTEXT/cgi">
AllowOverride all
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
#Options Indexes MultiViews ExecCGI
Order allow,deny
Allow from all
Deny from none
Require all granted
</Directory>

<Directory "D:/OPENTEXT/appData/supportasset/">
Options Indexes FollowSymlinks MultiViews
AllowOverride all
Allow from all
Deny from none
Require all granted
</Directory>

At the endo the httpd.conf I have included the mod_jk.conf directive.

   include "D:/Apache/Apache24/conf/mod_jk.conf"

Here is configuration of the mod_jk.conf

# Load mod_jk module
# Update this path to match your modules location
LoadModule jk_module D:/Apache/Apache24/modules/mod_jk.so

# Where to find workers.properties
# Update this path to match your conf directory location
JkWorkersFile D:/Apache/Apache24/conf/workers.properties

# Where to put jk logs
# Update this path to match your logs directory location
JkLogFile D:/Apache/Apache24/logs/mod_jk.log

# Set the jk log level [debug/error/info]
JkLogLevel debug

# Select the log format
JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"

# JkOptions indicate to send SSL KEY SIZE,
JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
#JkOptions +RejectUnsafeURI +ForwardKeySize +ForwardURICompat -ForwardDirectories


# JkRequestLogFormat set the request format
JkRequestLogFormat "%w %V %T"

JkExtractSSL On
#JkMountCopy All
#JkMount /* ajp13


# Send everything for context /otcs to worker ajp13
JkUnMount /otcs/*.exe ajp13

# Send everything for context /cws to worker ajp13
JkMount /cws ajp13
JkMount /cws/* ajp13

# Send everything for context /img to worker ajp13
JkMount /img ajp13
JkMount /img/* ajp13

JKMount /pulse ajp13
JKMount /pulse/* ajp13

JKMount /cgi-bin ajp13
JKMount /cgi-bin/* ajp13

JKMount /appimg ajp13
JKMount /appimg/* ajp13

  I have also included "JkMountCopy On" line in httpd-ssl.conf
   
<VirtualHost _default_:443>
JkMountCopy On

The problem I am running into:
As long as I exclude "JkUnMount /otcs/*.exe ajp13" the site works but then I get "jk_map_to_storage::mod_jk.c (3816): no match for /otcs/cs.exe found"
If I include the "JkMount /otcs/*.exe ajp13" the site fails with "HTTPS Status 404 – not found
   The requested resource [/otcs/cs.exe] is not available
   The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.

Let me know what you think of the problem or if it is possible please provide a link to a document that I can follow.

-AJ

[mario]

I never used mod jk, but mod_proxy_ajp

Code Select Expand


<VirtualHost *:80>
    ServerName ecmaa.epri.com
    DocumentRoot "/mario/Apache24/htdocs"
    <Directory "/mario/Apache24/htdocs">
        Options Indexes Includes FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>
    <Location />
        ProxyPass /ostcs/csas.exe !
        ProxyPass ajp://localhost:8009/
        ProxyPassReverse ajp://localhost:8009/
    </Location>
</virtualhost>


or is ajp is not working

Code Select Expand


<VirtualHost *:80>
    ...
    <Location />
        ProxyPass /ostcs/csas.exe !
        ProxyPass http://localhost:8080/
        ProxyPassReverse http://localhost:8080/
    </Location>
...


use the same location config also for the ssl vhost.

[AJPRO2021]

It seems easy enough to follow your steps to convert over to mod_proxy_ajp.

Trying to keep thing simple I have just added  the following line to the httpd.conf file to point to the below configuration. include "D:/Apache/Apache24/conf/ajp.conf"

No more redirect with "port 80 vhost" or "port 443 vhost"

[ajp.conf]
=================================================================
LoadModule proxy_ajp_module D:/Apache/Apache24/modules/mod_proxy_ajp.so
LoadModule proxy_module D:/Apache/Apache24/modules/mod_proxy.so

ProxyRequests Off
<Proxy *>
        Options Indexes Includes FollowSymLinks
        AllowOverride All
        Require all granted
</Proxy>

<Proxy "https://ecma.epri.com">
  ProxySet connectiontimeout=5 timeout=300
</Proxy>

ProxyPass /otcs/cs.exe !
ProxyPass       / ajp://ecma.epri.com:8009/  secret=namnetiq
ProxyPassReverse    / ajp://ecma.epri.com:8009/  secret=namnetiq
=================================================================


[httpd-vhosts.conf]
=================================================================
<VirtualHost _default_:80>
DocumentRoot "${SRVROOT}/htdocs"
#ServerName www.example.com:80
ServerName ecma.epri.com:80
ErrorLog "logs/ecma.epri.com-error.log"
CustomLog "logs/ecma.epri.com-access.log" common
</VirtualHost>
=================================================================

[httpd-ssl.conf]
=================================================================
Listen 443

SSLPassPhraseDialog  builtin

#SSLSessionCache       "dbm:${SRVROOT}/logs/ssl_scache"
SSLSessionCache        "shmcb:${SRVROOT}/logs/ssl_scache(512000)"
SSLSessionCacheTimeout  300

<VirtualHost _default_:443>
DocumentRoot "${SRVROOT}/htdocs"
ServerName ecma.epri.com:443
ServerAdmin ecma.epri.com

ErrorLog "${SRVROOT}/logs/error.log"
TransferLog "${SRVROOT}/logs/access.log"
SSLCertificateFile "${SRVROOT}/conf/ssl/ecm.cer"
SSLCertificateKeyFile "${SRVROOT}/conf/ssl/ecm.cer"

#SSLVerifyClient require
#SSLVerifyDepth  10

<FilesMatch "\.(cgi|shtml|phtml|php)$">
    SSLOptions +StdEnvVars
</FilesMatch>

<Directory "${SRVROOT}/cgi-bin">
    SSLOptions +StdEnvVars
</Directory>

BrowserMatch ".*MSIE.*" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0

CustomLog "${SRVROOT}/logs/ssl_request.log" \
          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

</VirtualHost>
=================================================================

However,  unfortunately I am back to the main issue I had initially reported "Gateway Timeout".
Since we've upgraded Tomcat Web Services from 2.4.46 to 2.4.47/48 is when we've started to have Gateway Timeout problem.
I have had no lock with neither of the two setups: mod_jk or mod_proxy_ajp.
In both setup cases, we seems to be bypassing the main application processor (/otcs/cs.exe) for the site to become operational.

I am not sure what we have so far accomplished and what more needs to be done to overcome the Gateway Timeout problem!

I greatly appreciate your support in this matter as I am not so certain what has changed in the latest release that has caused this problem.

Thanks again.

-AJ