How a DNS problem can put your Mysql server down

admin — Sat, 16 Jan 2010 15:10:39 +0000

Last week i was waked up from bed by the monitoring team from my company. There was a problem with my system, there was a DNS problem undergoing but as a side effect my app was down. Since it has a lot of traffic it had to be solved immediately.

I jumped to the computer and I quickly diagnosed the system. Everything was fine except the Mysql connection pool which was exhausted. The first thing that crossed my mind is that it was just a coincidence and I quickly ran show processlist to see a list of MySQL processes. The output was an infinite list of load balancer’s ip address having “login” text as status. In order to achieve high availability i am using Mysql by having a balanced ip address between two Mysql servers. The balancer runs a quick check every 5 seconds by connecting to Mysql and does a simple select on a table.

So for a particular reason the “load balancer” was not able to finish its login attempts and it was overloading my Mysql servers. While I was in the middle of the investigation the problem suddenly stopped. I was happy but somehow scared, i had no idea what the hell happened.

A quick search into Mysql documentation reveals that Mysql is doing a reverse DNS lookup which was the cause of my problems. Since the DNS server had a problem, the operation of reverse DNS was taking far more that 5 seconds to time out. This resulted in overloading the database servers. Check this explanation in the official documentation, How MySQL Uses DNS

After reading tha page I think that mysql needs this reverse DNS lookup only for its permission module and if you don’t use host names with the grant option then you are safe to disable this option. I quote here the parameter which does this:

–skip-name-resolve

Do not resolve host names when checking client connections. Use only IP numbers. If you use this option, all Host column values in the grant tables must be IP numbers or localhost. See Section 7.5.11, “How MySQL Uses DNS”.

I have been able to avoid this? Perhaps, but considering that I used MySQL in production for the first time, it is unlikely to think so.

Long live the reverse DNS, cheers!

Configure Apache and Tomcat severs together

bserban — Sat, 08 Aug 2009 08:12:00 +0000

The most common way to deploy your application in the production environment is to hide the Tomcat behind Apache. This has good and bad parts but it gives you a lot of flexibility and support from Apache. There are a couple of alternatives to put these two severs together:

mod_jk, this is the old connector developed under the Tomcat project and it is using the Tomcat’s AJP protocol. It is expected to be faster than the HTTP protocol which is text based.
mod_proxy, is the support module for HTTP protocol. It is TCP based and uses the HTTP which is plain text. When a web client makes a request to Apache, the Apache will make the same call to the Tomcat and then the Tomcat’s response is passed back to the web client. This connector is part of the Apache for a very long time and it is available also for older versions of Apache. This is the simplest way to put the Apache in front of a Tomcat but also the slowest way to do it.
mod_proxy_ajp, is new and is part of the Apache 2.2. It is working like mod_proxy, but as the name says it is using the AJP connector for sending and getting data from Tomcat. It is using also TCP and it is expected to be faster than plain mod_proxy

Using mod_proxy

Create your own tomcat-httpd.conf file and configure the proxy:

# # Server will not close the connection after each request allowing the browser to use the same connection # KeepAlive On MaxKeepAliveRequests 100 KeepAliveTimeout 5

#
# Load mod_proxy modules
#

LoadModule proxy_module modules/mod_proxy.so

LoadModule proxy_http_module modules/mod_proxy_http.so

ProxyRequests Off
ProxyPreserveHost On
ProxyTimeout 1000
TimeOut 1000

#
# Configure the mod_proxy
#
ProxyPass / http://127.0.0.1:8080/
ProxyPassReverse / http://127.0.0.1:8080/

Include your configuration into Apache httpd.conf using Include directive:

#load Tomcat proxy configuration Include /usr/local/tomcat6/conf/tomcat-httpd.conf

Using mod_proxy_ajp

The only difference between mod_proxy and mod_proxy_ajp is that you have to load mod_proxy_ajp and proxy the request to Tomcat using the ajp protocol.

# # Load mod_proxy modules # LoadModule proxy_module modules/mod_proxy.so

LoadModule proxy_http_module modules/mod_proxy_http.so

LoadModule proxy_ajp_module modules/mod_proxy_ajp.so

ProxyRequests Off
ProxyPreserveHost On
ProxyTimeout 1000
TimeOut 1000
#
# Enable the AJP proxy
#
ProxyPass / ajp://localhost:8009/
ProxyPassReverse / ajp://localhost:8009/

Include your configuration into Apache httpd.conf using Include directive:

#load Tomcat proxy configuration Include /usr/local/tomcat6/conf/tomcat-httpd.conf

Using mod_jk

Edit your tomcat-httpd.conf file and add mod_jk configuration:
# # Load mod_jk is is not loaded already # LoadModule jk_module modules/mod_jk.so # # # JkWorkersFile /etc/httpd/conf/workers.properties JkLogFile /var/logs/httpd/mod_jk.log JkLogLevel info JkLogStampFormat "[%a %b %d %H:%M:%S %Y] " JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories JkRequestLogFormat "%w %V %T" JkMount /test/* worker1

Include your configuration into Apache httpd.conf using Include directive:

#load mod_jk configuration configuration Include /usr/local/tomcat6/conf/tomcat-httpd.conf
Now edit the workers.properties and configure your worker:
worker.list=worker1 worker.worker1.type=ajp13 worker.worker1.host=localhost worker.worker1.port=8009 worker.worker1.connection_pool_size=150 worker.worker1.connection_pool_timeout=600 worker.worker1.socket_keepalive=1

Test each configuration using Apache Benchmark tool

Apache benchmark it is a great tool for testing the above configurations. All you need is to create a typical application page that should be hit with the ab tool. This tool takes a single url and makes requests repeatedly in separates threads. The number of threads is controlled by command line arguments. It also supports keep alive connections.

For more details about Apache Benchmark check this page Apache Benchmark

For testing purposes i have created a test war which has a test.jsp page. Because what we test does not influence the processing time overall we don’t need a complete test which includes a database call or working with certain frameworks. In the end all we need is the output of an application and to test how this output reach the browser using one of the three modes explained in the section above.

The test page will include a previous post of mine which is medium size: http://www.bserban.org/2009/05/put-together-struts2-jpa-hibernate-and-spring/. I have right-clicked in the browser, chosen the view source and copy and paste the content into test.jsp. The file has 112Kb in size.

bserban-mac:~ bserban$ ls -la ~/bin/srv/apache-tomcat-5.5.27/webapps/test/test.jsp
-rw-r–r– 1 bserban staff 113698 Aug 7 09:39 /Users/bserban/bin/srv/apache-tomcat-5.5.27/webapps/test/test.jsp

I am going to hit the Tomcat and the Apache with 10,000 request using 50 threads. The ab command looks like this:

ab -k -n 10000 -c 50 http://localhost/test/test.jsp

To access directly the Tomcat i am going to hit the 8080 port. To preserve the similar enviroment and test conditions I will restart the Apache and Tomcat after each test.

Test results

The table below summarize the results obtained.

	Direct	mode_proxy	mode_proxy_ajp
Throughput	992 reg/s	667 req/s	702 req/s
Average Response Time	50 ms	75 ms	71 ms
90% Response line	75 ms	90 ms	71 ms
100% Response line	207 ms	980 ms	972 ms

As you see, the fastest connecting mode is to connect directly to tomcat using HTTP. Direct HTTP connect will server more request per second that the other modes. The second choice is mode_proxy_ajp followed very closely by the mod_proxy. However the overhead added the the Apache will leverage for real life applications because the application processing time will minimize the impact of using Apache in front of Tomcat. Probably in real world the differences between direct HTTP connect and mod_proxy_ajp will not exceed 5-10% percents in terms of throughput and average time per request. This is the price to pay for the flexibility brought by Apache, because having the Apache in front of the Tomcat will give access to the whole Apache functionality and support.

For those who want to see test results in detail, I included in the post the tests logs.

Test result trace for Tomcat HTTP

bserban-mac:~ bserban$ ab -k -n 10000 -c 50 http://localhost:8080/test/test.jsp This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests

Server Software:        Apache-Coyote/1.1
Server Hostname:        localhost
Server Port:            8080

Document Path: /test/test.jsp
Document Length: 113678 bytes

Concurrency Level:      50
Time taken for tests:   10.079 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      1139157786 bytes
HTML transferred:       1137007356 bytes
Requests per second:    992.18 [#/sec] (mean)
Time per request:       50.394 [ms] (mean)
Time per request:       1.008 [ms] (mean, across all concurrent requests)
Transfer rate:          110376.03 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median   max
Connect:        0    7   7.3      5      67
Processing:     5   43 22.4     37     182
Waiting:        0   16 19.9     11     161
Total:          7   50 22.4     45     207

Percentage of the requests served within a certain time (ms)
50%     45
66%     52
75%     57
80%     61
90%     75
95%     91
98%    119
99%    147
100%    207 (longest request)

Test result trace for mod_proxy_ajp

bserban-mac:~ bserban$ ab -k -n 10000 -c 50 http://localhost/test/test.jsp This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/

Server Software:
Server Hostname: localhost
Server Port: 80

Document Path: /test/test.jsp
Document Length: 113678 bytes

Concurrency Level:      50
Time taken for tests:   14.251 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      1140884936 bytes
HTML transferred:       1139000800 bytes
Requests per second:    701.70 [#/sec] (mean)
Time per request:       71.256 [ms] (mean)
Time per request:       1.425 [ms] (mean, across all concurrent requests)
Transfer rate:          78179.54 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median   max
Connect:        0    7   8.3      5      87
Processing:     6   64 89.8     47     967
Waiting:        0   38 86.9     21     920
Total:          6   71 89.4     52     972

Percentage of the requests served within a certain time (ms)
50%     52
66%     58
75%     64
80%     69
90%     83
95%    115
98%    507
99%    689
100%    972 (longest request)

Test result trace for mod_proxy

Server Software:        Apache-Coyote/1.1
Server Hostname:        localhost
Server Port:            80

Document Path: /test/test.jsp
Document Length: 113678 bytes

Concurrency Level:      50
Time taken for tests:   14.985 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      1140715856 bytes
HTML transferred:       1138561771 bytes
Requests per second:    667.34 [#/sec] (mean)
Time per request:       74.925 [ms] (mean)
Time per request:       1.498 [ms] (mean, across all concurrent requests)
Transfer rate:          74339.90 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median   max
Connect:        0    9 10.1      5     123
Processing:     9   66 83.2     52     975
Waiting:        0   41 83.7     27     949
Total:          9   74 82.6     59     980

Percentage of the requests served within a certain time (ms)
50%     59
66%     66
75%     72
80%     76
90%     98
95%    123
98%    192
99%    675
100%    980 (longest request)

Tomcat Clustering & Java Servlet Specification

bserban — Tue, 04 Aug 2009 11:34:51 +0000

After I read more about Tomcat Clustering I realized that the main purpose of Tomcat clustering is to offer fault tolerance, failover and high availability support. I read a lot about load balancing but when it comes to Java Servlets I found out that the only choice you have in terms of balancing is to use sticky sessions. This is more a limitation that comes from Java Servlet Specification and not from Tomcat, but it make sense.

For an application to be “distributed” you have to mark it as “distributable” by add the tag in web.xml.

There are multiple ways to balance the client request to your server pool but when it comes to Java Servlet Specification you have only one choice, as the specs say:

“Within an application that is marked as distributable, all requests that are part of a session can only be handled on a single JVM at any one time.”

“You may have multiple JVMs, each handling requests from different clients concurrently for any given distributable web application”

So, I guess you can kiss goodbye the round robin and all other load balancing options, but at least Tomcat will provide you failover, scalability and high availability.

bogdan@j3e » clustering

How a DNS problem can put your Mysql server down

Configure Apache and Tomcat severs together

Using mod_proxy

Using mod_proxy_ajp

Using mod_jk

Test each configuration using Apache Benchmark tool

Test results

Test result trace for Tomcat HTTP

Test result trace for mod_proxy_ajp

Test result trace for mod_proxy

Tomcat Clustering & Java Servlet Specification