bogdan@j3e » Architecture

How a DNS problem can put your Mysql server down

admin — Sat, 16 Jan 2010 15:10:39 +0000

Last week i was waked up from bed by the monitoring team from my company. There was a problem with my system, there was a DNS problem undergoing but as a side effect my app was down. Since it has a lot of traffic it had to be solved immediately.

I jumped to the computer and I quickly diagnosed the system. Everything was fine except the Mysql connection pool which was exhausted. The first thing that crossed my mind is that it was just a coincidence and I quickly ran show processlist to see a list of MySQL processes. The output was an infinite list of load balancer’s ip address having “login” text as status. In order to achieve high availability i am using Mysql by having a balanced ip address between two Mysql servers. The balancer runs a quick check every 5 seconds by connecting to Mysql and does a simple select on a table.

So for a particular reason the “load balancer” was not able to finish its login attempts and it was overloading my Mysql servers. While I was in the middle of the investigation the problem suddenly stopped. I was happy but somehow scared, i had no idea what the hell happened.

A quick search into Mysql documentation reveals that Mysql is doing a reverse DNS lookup which was the cause of my problems. Since the DNS server had a problem, the operation of reverse DNS was taking far more that 5 seconds to time out. This resulted in overloading the database servers. Check this explanation in the official documentation, How MySQL Uses DNS

After reading tha page I think that mysql needs this reverse DNS lookup only for its permission module and if you don’t use host names with the grant option then you are safe to disable this option. I quote here the parameter which does this:

–skip-name-resolve

Do not resolve host names when checking client connections. Use only IP numbers. If you use this option, all Host column values in the grant tables must be IP numbers or localhost. See Section 7.5.11, “How MySQL Uses DNS”.

I have been able to avoid this? Perhaps, but considering that I used MySQL in production for the first time, it is unlikely to think so.

Long live the reverse DNS, cheers!

Creating a secure JMX Agent in JDK 1.5

bserban — Sat, 31 Oct 2009 17:03:39 +0000

What is JMX?

Java Management Extension is an open technology for management, and monitoring that can be deployed wherever management and monitoring are needed. The most common use in a web application is for application management. This is very often an afterthought which results in many unmanaged application deployments.

You can monitor you application for availability and performance but in the same time you can use the JMX to manage and monitor you application from business perspective. Application’s runtime metrics can be expose through JMX, or in a service oriented architecture you could use JMX to control your services.

All good but when you start to work with JMX and JDK 1.5 soon you will discover one big limitation that was fixed in jdk 1.6 update 16 if i recall correctly:

Default RMI JMX agent for remote access opens 2 ports, one which is set by the -Dcom.sun.management.jmxremote.port=XXXX and one randomly assigned port.. What about firewalls?

JMX service url

service:jmx:rmi://hostname:port1/jndi/rmi//hostname:port2/jmxrmi

Where:

port1 is the port number on which the RMIServer and RMIConnection remote objects are exported
port2 is the port number of the RMI Registry

To access the RMIAgent you only need to know were the RMI registry is located from which to obtain the connection objects. I guess this was the reason to randomly assign the port1, but you have pretty high chances to have a firewall problem.

The solution is to replace the default agent and to create your own version of JMX Agent to provide access to RMI connection on a specific port.

JMX Architecture

There are three main components that makes the JMX possible:

Instrumentation layer, the managed beans and their resources. What you want to manage
JMX Agent, standard management agent that directly controls resources and makes them available to remote management applications. It is a mean to expose the managed beans, mbean server, monitoring, timing, relation, and class-loading services.
Remote management, permits the interaction between remote clients and the JMX Agent. There are a couple of default adapters that are built in, these are: HTTP Adapter (for viewing management data), RMI Adapter, SOAP Adapter, and SNMP Adapter.

Creating the Agent

To overcome the problems in JDK 1.5 explained above we need to create an java agent to export the RMI Registry and the RMIServer on specific ports.

This is pretty straight forward, we just need to export the RMIRegistry o a specific port, to get the MBeanServer and to create a JMXConnectorServer between those two. Then we just need to start the connector server and it is done

....................
LocateRegistry.createRegistry(port);
MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
final String hostname = InetAddress.getLocalHost().getHostName();
JMXServiceURL url = new JMXServiceURL("service:jmx:rmi://"+hostname+":"+port+"/jndi/rmi://"+hostname+":"+port+"/jmxrmi");
JMXConnectorServer cs = JMXConnectorServerFactory.newJMXConnectorServer(url, env, mbs);
cs.start();

Simple as it gets, but right now we don;t have any security in place. If we want to add SSL and authorization things will complicate a little.

Securing the Agent

To make the access secure we have to expose the RMIRegisty over the SSL. For this we need to make the following modification:

..............
SslRMIClientSocketFactory csf = new SslRMIClientSocketFactory();
SslRMIServerSocketFactory ssf = new SslRMIServerSocketFactory();
Registry registry = LocateRegistry.createRegistry(port, csf, ssf);
................

// Now specify the SSL Socket Factories:
//
// For the client side (remote)
//
env.put(RMIConnectorServer.RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE, csf);
// For the server side (local)
//
env.put(RMIConnectorServer.RMI_SERVER_SOCKET_FACTORY_ATTRIBUTE, ssf);
// For binding the JMX RMI Connector Server with the registry
// created above:
//
env.put("com.sun.jndi.rmi.factory.socket", csf);

final RMIServerImpl stub = new RMIJRMPServerImpl(port, csf, ssf, env);

final JMXConnectorServer cs =
new RMIConnectorServer(new JMXServiceURL("rmi", hostname, port),
env, stub, mbs) {
@Override
public JMXServiceURL getAddress() {
return url;
}
@Override
public synchronized void start() throws IOException {
try {
registry.bind("jmxrmi", stub);
} catch (AlreadyBoundException x) {
final IOException io = new IOException(x.getMessage());
io.initCause(x);
throw io;
}
super.start();
}
};
cs.start();

As you see we had to secure also the connection between ConnectorServer and JMX agent and for RMI server and client. Not that simple as the unsecure version. To add the authorization we just need to provide the credentials. Usually these are store in files and passed to the jmx connector server using environment variables:

env.put("jmx.remote.x.password.file", pasword);
env.put("jmx.remote.x.access.file", access);

And their content:

admin password1
monitor password2
..................
admin readwrite
monitor readonly

In order that SSL connections to work we need to create a keystore. This is done using keytool and can be create like this:

keytool -genkey -keyalg RSA -keysize 1024 -dname "CN=org.bserban.www" -keystore ./jmx-demo.jks -storepass bserban

Running the Agent

java -javaagent:jmx-agent.jar -Djavax.net.ssl.trustStore=./jmx-demo.jks -Djavax.net.ssl.trustPassword=bserban -Djavax.net.ssl.keyStore=./jmx-demo.jks -Djavax.net.ssl.keyStorePassword=bserban org.abserban.jmx.agent.StartAgentStandalone

Running the Client

java -Djavax.net.ssl.trustStore=./jmx-demo.jks -Djavax.net.ssl.trustPassword=bserban -Djavax.net.ssl.keyStore=./jmx-demo.jks -Djavax.net.ssl.keyStorePassword=bserban org.abserban.jmx.client.StartClient -host:localhost -port:8787 -status:on

See the comple code from Resources section. It includes the complete java code, ant file and the keystore.

Resources

Configure Apache and Tomcat severs together

bserban — Sat, 08 Aug 2009 08:12:00 +0000

The most common way to deploy your application in the production environment is to hide the Tomcat behind Apache. This has good and bad parts but it gives you a lot of flexibility and support from Apache. There are a couple of alternatives to put these two severs together:

mod_jk, this is the old connector developed under the Tomcat project and it is using the Tomcat’s AJP protocol. It is expected to be faster than the HTTP protocol which is text based.
mod_proxy, is the support module for HTTP protocol. It is TCP based and uses the HTTP which is plain text. When a web client makes a request to Apache, the Apache will make the same call to the Tomcat and then the Tomcat’s response is passed back to the web client. This connector is part of the Apache for a very long time and it is available also for older versions of Apache. This is the simplest way to put the Apache in front of a Tomcat but also the slowest way to do it.
mod_proxy_ajp, is new and is part of the Apache 2.2. It is working like mod_proxy, but as the name says it is using the AJP connector for sending and getting data from Tomcat. It is using also TCP and it is expected to be faster than plain mod_proxy

Using mod_proxy

Create your own tomcat-httpd.conf file and configure the proxy:

# # Server will not close the connection after each request allowing the browser to use the same connection # KeepAlive On MaxKeepAliveRequests 100 KeepAliveTimeout 5

#
# Load mod_proxy modules
#

LoadModule proxy_module modules/mod_proxy.so

LoadModule proxy_http_module modules/mod_proxy_http.so

ProxyRequests Off
ProxyPreserveHost On
ProxyTimeout 1000
TimeOut 1000

#
# Configure the mod_proxy
#
ProxyPass / http://127.0.0.1:8080/
ProxyPassReverse / http://127.0.0.1:8080/

Include your configuration into Apache httpd.conf using Include directive:

#load Tomcat proxy configuration Include /usr/local/tomcat6/conf/tomcat-httpd.conf

Using mod_proxy_ajp

The only difference between mod_proxy and mod_proxy_ajp is that you have to load mod_proxy_ajp and proxy the request to Tomcat using the ajp protocol.

# # Load mod_proxy modules # LoadModule proxy_module modules/mod_proxy.so

LoadModule proxy_http_module modules/mod_proxy_http.so

LoadModule proxy_ajp_module modules/mod_proxy_ajp.so

ProxyRequests Off
ProxyPreserveHost On
ProxyTimeout 1000
TimeOut 1000
#
# Enable the AJP proxy
#
ProxyPass / ajp://localhost:8009/
ProxyPassReverse / ajp://localhost:8009/

Include your configuration into Apache httpd.conf using Include directive:

#load Tomcat proxy configuration Include /usr/local/tomcat6/conf/tomcat-httpd.conf

Using mod_jk

Edit your tomcat-httpd.conf file and add mod_jk configuration:
# # Load mod_jk is is not loaded already # LoadModule jk_module modules/mod_jk.so # # # JkWorkersFile /etc/httpd/conf/workers.properties JkLogFile /var/logs/httpd/mod_jk.log JkLogLevel info JkLogStampFormat "[%a %b %d %H:%M:%S %Y] " JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories JkRequestLogFormat "%w %V %T" JkMount /test/* worker1

Include your configuration into Apache httpd.conf using Include directive:

#load mod_jk configuration configuration Include /usr/local/tomcat6/conf/tomcat-httpd.conf
Now edit the workers.properties and configure your worker:
worker.list=worker1 worker.worker1.type=ajp13 worker.worker1.host=localhost worker.worker1.port=8009 worker.worker1.connection_pool_size=150 worker.worker1.connection_pool_timeout=600 worker.worker1.socket_keepalive=1

Test each configuration using Apache Benchmark tool

Apache benchmark it is a great tool for testing the above configurations. All you need is to create a typical application page that should be hit with the ab tool. This tool takes a single url and makes requests repeatedly in separates threads. The number of threads is controlled by command line arguments. It also supports keep alive connections.

For more details about Apache Benchmark check this page Apache Benchmark

For testing purposes i have created a test war which has a test.jsp page. Because what we test does not influence the processing time overall we don’t need a complete test which includes a database call or working with certain frameworks. In the end all we need is the output of an application and to test how this output reach the browser using one of the three modes explained in the section above.

The test page will include a previous post of mine which is medium size: http://www.bserban.org/2009/05/put-together-struts2-jpa-hibernate-and-spring/. I have right-clicked in the browser, chosen the view source and copy and paste the content into test.jsp. The file has 112Kb in size.

bserban-mac:~ bserban$ ls -la ~/bin/srv/apache-tomcat-5.5.27/webapps/test/test.jsp
-rw-r–r– 1 bserban staff 113698 Aug 7 09:39 /Users/bserban/bin/srv/apache-tomcat-5.5.27/webapps/test/test.jsp

I am going to hit the Tomcat and the Apache with 10,000 request using 50 threads. The ab command looks like this:

ab -k -n 10000 -c 50 http://localhost/test/test.jsp

To access directly the Tomcat i am going to hit the 8080 port. To preserve the similar enviroment and test conditions I will restart the Apache and Tomcat after each test.

Test results

The table below summarize the results obtained.

	Direct	mode_proxy	mode_proxy_ajp
Throughput	992 reg/s	667 req/s	702 req/s
Average Response Time	50 ms	75 ms	71 ms
90% Response line	75 ms	90 ms	71 ms
100% Response line	207 ms	980 ms	972 ms

As you see, the fastest connecting mode is to connect directly to tomcat using HTTP. Direct HTTP connect will server more request per second that the other modes. The second choice is mode_proxy_ajp followed very closely by the mod_proxy. However the overhead added the the Apache will leverage for real life applications because the application processing time will minimize the impact of using Apache in front of Tomcat. Probably in real world the differences between direct HTTP connect and mod_proxy_ajp will not exceed 5-10% percents in terms of throughput and average time per request. This is the price to pay for the flexibility brought by Apache, because having the Apache in front of the Tomcat will give access to the whole Apache functionality and support.

For those who want to see test results in detail, I included in the post the tests logs.

Test result trace for Tomcat HTTP

bserban-mac:~ bserban$ ab -k -n 10000 -c 50 http://localhost:8080/test/test.jsp This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests

Server Software:        Apache-Coyote/1.1
Server Hostname:        localhost
Server Port:            8080

Document Path: /test/test.jsp
Document Length: 113678 bytes

Concurrency Level:      50
Time taken for tests:   10.079 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      1139157786 bytes
HTML transferred:       1137007356 bytes
Requests per second:    992.18 [#/sec] (mean)
Time per request:       50.394 [ms] (mean)
Time per request:       1.008 [ms] (mean, across all concurrent requests)
Transfer rate:          110376.03 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median   max
Connect:        0    7   7.3      5      67
Processing:     5   43 22.4     37     182
Waiting:        0   16 19.9     11     161
Total:          7   50 22.4     45     207

Percentage of the requests served within a certain time (ms)
50%     45
66%     52
75%     57
80%     61
90%     75
95%     91
98%    119
99%    147
100%    207 (longest request)

Test result trace for mod_proxy_ajp

bserban-mac:~ bserban$ ab -k -n 10000 -c 50 http://localhost/test/test.jsp This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/

Server Software:
Server Hostname: localhost
Server Port: 80

Document Path: /test/test.jsp
Document Length: 113678 bytes

Concurrency Level:      50
Time taken for tests:   14.251 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      1140884936 bytes
HTML transferred:       1139000800 bytes
Requests per second:    701.70 [#/sec] (mean)
Time per request:       71.256 [ms] (mean)
Time per request:       1.425 [ms] (mean, across all concurrent requests)
Transfer rate:          78179.54 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median   max
Connect:        0    7   8.3      5      87
Processing:     6   64 89.8     47     967
Waiting:        0   38 86.9     21     920
Total:          6   71 89.4     52     972

Percentage of the requests served within a certain time (ms)
50%     52
66%     58
75%     64
80%     69
90%     83
95%    115
98%    507
99%    689
100%    972 (longest request)

Test result trace for mod_proxy

Server Software:        Apache-Coyote/1.1
Server Hostname:        localhost
Server Port:            80

Document Path: /test/test.jsp
Document Length: 113678 bytes

Concurrency Level:      50
Time taken for tests:   14.985 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      1140715856 bytes
HTML transferred:       1138561771 bytes
Requests per second:    667.34 [#/sec] (mean)
Time per request:       74.925 [ms] (mean)
Time per request:       1.498 [ms] (mean, across all concurrent requests)
Transfer rate:          74339.90 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median   max
Connect:        0    9 10.1      5     123
Processing:     9   66 83.2     52     975
Waiting:        0   41 83.7     27     949
Total:          9   74 82.6     59     980

Percentage of the requests served within a certain time (ms)
50%     59
66%     66
75%     72
80%     76
90%     98
95%    123
98%    192
99%    675
100%    980 (longest request)

Tomcat Clustering & Java Servlet Specification

bserban — Tue, 04 Aug 2009 11:34:51 +0000

After I read more about Tomcat Clustering I realized that the main purpose of Tomcat clustering is to offer fault tolerance, failover and high availability support. I read a lot about load balancing but when it comes to Java Servlets I found out that the only choice you have in terms of balancing is to use sticky sessions. This is more a limitation that comes from Java Servlet Specification and not from Tomcat, but it make sense.

For an application to be “distributed” you have to mark it as “distributable” by add the tag in web.xml.

There are multiple ways to balance the client request to your server pool but when it comes to Java Servlet Specification you have only one choice, as the specs say:

“Within an application that is marked as distributable, all requests that are part of a session can only be handled on a single JVM at any one time.”

“You may have multiple JVMs, each handling requests from different clients concurrently for any given distributable web application”

So, I guess you can kiss goodbye the round robin and all other load balancing options, but at least Tomcat will provide you failover, scalability and high availability.

Tomcat clustering configuration

bserban — Tue, 09 Jun 2009 08:33:36 +0000

The following steps assume that you have installed a Tomcat 5.5.x bundle or latest, i only tested on 5.5.27 but is should work for other configuration as well. The network configuration apply to Linux and may vary with the distribution. It should work as is for distributions based on Red Hat.

For Tomcat clustering we have two main things to configure:

Configure the network environment for clustering (open ports, add multicast route),
Configure Tomcat clustering support.

Configure the network support for cluster

Opening Specific HTTP Ports (e.g. Port 45564, 4001)

The cluster class will start up a membership service (multicast) and a replication service (tcp unicast). See also http://www.cyberciti.biz/faq/howto-rhel-linux-open-port-using-iptables/ for a brief article on this. You will need to have root access as noted above to complete this.

Your server may or may not already have this entry. Open iptables:

> vi /etc/sysconfig/iptables

Add the following entries:

-A RH-Firewall-1-INPUT -p udp -m udp --dport 45564 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 45564 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 4001 -j ACCEPT

Save and close the above file and after restart the iptables

  > /etc/init.d/iptables restart

Configure the multicast address and routes.

Clustering membership is established using very simple multicast pings. Each Tomcat instance will periodically send out a multicast ping, in the ping message the instance will broad cast its IP and TCP listen port for replication. If an instance has not received such a ping within a given timeframe, the member is considered dead.

Add route (the server’s ip address)

sudo /sbin/route add 228.0.0.4 gw 10.72.10.1 dev bond0

Edit rc.local to make the change persistent through restarts.

sudo vim /etc/rc.d/rc.local

Add this line at the end (the server’s ip address)

/sbin/route add 228.0.0.4 gw 10.72.10.1 dev bond0

Configure Tomcat to support clustering.

Application clustering with Tomcat has two steps:

Enable clustering support,
Make you application clusterizable.

Enable Tomcat clustering support

You need to enable the cluster support in Tomcat by editing the server.xml file. Open server.xml

sudo vim /usr/local/tomcat-5.5.27/conf/server.xml

Enable clustering configuration in the configuration file, notice that the default configuration is using the DeltaManager which will replicate only the session’s changes and not the entire object:

One main condition for replication to work is that your session content is serializable. Add a _jvmRoute_ to your Tomcat Engine section From

jvmRoute identifies unique a Tomcat instance in a cluster. If multiple servers are used I recommend you to use descriptive names.

Make your application clusterizable

Configuring Tomcat clustering is not enough to cluster your application. For that you need to tell Tomcat which application you want to be clusterizable. This is achieved in two ways:

by modifying the ROOT.xml (the context configuration file”
by modifying the web.xml

Enable application clustering by ROOT.xml

Edit ROOT.xml file

 sudo vim /usr/local/tomcat-5.5.27/conf/Catalina/localhost/ROOT.xml

Look for

Change it to

Enable application clustering by editing the web.xml

Edit the web.xml file

 sudo vim /usr/local/tomcat-5.5.27/webapps/ROOT/WEB-INF/web.xml

Look for:

 
	
	contextClass
	.............

Change it to:

 
	
	
	contextClass
	.............

Restart Tomcat

 cd /usr/local/tomcat-5.5.27/bin/
sudo ./shutdown.sh
sudo ./startup.sh
or if you have a init script
sudo /etc/init.d/tomcat5 restart

You need to configure all the nodes in the cluster as detailed above. Every node should have unique name provided by “jvmRoute” attribute.

Key-Value Storage using MemcacheDB

bserban — Mon, 30 Mar 2009 21:08:37 +0000

What is Entity-Attribute-Value model (aka key-value storage)

This is also know as Entity-Attribute-Value model, and it is used in circumstances where the number of attributes (properties) that can be used to describe an entity is very vast but the number of attributes that will actually be used is modest.

Let’s think in terms of a database how an Entity-Attribute-Value model would look like for storing an user profile.

id	user_id	key	value
1	101	screen_name	john
2	101	first_name	John
3	101	last_name	Smith

The table has one row for each Attribute-Value pair. In practice, we prefer to separate values based on data type to let the database to perform type validation checks and to support proper indexing. So programmers tend to create separate EAV tables for strings, real and integer numbers, dates, long text and BLOBS.

The benefits of such structure are:

Flexibility, there is no limit on attributes used to describe an entity. No schema redesign.
The storage is efficient on sparse data.
Easy to put the data into an XML format for interchange.

There are also some important drawbacks:

No real use of data types
Awkward use of database constraints
There are several problems in querying such a structure.

What is MemcacheDB

Memcachedb is a distributed key-value storage system designed for persistence. It is a very fast an reliable distributed storage. It includes transaction and replication. It is using Berkeley DB as persistence storage.

Why is better than a database?

Faster, no SQL engine on top of MemcacheDB
Designed for concurrency, design for millions of requests
Optimized for small data

Memcachedb is suitable for Messaging, metadata storage, Identity Management (Accounts, Profiles, Preferences, etc), index, counters, flags, etc.

The main features for Memcachedb are:

High performance read/write for a key-value based object
Rapid set/get for a key-value based object, not relational. Benchmark
will tell you the true later.
High reliable persistent storage with transaction Transaction is used to make your data more reliable
High availability data storage with replication Replication rocks! Achieve your HA, spread your read, make your transaction durable
Memcache protocol compatibility Lots of Memcached Client APIs can be used for Memcachedb, almost in any language, Perl, C, Python, Java

Storage, replication and recovery

Berkeley DB stores data quickly and easily without the overhead found in other databases. Read more about Berkeley DB here

MemcacheDB supports replication using Masters and Slaves nodes. The exact deployment design must chosen according with your application needs. A MemcacheDB environment consists intro three things:

Database files, files that store your data
Log files, all your transaction commit first into logs
Region files, back the share memory region

One problem could be spot in Log files, that record you transaction, over time they will contain a lot of data making the recovery a pain moment. For this Memcache DB has a Checkpoint. The checkpoint empties the in-memory cache, writes a checkpoint record, flushes the logs and writes a list of open database files.

Berkeley DB also allows hot backups and uses gzip and tar to compress the backup.

Monitoring

Memcache DB has a lot of built in commands for monitoring, such as:

Current status: stats
Database engine status: stats db
Replication status: stats rep

What i liked most at Memcached is that you can use telnet to log on the running process and issue commands from command prompt. The same thing is valid also for MemcacheDB.

Besides memcached built function the Berkeley DB engine comes with his own stats command:

db_stats, –c locking statistics, –l logging statistics, –m cache statistics, –r replication statistics, –t transaction statistics.

Overall i liked what i saw about this alternative and i think that this is the most suitable solution for storing user profiles and user data that don’t need to be queried. When you need to scale this is for sure a very reliable solution. Have fun!

Facebook temporarily lost data.

bserban — Thu, 12 Mar 2009 22:51:20 +0000

Last Sunday Facebook reported a data loss. We are talking about approximately 15% of users’ photos. Loosing your client’s data is the worst thing that could happen to you and reminded me what a guy said once in a tech talk: “The main rules in running an online community service are: Never lose data and never go to jail.”

Facebook has not yet made public the details of what happened but only assured users that their photos will be restored using a backup. The official report states that we are talking about a hardware failure at storage level.

First of some key facts about Facebook

Facebook is the number 5 site in the world, which means it has a huge traffic (source Alexa.com),
They have 10,000 servers including 1,800 MySQL servers (administrated only by two guys, they say),
Last October the users uploaded 10 billion pictures on Facebook and considering that they keep 4 back-up copies it means that they have to store 40 billion pictures,
2-3 Terabytes of photos are uploaded every day,
They serve 15 billion photo images per day,
Daily uploads are around 100 million photos,
The peak is about 450,000 images per second.

Based on the above numbers it means that they lost approximately 1.5 billion of pictures. Waw!

How is Facebook handling user’s images? Last year Jason Sobel, Manager of the Facebook Infrastructure Group, presented some insights about the current Facebook storage solution and the future one. We don’t know right now whether the new storage solution failed or the old one is to blame.

Writing files using the old way

They were using upload servers and stored images via NFS into a NetApp storage (last year they were planning to replace it). Each image is stored 4 times. This solution experienced heavy workload when processing metadata.

Reading files using the old way

Here all resumes to speed.

First level of Cache is done using CDN, which has a hit rate of 99.8% for profiles and 92% for the rest.
Second Level of Cache is done using Cachr for profiles which is a modified evhttpd with memcached as storage, and a File Handle Cache (lighttpd and memcached) for the rest of it to reduce metadata workload on NetApp.
NetApp storage via NFS. They tried to optimize it and to reduce the number of I/O access because of the the metadata heavy workload.

The main concerns with the above architecture are: Netapp storage is overwhelmed, they rely too much on CDNs.

Obviously when your app grows like hell, you start to think that is better to make your own toys, fully customized and optimized for your particular problem. So did Amazon back in 2001 and Google too. This is how the Facebook storage was born: Haystack

Haystack

The answer was to develop in house a distributed file system like GFS (Google File System). Haystack should run on inexpensive commodity hardware, and it should deliver high aggregate performance to a large number of clients.

Haystack is file based and stores arbitrary data in files. For 1Gb disk data file they create 1M in memory index. In this way they have one disk seek which is much better than NetApp which had 3.

The Haystack format is rather simple and efficient, Version number, Magic number (supplied by the client to prevent brute force attack), length, data, checksum. The index simply stores the Version, Photo key, Photo size, start, length.

Using a Haystack server

To write uses POST

/write/[pvid]_[key]_[magic]_[size].jpg

- writes data on disk haystack

- writes data on in memory index

To read uses GET

/[pvid]_[key]_[magic]_[size].jpg

- uses the in memory index to retrieve the offset

- reads data from the on-disk file

This simple approach allows Facebook to easily balance the reads and writes using Haystack clusters but to speed up the reads they still plan to use CDNs in areas where they don’t have datacenters and Cachr for profiles. This is their first step to create their own CDN network.

Additional readings

Needle in a haystack: efficient storage of billions of photos

Engineering[at]Facebook’s Notes

Simple way to scale your Web App – Part 1

bserban — Sun, 08 Feb 2009 12:12:50 +0000

Every time i made an J2EE application, my main concern was about how many requests it could handle in the end. In my early dawns of my career I have been in the situation when my app was his own victim of success, to much load for a single server. We always had a reactive attitude and tried to deal with the problem when it happened, but some times it was too damn late. To be able to scale you application it must be made to be scaled.

But what would be a simple scalable architecture?

Let’s consider the diagram from the left. Our application will be splited in three logical clusters. First one is the application cluster, second database cluster and finally logging and statistic cluster.

The Load Balancer

A load balancer distributes the traffic among you application servers. It can be software or a hardware device. A handy solution is to use a software balancer, such an Apache, but the software solution is not so robust and performant as a hardware balancer.

Hardware

A dedicated device for load balancing is more suitable and gives you more performance. Thus, this comes with an additional costs but there many devices on the market and you should choose the best one on cost/ features. When evaluating a load balancer some things must be kept in mind

Maximum connection supported
Throughput
RIPS (Real IP Address of a Real Server)
SSL
Gb-NICs
Price

Software alternatives

Apache, mode_rewrite, it can be used to provide simple url based balancing capabilities.
Apache, mod_proxy_balancer, It provides load balancing support for HTTP, FTP and AJP13 protocols. The best option for a medium web application.
Apache, mod backhand, http and log spread. It is considered more performant than mod_proxy_balancer
Perball from Danga, already used for LiveJournal, Vox and TypePad
Varnish, open source http server, a new commer. Has also load balancing capabilities , see here http://varnish.projects.linpro.no/wiki/FAQ
Pure Load Balancer [PLB], for http and smtp.
Linux Virtual Server, an advanced load balancing solution can be used to build highly scalable and highly available network services, such as scalable web, cache, mail, ftp, media and VoIP services. Though this solution is for bigger purposes than scale a simple application.

To balance SSL connections your balancer should provide SSL termination capability. Otherwise being a connection level protocol the SSL connection should persists between server and client by allocating the same host to the same client.

More about Load Balancing in a future post.

The Application Cluster

To be able to scale a web application, it has to be designed to scale. The main issue is session replication. Depending on the load balancing algorithm you will need to replicate the session or to use sticky session.

Stateless

This mode doesn’t require much from a load balancer. This is very common to REST applications. For a web 2.0 application this could be the common aproach. The application stores everything it needs on the client side. The first user request goes on the first machine while de second will hit a different machine. No data has to be shared between web servers. To handle more requiest new servers can be added in the web pool and the system will scale out.

Having a stateless design allows us a seamless failover. This can be achieved no matter what language we use to develop our application

Sticky session

Some times our application needs to store user specific data at session level. This means that every time a user hits the server we need some data to be able to process user request, we need local data and state. This data must be available on all server where the user request come. To cope with this problem we can use “sticky session” which means we need to ensure that the user will hit the same server as the initial request.

The most common aproach with sticky sessions is:

a cookie that holds the routing information
configuration that defines this cookie id for the balancer
configuration that defines routing for each back-end

Session replication

This technique is very common in J2EE where the Servlet Containers provide a way to replicate the session between the servers. There are several condition for a session to be replicated but it can be a viable alternative to sticky session.

The Database cluster

Obviously, my immediate choice will be to use MySQL as database server. It’s free, has a lot of community, it serves a lot of well known web 2.0 sites.

Using MySql you can scale horizontal by using MySql’s replication mechanism. When the database grows is time to partition our data horizontally into shards.

Replication

Master-Slave

This type of configuration will help you to scale the reads from the write. The reads will always go to the Slave while the transactions that alter the data will go to the Master. You can have one Master and multiple slaves.

Master – Master

Such an approach will distribute the load evenly and it also provides High Availability. MySQL 5.0 provides replication at statement level which often can crash the replication because of conflicts. The most common conflicts i ever met are for unique indexes but you can coupe with this problem by using “replace” command instead of insert. Anyway the MySQL 5.1 will have some semnificative changes in the replication module.

Tree Replication

This gives you a lot of posibilities, you can conbine master-master with master-slave in a tree structure and conbined with sharding it can result into a fine tuned MySQL cluster. More about Tree Replication and data partitioning in further posts.

Logging & Batch processing

In the end we reached the final component of our application.: the logging server and the batch processing machine. Why do we need them? Simply because somebody has to do them. Every application has some batch processing to be done. This is done usually by using cron and scripts. I recommend to use a scripting language for batch processing such as: bash, ruby, python etc.

For logging I would suggest you to use syslog or syslog-ng which has more advanced features than syslog, better performance in terms of cpu and supports UTF-8.

What’s next?

Next I would like to walk step by step through designing a J2EE based on the architecture discussed above by using Apache, Tomcat, Struts, Hibernate and MySQL.

bogdan@j3e » Architecture

How a DNS problem can put your Mysql server down

Creating a secure JMX Agent in JDK 1.5

What is JMX?

JMX service url

JMX Architecture

Creating the Agent

Securing the Agent

Running the Agent

Running the Client

Resources

Configure Apache and Tomcat severs together

Using mod_proxy

Using mod_proxy_ajp

Using mod_jk

Test each configuration using Apache Benchmark tool

Test results

Test result trace for Tomcat HTTP

Test result trace for mod_proxy_ajp

Test result trace for mod_proxy

Tomcat Clustering & Java Servlet Specification

Tomcat clustering configuration

Configure the network support for cluster

Opening Specific HTTP Ports (e.g. Port 45564, 4001)

Configure the multicast address and routes.

Configure Tomcat to support clustering.

Enable Tomcat clustering support

Make your application clusterizable

Enable application clustering by ROOT.xml

Enable application clustering by editing the web.xml

Further reading

Key-Value Storage using MemcacheDB

What is Entity-Attribute-Value model (aka key-value storage)

What is MemcacheDB

Storage, replication and recovery

Monitoring

Further reading

Facebook temporarily lost data.

Writing files using the old way

Reading files using the old way

Haystack

Additional readings

Simple way to scale your Web App – Part 1

The Load Balancer

Hardware

Software alternatives

The Application Cluster

Stateless

Sticky session

Session replication

The Database cluster

Replication

Master-Slave

Master – Master

Tree Replication

Logging & Batch processing

What’s next?