Alex Le’s Blog

For the past few days I’ve been working on the cluster project again, after a month of suspension. Finally today I was able to configure a (almost) working load balancer with 3 web servers (1 is the loadbalancer itself, and 2 other dedicated web servers, web01 and web02). I’m in the process of writing down the scripts which I will follow later on to do the screencast. At this stage, what I have is half of the whole picture of a web cluster:

• Failover loadbalancer server pair, running Heartbeat and ldirectord
• A small cluster of 3 load-balanced web servers, using weighted round robin
• A pair of Nameserver to provide name-lookup service for the entire cluster, no more “ping 10.10.10.10.101.101.10.12“!

What is missing is the loadbalanced MySQL cluster. I have been able to get a cluster of 3 MySQL boxes working, but now I’d like to add the ipvsadm and ldirectord to provide load-balancing feature so that the cluster can be scaled up easier.

I am pretty proud of myself for learning all of these amazing technologies within a relatively short period of time. Especially for the DNS, I have learned so much about the networking aspect of a network of computer. The two books, “DNS & BIND” and “BIND Cookbook”, both by O’Reilly, are amazing reference source. I particularly like the “DNS & BIND” book for its in depth coverage on the concept of how nameservers work. Without that fundamental knowledge, it’s hard to build such a working cluster with growing demands.

Here is the revised virtual Linux cluster diagram. So slowly but surely my cluster project is reaching its destination: a complete virtualized web cluster running on VMWare. And once this is done, I think I begin to learn how to develop cluster-ready applications.

I’ve been doing quite a bit of reading about High-Availability (HA) web clustering techniques for the past 2 weeks. Thanks to VMWare, now it is possible to create a virtual web server farm with multiple linux instances running concurrently to simulate a cluster setting. The initial result is very heart-warming: I’ve sucessfully installed a mysql database, configure fail-over (thus means high-avaibility) web servers using heartbeat, and name servers running BIND9 to do name lookup. The last bit of the puzzle is now the Linux Virtual Server (LVS) Director for load-balancing. That would be my next experiment.

Here is my initial diagram of my web servers farm and the IP/server name assignments

With this design, I tried to eliminate the single point of failure by implementing redundant, ready to fail-over servers for critical sections of the network, for example, the DNS, the load balancers for incoming traffic, and the load balancers for the database farm.

Basically, incoming traffic will pass through A, the load balancer. The load balancer A will spread out the load to the web servers C in the farm using LVS-DR method (direct-routing). There are also 2 kinds of web servers: one is supposed to be the beefy, powerful server with fast CPU to run the web applications, and the other is the media server which doesn’t need good CPU but requires fast HDD (SCSI) and lots of RAM. The applications server will do the number crunching, churning out pages as fast as they can while the media servers will provide all the images, CSS, and javascript files. Of course since I am using VMWare, it virtually costs me $0.00 to add a new scsi drive to the VM machine. Great!
The name servers B running Bind9 are located centrally to help with the name resolutions. Of course DNS is critical so we need to have a certain level of HA. Hearbeat will make sure the DNS is always up and ready.

Meanwhile, for the database farm E, which is on a separate network (supposed to be high-speed, low latency with very expensive switches) a pair of load balancers is needed to spread out the “read” (SELECT) load. I’m not quite sure how to implement the “write” (UPDATE/ DELETE, ALTER TABLE, etc.) DB servers yet, but I’m sure that we can improvise along the way. Again, Heartbeat will be implemented to keep the database load balancers up and happy. Our database farm will consist of 2 network storage nodes to store data and 2 “API” nodes to do the database heavy lifting. A fifth server is used to be the management node to manage (add, delete, or update) the database servers.

Finally, (and not shown in the above diagram as I just realize that I am missing something), a monitoring server running Nagios is implemented to do health-monitoring and network management. With the current design, all part of the network can be scaled independently: if more web servers are needed, we add new boxes to section C. If we need more database storage nodes, we can quickly add a new NDB node to the MySQL database cluster F. The bottle neck will now be our gateway, the load balancers in A. However, since it’s been confirmed (see the linux-ha.org site) that a decent load balancer can easily handle the amount traffic to saturate a 100Mbps connection, I would say for a small/ medium business settings, this is more than enough.

If you are asking why I am writing all of this down. I am doing this because I will begin to construct this web farm using VMWare with CentOS 4.4. The post and the diagram will serve as a guideline for this particular project. Moreover, I intend to do screencast of the entire process of setting up this web farm. Yes, I’d like to commoditize the knowledge of building Linux cluster using off-the-shelf tools. It’s a noble goal, I know, but I’m doing it for myself first so you don’t have to thank me now.
Now off to work I go. Keep on checking back alexle.net for more information about Web clustering. “This is Alex Le doing it so you don’t have to.” (yeah, I copy Ze frank’s line, so excuse me for the plagiarism.

Last nite, I was surfing around and found this page at the ThePlanet.com forum. Someone has configured a subdomain to proxy to the Plesk admin page. This way, in stead of accessing Plesk admin page via port 8443 (which I forget half of the time), I can access the page via https//plesk.mydomain.com (still encrypted with SSL for your privacy and security). In case I am behind a firewall that blocks most incoming/ outgoing ports, I still can access my plesk control panel page via the standard SSL port of 443.

From the forum thread (see the link above), the setup is pretty straightforward, but for some reasons, a few people could not configure the subdomain to act as a proxy to the Plesk control panel. The reason is there is a bug in how Plesk 8.0 writes the domain/ subdomain configuration files for Apache. And as there were a few people asking for a more detailed step-by-step tutorial, here is my take on it.

Let’s first go over the setup and configuration, then we can talk about the bug in Plesk. I assume that you have root access to your box because the Plesk manual explicitly states that you have to be root to create the vhost_ssl.conf file, which is the file we need to create to turn the subdomain into the proxy to the Plesk control panel.

The setup

• Create a subdomain in Plesk. Remeber to enable SSL support, PHP support, and CGI support. By checking these 2 options, we are forcing Plesk to add an “include” directive to the configuration file of the domain. We discuss more on this later. I created a subdomain called “plesk” so that I can use it like https://plesk.alexle.net/ to access to the Plesk CP. From now on, let’s use “plesk” as our subdomain and alexle.net as our main domain.
• Next, SSH to your box as root, then cd to the configurations folder of the newly created subdomain:
  #cd /var/www/vhosts/alexle.net/subdomains/plesk/conf/
• Create a vhost_ssl.conf file ins this conf folder using your favorite text editor. The content of this file should be
  

  SSLProxyEngine on
  ProxyRequests off
  ProxyPass / https://www.alexle.net:8443/
  ProxyPassReverse / https://www.alexle.net:8443/

  What we are doing is to turn on the SSLProxyEngine for this particular subdomain. In order for this to work, we have to turn the ProxyRequests off. Next, we tell Apache to pass all traffics (or requests) from the root / access of the subdomain to the destination URL (which means all future requests from https://plesk.alexle.net/ will be “ProxyPass“ed to https://www.alexle.net:8443/) . Then with ProxyPassReverse, we tell Apache to redirect the response from https://www.alexle.net:8443/ back to /, our “https://plesk.alexle.net”. (By the way, you will have to use your own domain/ subdomain instead www.alexle.net in the above configuration)

• Finally, for Apache to pickup the the new configuration file, we need to restart the httpd service
  #/etc/init.d/httpd restart
• Surf to htps://plesk.alexle.net/. Humh, it doesn’t work yet? Here is the Plesk bug.

The Bug

The way Plesk generates and stores configurration files are covered in the Plesk Manual Page. Basically there’s a master Apache config file at /etc/httpd/conf. This master Apache configuration file will include a whole bunch of other specific configuration files for each domains (located in /var/www/vhosts/domain_name/conf/httpd.include) and subdomains (/var/www/vhosts/domain_name/subdomains/your_subdomains/conf/vhost.conf or vhost_ssl.conf). Phew.

Remeber when we created the subdomain, we have selected PHP support and CGI support. By doing this, we have forced Plesk to write an “include” directive in the configuration file of the main domain to include the configuration file of the subdomain. To clarify, if you now open up the file httpd.include (in my case, at /var/www/vhosts/alexle.net/conf/httpd.include), find the VirtualHost section for your newly created subdomain, you will find a line similar to this

#file: /var/www/vhosts/alexle.net/conf/httpd.include
Include /var/www/vhosts/alexle.net/subdomains/plesk/conf/vhost.conf

However, there are 2 VirtualHost sections for the plesk subdomain: one is for port 80, and the other is for port 443, SSL. Moreover, both sections include the same vhost.conf file. This is wrong. Based on the Plesk’s Manual, the VirtualHost section for the SSL at port 443 should include the vhost_ssl.conf instead of the vhost.conf. So that’s the Plesk 8.0 bug. I’ve tried a couple times and was able to reproduce the Include bug.

The fix

Now we know why our configuration file for SSL access via the subdomain is not picked up. We need to change the Include directive of the VirtualHost SSL section for the subdomain to use the vhost_ssl.conf file instead. Then restart apache (#/etc/init.d/httpd restart) and everything should works fine.

The catch

We are modifying the main domain’s httpd.include file, which Plesk will overwrite everytime we make changes to this particular domain. Once that happens, you will have to re-modify the httpd.include file so that the correct vhost_ssl.conf file is used.

I hope that this short article helps you work and understand Plesk better. Thanks for the thread from ThePlanet.com forum to inspire me for this article. Comments are welcome as always.

I was trying to setup a MySQL cluster with CentOS 4.4 and VMWare and I was initially successful at getting the NDB storage engine to work. However, I could not start the MySQL API (or MySQL client, whatever you call it) because I ran into the infamous MySQL error of “ERROR 2002 (HY000): Can’t connect to local MySQL server through socket ‘/var/lib/mysql/mysql.sock’ (2)“. Googling didn’t help much either because people were just asking and very few got the answer.

After a while, I was desparate. My cluster was up and I had no way to connect to it and test out the exciting cluster’s replicate feature. I went to bed at 4:00am last night and kept thinking what could go wrong? Setting up the ndb engine for the nodes was pretty straightforward but why just getting mysql client to run is so problematic? Well, here’s my solution to the puzzle:

My Setup

• CentOS 4.4 installed from DVD with most packages unchecked, especially MySQL
• The newest MySQL 5.0.27-max was used. See, I’m on the bleeding-side of technology.
• MySQL is locaed in the default location at /usr/local/mysql/
• When I installed MySQL, the mysql.server script was cp from /usr/local/mysql/support-files/mysql.server to /etc/init.d/
• The my.cnf file is very simple like this

#file: /etc/my.cnf
ndbcluster
ndb-connectstring=’host=192.168.2.100′
[mysql_cluster]
ndb-connectring=’host=192.168.2.100′

Cause:

When the MySQL server (mysqld) starts, it created a sock file. Since I overwrote the original /etc/my.cnf without the socket parameter under [mysqld], the MySQL server (mysqld) created the socket file under /tmp/mysql.sock instead. Furthermore, I was also missing the socket parameter under [client] section in the my.cnf - I didn’t even have the [client] section.

What happens when I tried to start the mysql client (#mysql -u root) was that the mysql client was looking for the mysql.sock file but coudln’t find the path to the file. It then looked into the default path /usr/lib/mysql/ to find the mysql.sock file. Of course this doesn’t work because when mysqld was started, the socket file was created in /tmp instead. The client gave up, spit out the connect error prompt.

In brief, the cause of the infamous error is because the mysql client cannot find the socket file at run time.

Solutions

First of all, you should verify that mysqld is running by typing #ps aux | grep [m]ysqld. Then there are 2 ways to solve this: one is to fix the socket path in the /etc/my.cnf under [mysqld] and [client] like this

# file: /etc/my.cnf
[mysqld]
… # someother settings
socket=/var/lib/mysql/mysql.sock
[client]
… # some other settings
socket=/var/lib/mysql/mysql.sock

then restart the mysqld server so that the mysql.sock file is generated. The next time you start the client with #mysql -u root, you will magically get it. This fix is permanently but the catch is you will have to shutdown mysqld, the MySQL server itself.
The other way when you want to just login really fast to the mysql server, try this (actually I used this method to obtain the overall solution to the infamous error)

• #find / | grep mysql.sock
  this will do a look up on the entire disk and look for the mysql.sock file. You will get back something like /tmp/mysql.sock. This is the socket file we will need to use.
• #mysql -u root –socket=/tmp/mysql.sock
  this will tell the mysql client to login as root user and use the socket file located in /tmp/mysql.sock

Boom, you should get into the suddenly-loved mysql> prompt. I was so excited when I was able to fix the client and was able to access to the mysql prompt.

I hope this help you all. With this success in installing the MySQL 5.0.27 cluster, I am planning to do a screencast of setting the whole thing up (VMWare is just awesome. It allows me to have 3 CentOS 4.4 servers at the same time, making this installation a breeze)

Coincidentally with the HP scandal, I’m working on an email project, where thousands of emails are sent out every morning to important people. The sender’d like to know the open rate of the emails. The technique used is pretty standard using “web bug”. They implemented a 1×1 transparent gif image whose source url is something like mydomain.com?userID=1234&otherParam=asd. The problem is: the collected open rate is not at all accurate because many of the recipients are using Outlook 2003. To avoid cross-site scripting (XSS) and other possible vulnerabilities, by default, Outlook 2003 (also Thunderbird 1.5) blocks all external images and display them with great skeptical even when the user chooses to unblock the images. As the web bug image is not requested, the collected open rate is way low as most users don’t bother unblock the images (Outlook’s warning text about external images is too blended in so I guess most users don’t even see that feature)

After messing with our own code to keep track of the open rate for the emails, my work buddy and I were set out to use Readnotify.com. We found out about ReadNotify.com mostly through Google. Some sites (wikipdia, for example), said that Readnotify.com was used during the HP’s scandal. I thought that Readnotify.com must be doing some neat tricks to by-pass the strict restriction of Outlook 2003. I was wrong. **Readnotify.com could neither detect if the email has been opened in Outlook 2003 (or Thunderbird 1.5) with the images blocked.** Don’t waste your money on the subscription if your audience are mostly Outlook 2003 users, which happens to be the case of 99% of US corporates. I’m not sure if HP’s Patty had known about this when she probed the emails.

===== Here’s how Readnotify.com do it: =====
When you send an email to your-receipient@somedomain.com.readnotify.com, Readnotify.com will automatically appended some HTML code to the email. This works only for HTML-based emails. For text-only emails, Readnotify.com employs the traditional “read-receipt” mail header which the reciever has to agree to send back a “read receipt” to the sender, which is Readnotify.com, not the actual sender.

With that’s said, below is a copy-paste code segment that Readnotify.com appended to an email