Alex Le’s Blog

I was just on heaven this morning. A few days ago, a lady at work gave me a bar of chocolate biscuit. The brand is Choceur. I saved it and didn’t eat it till this morning. I unwrapped it and took a bite. I was in heaven. It was THE best and the smoothest chocolate I have ever eaten. The chocolate just melted in my mouth in a way that I completely lost all my other senses. My entire concentration was on my tongue, where the piece of chocolate was happily melting. Then I started to feel the coffee flavor. Oh my … It was the first time I have ever enjoyed such a masterpiece of chocolate like this. Then it came the crunchiness of the biscuit, contrasting with the smoothness of the chocolate itself. The biscuit is different from the regular “crunchy chocolate bar” made by Hershey’s. The biscuit is not as hard and tough to chew, but hard enough for me to feel and enjoy the various texture of it. And I had to stop myself after the first bite and shouted to myself… “this is amazing!“. At the moment of writing, 10 minutes after I already finished the choco-biscuit (oh no!!!!), I can still feel the cappuccino flavor in my mouth mixing up with the smoothness and the crunchiness. And somewhere up in my head, it’s craving for more of this heaven-sent Choceur choco & bisbuit bar.

It just makes my day so much better. Thank you.
I’m proud to be a chocolate lover.

I’ve been doing quite a bit of reading about High-Availability (HA) web clustering techniques for the past 2 weeks. Thanks to VMWare, now it is possible to create a virtual web server farm with multiple linux instances running concurrently to simulate a cluster setting. The initial result is very heart-warming: I’ve sucessfully installed a mysql database, configure fail-over (thus means high-avaibility) web servers using heartbeat, and name servers running BIND9 to do name lookup. The last bit of the puzzle is now the Linux Virtual Server (LVS) Director for load-balancing. That would be my next experiment.

Here is my initial diagram of my web servers farm and the IP/server name assignments

With this design, I tried to eliminate the single point of failure by implementing redundant, ready to fail-over servers for critical sections of the network, for example, the DNS, the load balancers for incoming traffic, and the load balancers for the database farm.

Basically, incoming traffic will pass through A, the load balancer. The load balancer A will spread out the load to the web servers C in the farm using LVS-DR method (direct-routing). There are also 2 kinds of web servers: one is supposed to be the beefy, powerful server with fast CPU to run the web applications, and the other is the media server which doesn’t need good CPU but requires fast HDD (SCSI) and lots of RAM. The applications server will do the number crunching, churning out pages as fast as they can while the media servers will provide all the images, CSS, and javascript files. Of course since I am using VMWare, it virtually costs me $0.00 to add a new scsi drive to the VM machine. Great!
The name servers B running Bind9 are located centrally to help with the name resolutions. Of course DNS is critical so we need to have a certain level of HA. Hearbeat will make sure the DNS is always up and ready.

Meanwhile, for the database farm E, which is on a separate network (supposed to be high-speed, low latency with very expensive switches) a pair of load balancers is needed to spread out the “read” (SELECT) load. I’m not quite sure how to implement the “write” (UPDATE/ DELETE, ALTER TABLE, etc.) DB servers yet, but I’m sure that we can improvise along the way. Again, Heartbeat will be implemented to keep the database load balancers up and happy. Our database farm will consist of 2 network storage nodes to store data and 2 “API” nodes to do the database heavy lifting. A fifth server is used to be the management node to manage (add, delete, or update) the database servers.

Finally, (and not shown in the above diagram as I just realize that I am missing something), a monitoring server running Nagios is implemented to do health-monitoring and network management. With the current design, all part of the network can be scaled independently: if more web servers are needed, we add new boxes to section C. If we need more database storage nodes, we can quickly add a new NDB node to the MySQL database cluster F. The bottle neck will now be our gateway, the load balancers in A. However, since it’s been confirmed (see the linux-ha.org site) that a decent load balancer can easily handle the amount traffic to saturate a 100Mbps connection, I would say for a small/ medium business settings, this is more than enough.

If you are asking why I am writing all of this down. I am doing this because I will begin to construct this web farm using VMWare with CentOS 4.4. The post and the diagram will serve as a guideline for this particular project. Moreover, I intend to do screencast of the entire process of setting up this web farm. Yes, I’d like to commoditize the knowledge of building Linux cluster using off-the-shelf tools. It’s a noble goal, I know, but I’m doing it for myself first so you don’t have to thank me now.
Now off to work I go. Keep on checking back alexle.net for more information about Web clustering. “This is Alex Le doing it so you don’t have to.” (yeah, I copy Ze frank’s line, so excuse me for the plagiarism.

On slashdot today…

Last night we crossed over 16,777,216 comments in the database. The wise amongst you might note that this number is 2^24, or in MySQLese an unsigned mediumint. Unfortunately, like 5 years ago we changed our primary keys in the comment table to unsigned int (32 bits, or 4.1 billion) but neglected to change the index that handles parents. We’re awesome! Fixing is a simple ALTER TABLE statement… but on a table that is 16 million rows long, our system will take 3+ hours to do it, during which time there can be no posting. So today, we’re disabling threading and will enable it again later tonight. Sorry for the inconvenience. We shall flog ourselves appropriately.

This is kinda amusing.  After all, we are all human.

Last nite, I was surfing around and found this page at the ThePlanet.com forum. Someone has configured a subdomain to proxy to the Plesk admin page. This way, in stead of accessing Plesk admin page via port 8443 (which I forget half of the time), I can access the page via https//plesk.mydomain.com (still encrypted with SSL for your privacy and security). In case I am behind a firewall that blocks most incoming/ outgoing ports, I still can access my plesk control panel page via the standard SSL port of 443.

From the forum thread (see the link above), the setup is pretty straightforward, but for some reasons, a few people could not configure the subdomain to act as a proxy to the Plesk control panel. The reason is there is a bug in how Plesk 8.0 writes the domain/ subdomain configuration files for Apache. And as there were a few people asking for a more detailed step-by-step tutorial, here is my take on it.

Let’s first go over the setup and configuration, then we can talk about the bug in Plesk. I assume that you have root access to your box because the Plesk manual explicitly states that you have to be root to create the vhost_ssl.conf file, which is the file we need to create to turn the subdomain into the proxy to the Plesk control panel.

The setup

• Create a subdomain in Plesk. Remeber to enable SSL support, PHP support, and CGI support. By checking these 2 options, we are forcing Plesk to add an “include” directive to the configuration file of the domain. We discuss more on this later. I created a subdomain called “plesk” so that I can use it like https://plesk.alexle.net/ to access to the Plesk CP. From now on, let’s use “plesk” as our subdomain and alexle.net as our main domain.
• Next, SSH to your box as root, then cd to the configurations folder of the newly created subdomain:
  #cd /var/www/vhosts/alexle.net/subdomains/plesk/conf/
• Create a vhost_ssl.conf file ins this conf folder using your favorite text editor. The content of this file should be
  

  SSLProxyEngine on
  ProxyRequests off
  ProxyPass / https://www.alexle.net:8443/
  ProxyPassReverse / https://www.alexle.net:8443/

  What we are doing is to turn on the SSLProxyEngine for this particular subdomain. In order for this to work, we have to turn the ProxyRequests off. Next, we tell Apache to pass all traffics (or requests) from the root / access of the subdomain to the destination URL (which means all future requests from https://plesk.alexle.net/ will be “ProxyPass“ed to https://www.alexle.net:8443/) . Then with ProxyPassReverse, we tell Apache to redirect the response from https://www.alexle.net:8443/ back to /, our “https://plesk.alexle.net”. (By the way, you will have to use your own domain/ subdomain instead www.alexle.net in the above configuration)

• Finally, for Apache to pickup the the new configuration file, we need to restart the httpd service
  #/etc/init.d/httpd restart
• Surf to htps://plesk.alexle.net/. Humh, it doesn’t work yet? Here is the Plesk bug.

The Bug

The way Plesk generates and stores configurration files are covered in the Plesk Manual Page. Basically there’s a master Apache config file at /etc/httpd/conf. This master Apache configuration file will include a whole bunch of other specific configuration files for each domains (located in /var/www/vhosts/domain_name/conf/httpd.include) and subdomains (/var/www/vhosts/domain_name/subdomains/your_subdomains/conf/vhost.conf or vhost_ssl.conf). Phew.

Remeber when we created the subdomain, we have selected PHP support and CGI support. By doing this, we have forced Plesk to write an “include” directive in the configuration file of the main domain to include the configuration file of the subdomain. To clarify, if you now open up the file httpd.include (in my case, at /var/www/vhosts/alexle.net/conf/httpd.include), find the VirtualHost section for your newly created subdomain, you will find a line similar to this

#file: /var/www/vhosts/alexle.net/conf/httpd.include
Include /var/www/vhosts/alexle.net/subdomains/plesk/conf/vhost.conf

However, there are 2 VirtualHost sections for the plesk subdomain: one is for port 80, and the other is for port 443, SSL. Moreover, both sections include the same vhost.conf file. This is wrong. Based on the Plesk’s Manual, the VirtualHost section for the SSL at port 443 should include the vhost_ssl.conf instead of the vhost.conf. So that’s the Plesk 8.0 bug. I’ve tried a couple times and was able to reproduce the Include bug.

The fix

Now we know why our configuration file for SSL access via the subdomain is not picked up. We need to change the Include directive of the VirtualHost SSL section for the subdomain to use the vhost_ssl.conf file instead. Then restart apache (#/etc/init.d/httpd restart) and everything should works fine.

The catch

We are modifying the main domain’s httpd.include file, which Plesk will overwrite everytime we make changes to this particular domain. Once that happens, you will have to re-modify the httpd.include file so that the correct vhost_ssl.conf file is used.

I hope that this short article helps you work and understand Plesk better. Thanks for the thread from ThePlanet.com forum to inspire me for this article. Comments are welcome as always.

I spent a few hours yesterday and today working on a small AJAX page with multiple combo boxes that are populated from the database. I use Firefox 99% of the time and I utilize the excellent YUI library for cross-browser compability. I almost done with the page (I’m pretty proud of it too) so I decided to show it to my work buddy. On IE (both 6 and 7), my beatiful double combo boxes just failed miserably. A part of what I was doing was to populate a double combo box with dynamic data from the database. I created an empty SELECT tag (dynamically via JS), assigned its an ID, made an AJAX request, then populated the empty SELECT tag via the innerHTML with the AJAX response, which is a list of OPTION elements. On Firefox, the double combo works so well that I thought it was time wrap up the project. However, on IE, the combo box just mysteriously failed to get populated.

First of all, I tried to eliminiate the possibility of my AJAX requests got messed up somewhere. I traced through the flow of the functions and put alert() to assert the responses of the AJAX requests as I was afraid that the IE 7 I was using to test the page is not compatible with the YUI’s Connection Manager.  This was also the first time I ran IE7 after deciding to install it the other day.  IE7’s AJAX feature turned out to work just fine, however, after several checkpoints using alert(), I finally discovered a bug in how IE handle the innerHTML property of select item: If the select item’s innerHTML propety is set programmatically, IE will truncate the first opening option tag.

In my case, what I got back from the AJAX request was a list of OPTION elements. When I do the assignment to the innerHTML of the empty SELECT element, the OPTION list’s HTML code got truncated, specfically the first opening OPTION tag. Hence, the html code I used to populate the select list was malformed. And thanks to IE’s impotency, I could not see the generated HTML code of the page (which in Firefox, the Web Developer toolbar works nicely) so I could not even debug HTML source.

I did some Google lookups on IE select element’s innerHTML bug and found out that it is a confirmed bug by Microsoft, moreover, it’s been confirmed since IE 5 (Microsoft’s link). That was so 1999 and yet the bug still lives on until now with IE7. Humh, I totally understand that fixing bugs is being taken care seriously at Microsoft. But the innerHTML bug should be trivial to fix (no security risks that I can think of) and it’s been there for years without anyone from Microsoft bothering to do anything. It’s just ridiculous.

Oh there’s more, the Microsoft Bug page (the above link) shows that the bug report is at revision 2.0. Yeah, I thought “bug 1.0″ is more than enough but apparently I’m wrong because it’s Microsoft.

-1 to microsoft for the slowness of *TRIVIAL* bugs fixing.

-1 to micrsoft for the lack of developer’s handytools. Study Firefox and how the community are doing it, please.  (I do aware of the Developers Toolbar for IE, but it is not nearly as useful as the Firefox Web Developers Toolbar)