Alex Le’s Blog

My blog’s been up and down (mostly down) for the past few days. The reason was that I was running out of files quota. I’m still hosting my site on 1and1 despite the fact that 1and1 and I we had some conflicts. 1and1 imposes a hard-limit of the number of files you can have on your account( ~260k). I didn’t move the blog to my dedicated servers because I know I won’t have enough time to maintain the servers myself (make sure DNS doesn’t go down, mail server up and spam-free, etc.). I don’t want to be a fulltime sys-admin just to make sure my blog is up. So as much as I want to be independent and in total control, I’d rather leave my site on 1and1’s server and pay them $10 to do so.

Since with 1and1 you can host multiple domains, I host a few other sites together with alexle.net under 1 single account. Suddenly there were a large number of files on my hosting account and I didn’t know what was happening. My ~260,000 files quota was quickly used up. I SSH’ed in to the account and did some searches (just do find ./ | wc for different suspicious site folders). It turned out that the joomla portal for www.warsofearth.com began to generate a huge amount of cache files. I originally had Page Cache on for WarsOfEarth to speed up the site (sometimes 1and1 server is not the fastest one. You can tell, alexle.net’s speed is terrible without cache) and thus for each hit to the site, a cache file is generated. My guess is that the site was crawled by some bots and the OpenSEF link component I used was generating cyclic links. I was aware of the OpenSEF issue for WarsOfEarth.com portal but just never bothered to fix it. And once the bot crawled the site, it kept on indexing and following the cyclic links, thus generating the huge amount of cache file.

Anyway, I rm’ed all the cache files for warsofearth.com and was able to reduce the amount of files on the server to roughly 40k, instead of more than 260,000. For now, the problem is fixed.

Sorry for the inconvenience that may have caused to you.

Until recently I have decided to setup a local SVN repository on my computer, which runs Windows XP pro. Setting up SVN with Apache is the best way to access a SVN repository but the thing is, my computer has also IIS installed for ASP.NET development. Normally I have XAMPP installed and have to switch start/ stop Apache and IIS manually whenever I need to do something in either ASP.NET or PHP. To clarify, I do have PHP 5.x setup for IIS but Just didn’t bother to fix the session file permission issues of IIS; I’d rather just switch to the working Apache installation instead using the XAMPP control panel.

I started by installing the [[http://svn1clicksetup.tigris.org/|SVN1clickSetup]]. The installation was painless, however, I needed to restart the service to change some configurations. To register svnserve.exe to run as a serice again, I used this full command:


I:\Program Files\Subversion\bin\svnservice -install -d --listen-port 3690 --listen-host 0.0.0.0 -r K:\svnrepos


The -d param enables svnserve to run as a daemon. We specify the list-host of SVNServe.exe to be 0.0.0.0, which means it will bind to any IP available to the system. The -r specifies which folder to serve as the SVN repository. What left was opening up the 3690 port on the router and have the requests for the SVN service on this port routed to my box. Done!

* If you run `svnservice -debug` and see the `RegQueryValueEx failed with error 0, type 1`, it means that SVNService has successfully registered the SVNServe to run as a service. In fact, `RegQueryValueEx failed with error 0` is not an error or a failure.

* In case you have already run the above svnservice to register the SVNserve as a service but still cannot access the repository, you may have to start the SVNService manually by going through Start > Run > services.msc. Find the service called SVNService and start it. Also switch its Startup Type to Automatic so that SVNService is started on start up.

Cheers!

For the past few days I’ve been working on the cluster project again, after a month of suspension. Finally today I was able to configure a (almost) working load balancer with 3 web servers (1 is the loadbalancer itself, and 2 other dedicated web servers, web01 and web02). I’m in the process of writing down the scripts which I will follow later on to do the screencast. At this stage, what I have is half of the whole picture of a web cluster:

• Failover loadbalancer server pair, running Heartbeat and ldirectord
• A small cluster of 3 load-balanced web servers, using weighted round robin
• A pair of Nameserver to provide name-lookup service for the entire cluster, no more “ping 10.10.10.10.101.101.10.12“!

What is missing is the loadbalanced MySQL cluster. I have been able to get a cluster of 3 MySQL boxes working, but now I’d like to add the ipvsadm and ldirectord to provide load-balancing feature so that the cluster can be scaled up easier.

I am pretty proud of myself for learning all of these amazing technologies within a relatively short period of time. Especially for the DNS, I have learned so much about the networking aspect of a network of computer. The two books, “DNS & BIND” and “BIND Cookbook”, both by O’Reilly, are amazing reference source. I particularly like the “DNS & BIND” book for its in depth coverage on the concept of how nameservers work. Without that fundamental knowledge, it’s hard to build such a working cluster with growing demands.

Here is the revised virtual Linux cluster diagram. So slowly but surely my cluster project is reaching its destination: a complete virtualized web cluster running on VMWare. And once this is done, I think I begin to learn how to develop cluster-ready applications.

I’ve been doing quite a bit of reading about High-Availability (HA) web clustering techniques for the past 2 weeks. Thanks to VMWare, now it is possible to create a virtual web server farm with multiple linux instances running concurrently to simulate a cluster setting. The initial result is very heart-warming: I’ve sucessfully installed a mysql database, configure fail-over (thus means high-avaibility) web servers using heartbeat, and name servers running BIND9 to do name lookup. The last bit of the puzzle is now the Linux Virtual Server (LVS) Director for load-balancing. That would be my next experiment.

Here is my initial diagram of my web servers farm and the IP/server name assignments

With this design, I tried to eliminate the single point of failure by implementing redundant, ready to fail-over servers for critical sections of the network, for example, the DNS, the load balancers for incoming traffic, and the load balancers for the database farm.

Basically, incoming traffic will pass through A, the load balancer. The load balancer A will spread out the load to the web servers C in the farm using LVS-DR method (direct-routing). There are also 2 kinds of web servers: one is supposed to be the beefy, powerful server with fast CPU to run the web applications, and the other is the media server which doesn’t need good CPU but requires fast HDD (SCSI) and lots of RAM. The applications server will do the number crunching, churning out pages as fast as they can while the media servers will provide all the images, CSS, and javascript files. Of course since I am using VMWare, it virtually costs me $0.00 to add a new scsi drive to the VM machine. Great!
The name servers B running Bind9 are located centrally to help with the name resolutions. Of course DNS is critical so we need to have a certain level of HA. Hearbeat will make sure the DNS is always up and ready.

Meanwhile, for the database farm E, which is on a separate network (supposed to be high-speed, low latency with very expensive switches) a pair of load balancers is needed to spread out the “read” (SELECT) load. I’m not quite sure how to implement the “write” (UPDATE/ DELETE, ALTER TABLE, etc.) DB servers yet, but I’m sure that we can improvise along the way. Again, Heartbeat will be implemented to keep the database load balancers up and happy. Our database farm will consist of 2 network storage nodes to store data and 2 “API” nodes to do the database heavy lifting. A fifth server is used to be the management node to manage (add, delete, or update) the database servers.

Finally, (and not shown in the above diagram as I just realize that I am missing something), a monitoring server running Nagios is implemented to do health-monitoring and network management. With the current design, all part of the network can be scaled independently: if more web servers are needed, we add new boxes to section C. If we need more database storage nodes, we can quickly add a new NDB node to the MySQL database cluster F. The bottle neck will now be our gateway, the load balancers in A. However, since it’s been confirmed (see the linux-ha.org site) that a decent load balancer can easily handle the amount traffic to saturate a 100Mbps connection, I would say for a small/ medium business settings, this is more than enough.

If you are asking why I am writing all of this down. I am doing this because I will begin to construct this web farm using VMWare with CentOS 4.4. The post and the diagram will serve as a guideline for this particular project. Moreover, I intend to do screencast of the entire process of setting up this web farm. Yes, I’d like to commoditize the knowledge of building Linux cluster using off-the-shelf tools. It’s a noble goal, I know, but I’m doing it for myself first so you don’t have to thank me now.
Now off to work I go. Keep on checking back alexle.net for more information about Web clustering. “This is Alex Le doing it so you don’t have to.” (yeah, I copy Ze frank’s line, so excuse me for the plagiarism.

Last nite, I was surfing around and found this page at the ThePlanet.com forum. Someone has configured a subdomain to proxy to the Plesk admin page. This way, in stead of accessing Plesk admin page via port 8443 (which I forget half of the time), I can access the page via https//plesk.mydomain.com (still encrypted with SSL for your privacy and security). In case I am behind a firewall that blocks most incoming/ outgoing ports, I still can access my plesk control panel page via the standard SSL port of 443.

From the forum thread (see the link above), the setup is pretty straightforward, but for some reasons, a few people could not configure the subdomain to act as a proxy to the Plesk control panel. The reason is there is a bug in how Plesk 8.0 writes the domain/ subdomain configuration files for Apache. And as there were a few people asking for a more detailed step-by-step tutorial, here is my take on it.

Let’s first go over the setup and configuration, then we can talk about the bug in Plesk. I assume that you have root access to your box because the Plesk manual explicitly states that you have to be root to create the vhost_ssl.conf file, which is the file we need to create to turn the subdomain into the proxy to the Plesk control panel.

The setup

• Create a subdomain in Plesk. Remeber to enable SSL support, PHP support, and CGI support. By checking these 2 options, we are forcing Plesk to add an “include” directive to the configuration file of the domain. We discuss more on this later. I created a subdomain called “plesk” so that I can use it like https://plesk.alexle.net/ to access to the Plesk CP. From now on, let’s use “plesk” as our subdomain and alexle.net as our main domain.
• Next, SSH to your box as root, then cd to the configurations folder of the newly created subdomain:
  #cd /var/www/vhosts/alexle.net/subdomains/plesk/conf/
• Create a vhost_ssl.conf file ins this conf folder using your favorite text editor. The content of this file should be
  

  SSLProxyEngine on
  ProxyRequests off
  ProxyPass / https://www.alexle.net:8443/
  ProxyPassReverse / https://www.alexle.net:8443/

  What we are doing is to turn on the SSLProxyEngine for this particular subdomain. In order for this to work, we have to turn the ProxyRequests off. Next, we tell Apache to pass all traffics (or requests) from the root / access of the subdomain to the destination URL (which means all future requests from https://plesk.alexle.net/ will be “ProxyPass“ed to https://www.alexle.net:8443/) . Then with ProxyPassReverse, we tell Apache to redirect the response from https://www.alexle.net:8443/ back to /, our “https://plesk.alexle.net”. (By the way, you will have to use your own domain/ subdomain instead www.alexle.net in the above configuration)

• Finally, for Apache to pickup the the new configuration file, we need to restart the httpd service
  #/etc/init.d/httpd restart
• Surf to htps://plesk.alexle.net/. Humh, it doesn’t work yet? Here is the Plesk bug.

The Bug

The way Plesk generates and stores configurration files are covered in the Plesk Manual Page. Basically there’s a master Apache config file at /etc/httpd/conf. This master Apache configuration file will include a whole bunch of other specific configuration files for each domains (located in /var/www/vhosts/domain_name/conf/httpd.include) and subdomains (/var/www/vhosts/domain_name/subdomains/your_subdomains/conf/vhost.conf or vhost_ssl.conf). Phew.

Remeber when we created the subdomain, we have selected PHP support and CGI support. By doing this, we have forced Plesk to write an “include” directive in the configuration file of the main domain to include the configuration file of the subdomain. To clarify, if you now open up the file httpd.include (in my case, at /var/www/vhosts/alexle.net/conf/httpd.include), find the VirtualHost section for your newly created subdomain, you will find a line similar to this

#file: /var/www/vhosts/alexle.net/conf/httpd.include
Include /var/www/vhosts/alexle.net/subdomains/plesk/conf/vhost.conf

However, there are 2 VirtualHost sections for the plesk subdomain: one is for port 80, and the other is for port 443, SSL. Moreover, both sections include the same vhost.conf file. This is wrong. Based on the Plesk’s Manual, the VirtualHost section for the SSL at port 443 should include the vhost_ssl.conf instead of the vhost.conf. So that’s the Plesk 8.0 bug. I’ve tried a couple times and was able to reproduce the Include bug.

The fix

Now we know why our configuration file for SSL access via the subdomain is not picked up. We need to change the Include directive of the VirtualHost SSL section for the subdomain to use the vhost_ssl.conf file instead. Then restart apache (#/etc/init.d/httpd restart) and everything should works fine.

The catch

We are modifying the main domain’s httpd.include file, which Plesk will overwrite everytime we make changes to this particular domain. Once that happens, you will have to re-modify the httpd.include file so that the correct vhost_ssl.conf file is used.

I hope that this short article helps you work and understand Plesk better. Thanks for the thread from ThePlanet.com forum to inspire me for this article. Comments are welcome as always.