Alex Le’s Blog

My blog’s been up and down (mostly down) for the past few days. The reason was that I was running out of files quota. I’m still hosting my site on 1and1 despite the fact that 1and1 and I we had some conflicts. 1and1 imposes a hard-limit of the number of files you can have on your account( ~260k). I didn’t move the blog to my dedicated servers because I know I won’t have enough time to maintain the servers myself (make sure DNS doesn’t go down, mail server up and spam-free, etc.). I don’t want to be a fulltime sys-admin just to make sure my blog is up. So as much as I want to be independent and in total control, I’d rather leave my site on 1and1’s server and pay them $10 to do so.

Since with 1and1 you can host multiple domains, I host a few other sites together with alexle.net under 1 single account. Suddenly there were a large number of files on my hosting account and I didn’t know what was happening. My ~260,000 files quota was quickly used up. I SSH’ed in to the account and did some searches (just do find ./ | wc for different suspicious site folders). It turned out that the joomla portal for www.warsofearth.com began to generate a huge amount of cache files. I originally had Page Cache on for WarsOfEarth to speed up the site (sometimes 1and1 server is not the fastest one. You can tell, alexle.net’s speed is terrible without cache) and thus for each hit to the site, a cache file is generated. My guess is that the site was crawled by some bots and the OpenSEF link component I used was generating cyclic links. I was aware of the OpenSEF issue for WarsOfEarth.com portal but just never bothered to fix it. And once the bot crawled the site, it kept on indexing and following the cyclic links, thus generating the huge amount of cache file.

Anyway, I rm’ed all the cache files for warsofearth.com and was able to reduce the amount of files on the server to roughly 40k, instead of more than 260,000. For now, the problem is fixed.

Sorry for the inconvenience that may have caused to you.

{{http://alexle.net/wp-content/uploads/2007/06/rails.thumbnail.png }}Working on Wars of Earth by myself, I run into several situations when I’d like to move migration files around to get them executed in a particular order. The trick is simple: just rename the sequence of the migration files to get them to be executed in the order you want.

Besides sloppy schema design (hey, I’m trying to be agile here - fix it as you go), an example of a situation when you want to move migration files around: table A has a foreign key to table B. Migration file for A is at 005, while that of B is at 010. If you try to put in the FORIEGN KEY constraint in A migration file after the table creation, migration will fail since table B hasn’t been created yet, not until migration #010. You can add another migration file just to add the FORIEGN KEY constraint from A to B, say at #020. But then you will have fragmented migration code all over the place. For production environment, this is the only solid and proven way to perform database changes. However, during development, you have the luxury to drop and recreate the entire database from scratch, it’s just a lot more convenient and makes more sense to be able to move migration files around. Ideally, you just need to run migration for B first, then you can run migration for A, which also execute the SQL to add in the FOREIGN KEY constraint properly.

Until now, you have to it manually. Imagine if you have 20 migration files in between, from 5 to 25 (like what I have), you will probably have to renumber 20 migration files. And if your project is in SVN, it would be more time-consuming and error-prone. You can either do “svn rename”, or just rename in the shell, then deleted the previous migration files and added the newly-named ones back into the repository. But what would happen if you yawn for a second and misnumber a file at #6, so you have two 006 migration files. Oops.

Worry no more, here comes **numergrate** utility to the rescue. In short, numergrate is “to numerate migration files” (or to renumber migration files)

What you need is to drop in the numergrate script (download below) inside your Rails’ script folder and you are ready to go. With this utility, you can execute at the root of the Rails project

ruby script/numergrate 5 before 25
to (test) move 005 to position 024 instead.

or

ruby script/numergrate 10 after 2 --svn
to move 010 before 002 and also rename the files in the Subversion repository.

I also took the opportunity to integrate the utility with Subversion in case your migration files are in a repository. There’s no easy way for CVS so I didn’t implement it. However, implementing renaming mechanism for other SCM should be straight forward if you know their command line renaming tool.

I hope this comes handy for you.

==== Supported tools ====

* {{http://alexle.net/wp-content/uploads/2007/06/subversion_logo.thumbnail.jpg}} Subversion (renaming files in repository)
* {{http://alexle.net/wp-content/uploads/2007/06/utilities-terminal.png}} Shell (just like renaming manually)

==== Script ====

#!/usr/bin/env ruby
# (c) 2007 Alex Le
# www.alexle.net - nworld3d@yahoo.com
# This script is developed for www.warsofearth.com. (shameless self-promotion)
# Released under the same license as Ruby
# Disclaimer: The author is not responsible for any incorrect results from running
# the script. Use it at your own risk.
#
# USAGE:
# ------
# This script is used to re-number the migration files into the desired sequence
# In development, this comes handy as you can organize migration files into logical groups
# by running a simple command line utility instead of manually renaming the filenames.
#
# numergration = numerate migration files
#
# INSTALL
# -------
# Copy numergration into script/ folder of your Rails application.
# On linux system you may have to chmod the script to be executable (+x)
#
# HOW TO RUN
# ----------
# At your Rails application root, run:
# On Windows:
# > ruby script/numergrate before|after [mode]
# On Linux: users can just run the script without calling the ruby executable since
# there’s a #! on top, provided that you set the permission correctly. (chmod to +x)
# $ script/numergrate before|after [mode]
#
# The [mode] options are
# –test Default mode to test the result before you run
# with –shell or –svn
#
# –shell Renaming file as you would do manually in the shell
#
# –svn Integrate with Subversion by executing `svn rename` on each file
# This option alters your working copy so please be extra careful.
# (a.k.a. use it at your own risk)”
#
# Example:
# ——–
# a. Move Migration file 50 to position 3, hence shifting migration files
# from 3 to 49 to the right by 1.
# > ruby script/numergrate 50 before 3
# (the above will just execute with the –test default option)
#
# Or to actually rename the files,
# > ruby script/numergrate 50 before 3 –shell (or –svn)
#
# Or you can even run
# > ruby script/numergrate 50 after 3
# to put migration file 50 after migration file 3 (shifting migration
# file 4 to 49 to the right by 1)
#
# TODO:
# —-
# 1. Better sequence handling. Currently it’s default to 000 for the
# sequence series. However, there can be potentially a lot migration files.
# (more than 999 files). The solution is to find the max sequence and
# use that as the series template
# 2. Better SVN integration.
# a. Do some checking to see if the svn client exists before running.
# Otherwise throw an error
# 3. Better sequence checking. Currently it doesn’t check for input
# range so we can have “index out of bound” errors.
#

files.each{ |file|
if file.is_changed?
if mode == “–shell”
puts ” rename ” << "db/migrate/" << file.old_name
puts " to " << "db/migrate/" << file.new_name
cp("db/migrate/" << file.old_name, "db/migrate/" << file.new_name )
rm("db/migrate/" << file.old_name)
elsif mode.downcase == "--svn"
#puts "executing svn commmand here"
puts " svn rename " << "db/migrate/" << file.old_name
puts " to " << "db/migrate/" << file.new_name
system 'svn rename --force db/migrate/' << file.old_name << " db/migrate/" << file.new_name
elsif mode.downcase == "--test"
puts " [TEST] rename " << "db/migrate/" << file.old_name
puts " to " << "db/migrate/" << file.new_name
end
end
}


==== Download ====
* To download, please click here: [[http://alexle.net/wp-content/uploads/2007/06/numergrate.zip|numergrate]] (version 1.0, 06/17/2007)

==== Note ====
* Moving migration files around can do serious damage to your database. Please be careful. This comes extremely dangerous if you are working with other people on the same repository. Please be smart about it. Use it at your own rick.
* Comments and suggestions are very welcome.
* Support me at [[http://www.warsofearth.com|Wars of Earth]] if you can. Another shameless self-promotion.

{{http://alexle.net/wp-content/uploads/2007/06/rails.png }}It’s been a week since I started messing with Ruby on Rails seriously. My current Rails project is rebuilding Wars of Earth, the game that my friend Giao and I started almost a year and a half ago. Since Rails is so flexible and powerful, within a span of a few days (like 5 days), I was able to achieve as much as 2 weeks worth of work compared to the past. Before, as Giao is much better at programming than I am, I asked him do the framework design in PHP (we used Pear’s DB_DataObject, Smarty, and home-grown controller for a MVC architecture.) I did the design (game designs, coming up with ideas, some coding) instead as well as preparing the site in Joomla (check it out [[http://www.warsofearth.com|here]]). Nonetheless, I was getting ready to graduate, and my buddy had to take on freelance work to make some money, I decided to put Wars of Earth aside and focus on what mattered most at the time (mainly job and money). As I have more free time on my own now, I am restarting the Woe project but this time, I am rebuilding it from scratch using Rails. And I don’t think I would go back to PHP for any web-based project if I don’t have to.

The major bottle-neck between Giao and I, as I would boldly claim for any other projects that involved geographically distributed team, was the communications. Giao is still back in Vietnam, and I’m in Chicago. I’m 12 hours behind him. Whatever we needed to do, we put in TOTO tasks and message in our Basecamp site. At the time I haven’t thought too thoroughly about the story and the game play, so we just kept on moving until we hit a problem, then we would sit down and talk. But words can only describe so much. Sometimes what I tried to get across, or what my buddy tried to tell me, just failed to get both of us on the same page. Giao would implemented something and it was not the same as I thought, so we just kept on building, discussing, and sadly, compromising. Moreover, I constantly having more ideas for the project, creating our eternal scope-creep: I just tried to add many features right at once (I was trying to get it right the first time, but of course it was almost impossible)

Now with Rails, things are much different. Rails are so flexible and powerful that what I only need to put down is mostly business logic, not low-level, database accessing code. The syntax of Ruby is much superior to that of Perl (I hated Perl with passion, and still do) and Rails is plainly a joy to use. Suddenly I realize the meaning of the [[http://www.loudthinking.com|blog]] of [[http://www.loudthinking.com/about/|David H. Hasson]], Rails’ creator, “**Loud Thinking**.” With Ruby, you can actually __think out loud__ through your code. Since Ruby’s syntax is so flexible, it allows the user to write the code in an unheard-of conversational way. I can almost read the code as I would read a novel (since reading code requires some imagination and the code is no longer boring in Ruby, so I can’t compare the program with a textbook - that would be too boring don’t you think?). I don’t have to restrain myself to the curly braces of C#/Java/PHP style, instead I can tell the Model to fetch all the record then smash the id’s in to an array with some logic in one line. Something like CharacteUnit.get_all_units().collect{|unit| unit.id unless unit.is_inactive} and of course you can add a lot more to that as you can stack as many operators/methods as possible. Pure programming joy.

One interesting part about Rails is the testing tool: unit test and functional test. I can’t wait until I setup the first functional test for the application to simulate a web user. And of course the best part is: everything comes in an elegant, well-thought, scalability-proven web framework that took you 10 minutes to get started. Suddenly the barrier to entry to mid/ large-scale web projects is dropped significantly (it’s __free__ now, only costs you your opportunity cost). In stead of just WYSIWYG, it’s now **What-You-Think-Is-What-You-Can-Get**, **WYTIWYCG** (weet-tee-wic). Imagination goes wild, how awesome.

Whomever I talk to nowadays, I can’t stop talking non-stop about Ruby on Rails. I feel good sharing it to anyone I know so they can start feeling the same way as I do, as sharing is caring.

I feel good. Do you?