Put your message here! Contact me for more information
 
 







 

Archive for the ‘Ruby on Rails’ Category


 

For a Rails/SQLServer application I’m working on, I had to deal with pagination with custom queries because of the different joins. The mislav-will_paginate plugin works great for MySQL, but for SQL Server, the paginated query generated by the current SQL Server Adapter (I’m using activerecord-sqlserver-adapter-1.0.0.9250) does not work very well. The current implementation is targetted really for SQL Server 2000 and older versions since these versions do not have support for ROW_NUMBER() method. It is a major pain in the butt to do pagination with these databases. With the newer SQL Sever 2005, the job is a bit easier. Microsoft implemented the ROW_NUMBER() method with a convoluted syntax to have better support for pagination, but it is still a drag because of the weird syntax.

Semergence wrote in his blog about patching the SQLServerAdapter to support pagination. Based on his post, I improved ActiveRecord::ConnectionAdapters::SQLServerAdapter::add_limit_offset! to make the query work in a more general way with free-form queries, e.g. queries ran with the paginate_by_sql() method provided by mislav-will_paginate

Include this script in your environment.rb file, or an external file and “require” the file within environment.rb.

  # monkey-patching SQLServerAdapter to support SQL Server 2005-style pagination
  module ActiveRecord
    module ConnectionAdapters
      class SQLServerAdapter
        def add_limit_offset!(sql, options)
          puts sql
          options[:offset] ||= 0
          options_limit = options[:limit] ? "TOP #{options[:limit]}" : ""
          options[:order] ||= if order_by = sql.match(/ORDER BY(.*$)/i)
                                order_by[1]
                              else
                                sql.match('FROM (.+?)\b')[1] + '.id'
                              end
          sql.sub!(/ORDER BY.*$/i, '')
          sql.sub!(/SELECT/i, "SELECT #{options_limit} * FROM ( SELECT ROW_NUMBER() OVER( ORDER BY #{options[:order] } ) AS row_num, ")
          sql << ") AS t WHERE row_num > #{options[:offset]}”
          puts sql
          sql
        end
      end
    end
  end

The method above monkey-patches the SQLServerAdapter by overwriting the add_limit_offset! method.

Here’s a custom query that I used and the transformed result:

Resource.paginate_by_sql([
      %!SELECT  resources.*
        	,skills_count.skill_count
        FROM resources
        	,(
        		SELECT resource_id
        			, COUNT(*) AS skill_count
        		FROM resource_skills
            WHERE meta_skill_id IN (1,2,3,4,5,6,7,8,9,10)
        		GROUP BY resource_id
        	) AS skills_count
        WHERE resources.is_active = ?
          AND resources.id = skills_count.resource_id
        ORDER BY skill_count DESC
      !, true ], :page => page, :per_page => per_page

With :page = 1, :per_page = 2, the resulted SQL is:

SELECT TOP 2 * FROM ( SELECT ROW_NUMBER() OVER( ORDER BY skill_count DESC ) AS row_num, resources.*
 	,skills_count.skill_count
 FROM resources
 	,(
 		SELECT resource_id
 			, COUNT(*) AS skill_count
 		FROM resource_skills
 WHERE meta_skill_id IN (1,2,3,4,5,6,7,8,9,10)
 		GROUP BY resource_id
 	) AS skills_count
 WHERE resources.is_active = 1
 AND resources.id = skills_count.resource_id

 ) AS t WHERE row_num > 0

The will_pagination’s COUNT query is

SELECT COUNT(*) FROM (
 SELECT resources.*
 	,skills_count.skill_count
 FROM resources
 	,(
 		SELECT resource_id
 			, COUNT(*) AS skill_count
 		FROM resource_skills
 WHERE meta_skill_id IN (21,22)
 		GROUP BY resource_id
 	) AS skills_count
 WHERE resources.is_active = 1
 AND resources.id = skills_count.resource_id
 ) AS count_table

The ORDER BY part is automatically removed from the main query (which becomes a sub-select) by the plugin to speed up the query. This in turns sanatizes the sql so that SQL Server doesn’t not complain about nested “ORDER BY” within a sub-select. Neat!

The only catch with the current add_limit_offset! is that it does not support ALIAS-ing, because the aliasing confuses the reqex to parse out the ORDER BY condition in the OVER() part of the query.

For regular find() queries, here’s a sample result

Resource.find(:first)
# original query:  SELECT * FROM resources
# transformed:   SELECT TOP 1 * FROM ( SELECT ROW_NUMBER() OVER( ORDER BY resources.id ) AS row_number, * FROM resources ) AS t WHERE row_num > 0

Hope this helps and cheers!

view comments
 

I wanted to give mechanize a shot for a small prototype I was building.  It’s been a while since I last ran the gem command so of course the first thing it would do was to update itself.  For some reason, it got stuck forever.  It turned out that the latest hpricot gem was trying to build itself from source (! - very strange!) and gem updater got stuck looking for a compatible compiler.

So I downloaded the mechanize gem from Rubyforge instead and ran

gem install mechanize-0.7.7.gem

However, the gem still tried to access the gem index and eventually got stuck somewhere.  The ruby runtime ballooned up to more than 600MB as showed in Task Manager.  I dug up the gem install command and it turns out the gem install can be forced to install without checking the dependencies and without updaing the local index.   So the final gem command is

gem install mechanize-0.7.7.gem –no-ri –no-rdoc -f –no-update-sources

With the –no-update-sources flag, the gem won’t try to go off to http://gems.rubyforge.org and get the latest repository info but expecting there is already a local gem file.

Hope this help someone :)

view comments
 

I just checked out the 280Slides.com, a YCombinator’s funded company. They are developing an online Slideshow/Powerpoint site (yes, another one). The application is pretty slick and it has the Apple’s look and feel to it (both the 2 founds were from Apple). Paul Graham used their app to create his slide at StartupSchool 2008, which at the time, took him 10 minutes to start his slides due to “technical issues”.

What really got me amazed is that they developed a JavaScript UI framework called Objective-J. Apple do a lot of their apps and gadgets in Objective-C (iPhone for one), and since the guys behind 280Slides were from Apple, probably they took the concepts from Objective C and bring it over to JavaScript. Having coded TubeCaption’s caption editor, the Captionizer, from scratch, I understand how much work it takes to do something non-trivial in JavaScript. Programming an online application is totally different than programming a simple “Ajaxy, Web2.0″ page because the amount of work involved. We have hundreds of objects potentially interact with one another and trying to compete for CPU. Without a solid foundation, the application won’t be able to bear the performance and complexity weight.

I’m pretty excited to hear that 280Slides is planning to open-source their framework (probably the Objective J) in the near future. It will be a fresh idea besides the currently established frameworks such as YUI, Ext, Prototype.

Increasingly I see the trend of JavaScript being used as the underlining cross-platform language to build other frameworks and programming languages on top of it. John Resig (from jQuery) recently ported the processing visualizing language to JavaScript. His JS implementation looks AMAZING (check out the parser’s code, what a work of art) and performance-wise the library kicks major ass. Then somebody wrote a Ruby VM in JavaScript (HotRuby) and Ruby code can get executed natively in the browser. A VM written in JavaScript? WOW! Just the thought of Rails *may* work in the browser (hopefully not IE6) makes me feel dreamy. These stuffs are truly innovative and that’s what really push the web technologies forward. And with the new JavaScript engines that promise excellent performance (Webkit’s Squirrelfish, Mozilla’s SpiderMonkey), for sure we will even see MORE of the creative innovations.

PS: don’t forget to check out www.tubecaption.com’s Captionizer, the first timeline-based caption editor.

view comments
 

Recently I had to configure my VPS at Slicehost.com to send out emails for my TubeCaption application. Due to the time constraint of the launch, we decided to outsource the mail delivery to Gmail SMTP server.

Using Gmail SMTP server with ActionMailer

When I first deployed the application, I didn’t configure sendmail correctly (I’m a total stranger to sendmail or postfix) and went ahead with Google’s SMPT server.  Since I am using Google Apps to host the mailboxes, I created an extra account specifically to use for mail delivery. In my development.rb and production.rb file, I have ActionMailer configured as follow:

ActionMailer::Base.delivery_method = :smtp
ActionMailer::Base.smtp_settings = {
:tls => true,
:address => “smtp.gmail.com”,
:port => 587,
:domain => “tubecaption.com”,
:authentication => :plain,
:user_name => “sample-mailer-account@tubecaption.com”,
:password => “the-very-secured-password”
}

Within my ActionMailer handler, I have a function to setup the default params as follow

class Postoffice < ActionMailer::Base
def setup_default_params
@from = “my-email@tubecaption.com”
@headers = “Reply-to my-email@tubecaption.com”
@subject = “TubeCaption.com”
@sent_on = Time.now
@content_type = “text/html”
end
end

Then for each email, before sending, I invoke setup_default_params() to have all the params setup. The site could now sending out emails, albeit slow. ActionMailer had to open a connection to GMail, authenticated, and then started writing the content as a stream of text. The whole process took from 1 to 3, or even 4 seconds some time on my machines, which means the mongrel process was also stuck there waiting.  But the mailing piece worked. However, I came to know the real issue:  all the emails were marked as SPAM or JUNK if they are sent to GMail or Yahoo.

I talked to a friend of mine, who has more knowledge in the emailing world. He suggested the cause of this is because emails sent from my VPS failed the reversed DNS lookup verification. Yahoo mail servers perform a reverse DNS look up to check my domain’s DNS records, while the SMTP server I used was from Gmail. Hence the emails were marked as spam in Yahoo and junk in Gmail due to the DNS check mis-match.

So I decided that it was about time for me to configure sendmail for the box and switch ActionMailer to use sendmail instead.

ActionMailer with local sendmail

Truthfully speaking, I would love to get away as far as I can from configuring sendmail. The 4th edition O’Reilly book on Sendmail is more than 1,000 pages! I only want something that can send out emails… Luckily, there is yum.

# sudo yum install sendmail

Now I switch my ActionMailer config back to use sendmail. Since my sendmail is installed in the default folder, I don’t need to configure ActionMailer any further. In production.rb, I have

ActionMailer::Base.delivery_method = :sendmail

ActionMailer now delivered mails using the local sendmail program.

Using the console, I sent out a test email to my yahoo acount.   Everything works.  Then, I received a notice from my server, telling me I have new mails at /var/spool/mail. Great! I checked the mail log, and saw …

—– The following addresses had permanent fatal errors —–
<nworld3d@yahoo.com>
(reason: 553 Mail from XX.XXX.XXX.XXX not allowed - 5.7.1 [BL23] Connections not accepted from IP addresses on Spamhaus XBL; see http://postmaster.yahoo.com/550-bl23.html [550])

—– Transcript of session follows —–
… while talking to b.mx.mail.yahoo.com.:
<<< 553 Mail from XX.XXX.XXX.XXX not allowed - 5.7.1 [BL23] Connections not accepted from IP addresses on Spamhaus XBL; see http://postmaster.yahoo.com/550-bl23.html [550]
… while talking to d.mx.mail.yahoo.com.:
<<< 553 Mail from XX.XXX.XXX.XXXnot allowed - 5.7.1 [BL23] Connections not accepted from IP addresses on Spamhaus XBL; see http://postmaster.yahoo.com/550-bl23.html [550]
…….
<<< 553 Mail from XX.XXX.XXX.XXX not allowed - 5.7.1 [BL23] Connections not accepted from IP addresses on Spamhaus XBL; see http://postmaster.yahoo.com/550-bl23.html [550]
554 5.0.0 Service unavailable

…….

Final-Recipient: RFC822; nworld3d@yahoo.com
Action: failed
Status: 5.5.0
Diagnostic-Code: SMTP; 553 Mail from XX.XXX.XXX.XXX not allowed - 5.7.1 [BL23] Connections not accepted from IP addresses on Spamhaus XBL; see http://postmaster.yahoo.com/550-bl23.html [550]
Last-Attempt-Date: Tue, 3 Jun 2008 14:02:54 -0400

OUCH! My VPS got black-listed! Somehow the Spamhaus XBL list decided to black list the IP address of the server. Probably because I had port 25 opened in the firewall and somebody has taken advantage of my lack-of-linux-sysadmin-skills to start relaying spams.  CRAP! Yahoo outrightly refused any SMTP connection from my server because it was marked as a spammer-wannabe .  Good thing Yahoo included the  link to their help page and references to Spamhaus XBL.

I went to Spamhaus website and looked up my server’s IP:

The CBL (composite blocking list) was blocking my server.  I went ahead and request a removal from the list.  Afterwards, the CBL list showed

A few minutes later, I checked Spamhaus and it’s not showing the IP listed anymore.  Phew!

It would take a while before this list is propagated through out the internet.  At the moment, I still can’t connect to Yahoo’s SMTP server and my mails is still delivered right into Google’s Junk mailbox.  Hopefully a few hours more and my server would be good to go.

I don’t know if Spamhaus is my friend or foe.  I have a hard time deciding it.  I don’t know what people’s experiences with emailing services in general, but I can tell it’s a pain in the butt, and everything will only get worse from here.  I wonder what it will be like when ipv6 is used…

view comments
 

If you do “gem install capistrano”, the gem package manager will go fetch the latest gem version of capistrano, currently 2.3.0 and installed in the gem repository. In case you want to have multiple version s of capistrano running, here’s how to do it.

To install older gem version of Capistrano
gem install –version 1.4.2 capistrano
(1.4.2 is the latest one in the 1.x branch before the release of 2.0)

To run specifically the 1.4.2 version, use
cap _1.4.2_ *your_tasks_here*

Shortcut
To reduce the typing, you can make an alias in your .bash_profile on Linux to run the 1.4.2 version as cap1 (cap 2.x.x is still running as cap) using

# add this line to .bash_profile
alias cap1=”cap _1.4.2_”

Afterwards, reload the profile with

$ source .bash_profile


Since I am on Windows, what I did was creating a batch file called “cap1.bat” and saved it within my system’s PATH environment. For simplicity’s sake, I save the cap1.bat file inside my C:\Windows\System32 folder

@echo off
cap _1.4.2_ %*

The special wildcard %* will be replaced with your command-line arguments, saving you the typing.

To read more about the Capistrano 1.4.2 version, check out Jamis’s post here.

view comments