Alex Le’s Blog

Recently I had to configure my VPS at Slicehost.com to send out emails for my TubeCaption application. Due to the time constraint of the launch, we decided to outsource the mail delivery to Gmail SMTP server.

Using Gmail SMTP server with ActionMailer

When I first deployed the application, I didn’t configure sendmail correctly (I’m a total stranger to sendmail or postfix) and went ahead with Google’s SMPT server.  Since I am using Google Apps to host the mailboxes, I created an extra account specifically to use for mail delivery. In my development.rb and production.rb file, I have ActionMailer configured as follow:

ActionMailer::Base.delivery_method = :smtp
ActionMailer::Base.smtp_settings = {
:tls => true,
:address => “smtp.gmail.com”,
:port => 587,
:domain => “tubecaption.com”,
:authentication => :plain,
:user_name => “sample-mailer-account@tubecaption.com”,
:password => “the-very-secured-password”
}

Within my ActionMailer handler, I have a function to setup the default params as follow

class Postoffice < ActionMailer::Base
def setup_default_params
@from = “my-email@tubecaption.com”
@headers = “Reply-to my-email@tubecaption.com”
@subject = “TubeCaption.com”
@sent_on = Time.now
@content_type = “text/html”
end
end

Then for each email, before sending, I invoke setup_default_params() to have all the params setup. The site could now sending out emails, albeit slow. ActionMailer had to open a connection to GMail, authenticated, and then started writing the content as a stream of text. The whole process took from 1 to 3, or even 4 seconds some time on my machines, which means the mongrel process was also stuck there waiting.  But the mailing piece worked. However, I came to know the real issue:  all the emails were marked as SPAM or JUNK if they are sent to GMail or Yahoo.

I talked to a friend of mine, who has more knowledge in the emailing world. He suggested the cause of this is because emails sent from my VPS failed the reversed DNS lookup verification. Yahoo mail servers perform a reverse DNS look up to check my domain’s DNS records, while the SMTP server I used was from Gmail. Hence the emails were marked as spam in Yahoo and junk in Gmail due to the DNS check mis-match.

So I decided that it was about time for me to configure sendmail for the box and switch ActionMailer to use sendmail instead.

ActionMailer with local sendmail

Truthfully speaking, I would love to get away as far as I can from configuring sendmail. The 4th edition O’Reilly book on Sendmail is more than 1,000 pages! I only want something that can send out emails… Luckily, there is yum.

# sudo yum install sendmail

Now I switch my ActionMailer config back to use sendmail. Since my sendmail is installed in the default folder, I don’t need to configure ActionMailer any further. In production.rb, I have

ActionMailer::Base.delivery_method = :sendmail

ActionMailer now delivered mails using the local sendmail program.

Using the console, I sent out a test email to my yahoo acount.   Everything works.  Then, I received a notice from my server, telling me I have new mails at /var/spool/mail. Great! I checked the mail log, and saw …

—– The following addresses had permanent fatal errors —–
<nworld3d@yahoo.com>
(reason: 553 Mail from XX.XXX.XXX.XXX not allowed - 5.7.1 [BL23] Connections not accepted from IP addresses on Spamhaus XBL; see http://postmaster.yahoo.com/550-bl23.html [550])

—– Transcript of session follows —–
… while talking to b.mx.mail.yahoo.com.:
<<< 553 Mail from XX.XXX.XXX.XXX not allowed - 5.7.1 [BL23] Connections not accepted from IP addresses on Spamhaus XBL; see http://postmaster.yahoo.com/550-bl23.html [550]
… while talking to d.mx.mail.yahoo.com.:
<<< 553 Mail from XX.XXX.XXX.XXXnot allowed - 5.7.1 [BL23] Connections not accepted from IP addresses on Spamhaus XBL; see http://postmaster.yahoo.com/550-bl23.html [550]
…….
<<< 553 Mail from XX.XXX.XXX.XXX not allowed - 5.7.1 [BL23] Connections not accepted from IP addresses on Spamhaus XBL; see http://postmaster.yahoo.com/550-bl23.html [550]
554 5.0.0 Service unavailable

…….

Final-Recipient: RFC822; nworld3d@yahoo.com
Action: failed
Status: 5.5.0
Diagnostic-Code: SMTP; 553 Mail from XX.XXX.XXX.XXX not allowed - 5.7.1 [BL23] Connections not accepted from IP addresses on Spamhaus XBL; see http://postmaster.yahoo.com/550-bl23.html [550]
Last-Attempt-Date: Tue, 3 Jun 2008 14:02:54 -0400

OUCH! My VPS got black-listed! Somehow the Spamhaus XBL list decided to black list the IP address of the server. Probably because I had port 25 opened in the firewall and somebody has taken advantage of my lack-of-linux-sysadmin-skills to start relaying spams.  CRAP! Yahoo outrightly refused any SMTP connection from my server because it was marked as a spammer-wannabe .  Good thing Yahoo included the  link to their help page and references to Spamhaus XBL.

I went to Spamhaus website and looked up my server’s IP:

The CBL (composite blocking list) was blocking my server.  I went ahead and request a removal from the list.  Afterwards, the CBL list showed

A few minutes later, I checked Spamhaus and it’s not showing the IP listed anymore.  Phew!

It would take a while before this list is propagated through out the internet.  At the moment, I still can’t connect to Yahoo’s SMTP server and my mails is still delivered right into Google’s Junk mailbox.  Hopefully a few hours more and my server would be good to go.

I don’t know if Spamhaus is my friend or foe.  I have a hard time deciding it.  I don’t know what people’s experiences with emailing services in general, but I can tell it’s a pain in the butt, and everything will only get worse from here.  I wonder what it will be like when ipv6 is used…

This is a quick summary for this process so that I can refer to it later on, and hopefully someone will find it useful as well.

Memcached requires libevent to handle its network IO stuff. The bundled libevent in the standard yum repository is old so it’s pretty useless. The newer versions memcached runs on newer libevent library so I ended up compiling libevent and memcached from the latest stable sources. I’m using libevent-1.4.4-stable and memcached-1.2.5.

First off, uninstall the libevent that yum may have installed on your machine

# sudo yum remove libevent

Download the sources for libevent and memcached , unzip( # gunzip *.gz ), untar (# tar -xvf *.tar), CD to the libevent folder. We will compile the libevent first.

# ./configure –prefix=/usr/local

# make

# make install

Basically we are telling libevent to install itself under /usr/local/lib/. When we compile memcached, we need to point it to the correct location as well. Once libevent is done installing (it’s really quick), we can move on and complie memcached.

CD to the un-tar memcached folder,

# ./configure –with-lib-event=/usr/local/

# make

# make install

After memcached is installed, you can try

# memcached

In my situation, I ran into an error

error while loading shared libraries: libevent-1.4.so.2: cannot open shared object file: No such file or directory

It turned out that the new libevent get installed, it doesn’t “register” the actual library file (similar to DLL on Windows) with the system. When Memcached runs, it tries to look for the libevent-1.4.so.2 file but since libevent is still playing hide and seek somewhere, memcached cries.

To fix this, we need to manually load the libevent library file into the system via the ld configuration. From the man page of ld:

ld combines a number of object and archive files, relocates their data and ties up symbol references. Usually the last step in compiling a program is to run ld.

I like to think ld as the regsrv32 used to register DLL’s on Windows. Now to fix up the reference to the libevent so file, we need to create a file under /etc/ld.so.conf.d/

# vi /etc/ld.so.conf.d/libevent-i386.conf

then enter

/usr/local/lib/

Write and quit (:wq!)

The path in the libevent-i386.conf is the path where the actual .so files are located at. We set this path when we run the ./configure –prefix=/usr/local/ during the libevent compilation. Reloading the ld configuration with

# ldconfig

now, you can start memcached in verbose mode (-vv) for testing

# memcached -vv

If you see something like ..

slab class 1: chunk size 104 perslab 10082
slab class 2: chunk size 136 perslab 7710
slab class 3: chunk size 176 perslab 5957
slab class 4: chunk size 224 perslab 4681
slab class 5: chunk size 280 perslab 3744

….

slab class 37: chunk size 367192 perslab 2
slab class 38: chunk size 458992 perslab 2
<6 server listening
<7 send buffer was 126976, now 268435456
<7 server listening (udp)

Congratulations! Memcached is up and running!

PS:  I’m renting the VPS from www.slicehost.com and so far my experience with them ( 1.5 months) is excellent.

If you do “gem install capistrano”, the gem package manager will go fetch the latest gem version of capistrano, currently 2.3.0 and installed in the gem repository. In case you want to have multiple version s of capistrano running, here’s how to do it.

To install older gem version of Capistrano
gem install –version 1.4.2 capistrano
(1.4.2 is the latest one in the 1.x branch before the release of 2.0)

To run specifically the 1.4.2 version, use
cap _1.4.2_ *your_tasks_here*

Shortcut
To reduce the typing, you can make an alias in your .bash_profile on Linux to run the 1.4.2 version as cap1 (cap 2.x.x is still running as cap) using

# add this line to .bash_profile
alias cap1=”cap _1.4.2_”

Afterwards, reload the profile with

$ source .bash_profile

Since I am on Windows, what I did was creating a batch file called “cap1.bat” and saved it within my system’s PATH environment. For simplicity’s sake, I save the cap1.bat file inside my C:\Windows\System32 folder

@echo off
cap _1.4.2_ %*

The special wildcard %* will be replaced with your command-line arguments, saving you the typing.

To read more about the Capistrano 1.4.2 version, check out Jamis’s post here.

I’m in the process of installing Ruby on Rails on a VPS running CentOS 5.1 and run into an issue with gem complaining for the missing zlib library. What happened was that I first used “sudo yum install ruby” to install the ruby runtime. However, the pre-packaged ruby in the Yum repository was older, 1.8.5. I only found out about this when I tried to run the mongrel_cluster to set up the mongrel instances. Mongrel was complaining about the older version of Ruby and the need for the fcgi fix. I decided to uninstall the yum ruby package and compile the newer version of Ruby from the source.

I compiled and installed Ruby 1.8.6 from the tar ball source first (./configure, make, make install), then I removed the pre-installed package using “sudo yum remove ruby” and removed all the dependencies with it. Next I installed gem 1.1.1 from the source (ruby setup.rb). I proceeded to install rails 2.0.2 again (”sudo gem install rails -y”). Bam, gem started to complain:

/usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in
`gem_original_require': no such file to load -- zlib (LoadError)

Not good!

I did some research and it turned out that my CentOS 5.1 VPS is a barebone box, it did have the extra development libraries installed. Hence when I compiled Ruby from source, make did not compile the zlib extension because I did not have the zlib-devel package installed.

After installing zlib-devel package through yum (”sudo yum install zlib-devel”), I cd’ed into my Ruby source folder and into the ext/zlib folder (in my case, ruby-1.8.6-p114/ext/zlib/). Then I proceeded to install the zlib library (”ruby extconf.rb”, “make”, “make install”). Afterwards, gem was working fine again! Yay!

Based on an article on detection connection speed using JavaScript, I implemented a more completed speed detection solution which you can drop into your project with minimal fuss. An example would be to detect the user’s connection speed to show either high or low bandwidth version of the website. All the ingredients you need are an image, a small JavaScript snippet, and some imagination. There is no dependency and no other 3rd party library needed.

The main idea is that we load an image programmatically and measure the time it takes the browser to load such an file. Once an image is done loading, the browser will fire the onLoad event and we can capture the end time. I included two different versions of the code. The first one is quite simple with the image being loaded only once. The second one is more sophisticated, loading the image multiple times and taking the average loading time to arrive at a more accurate answer.

The key technique being used here are closure and passing functions as arguments. These two techniques are very commonly used in modern JavaScript frameworks such as Prototype, Scriptaculous, jQuery, Ext, YUI, etc.

In order for this technique to work, we need to have an image with an average size. Too small of an image won’t give us a very good result while too big of an image will cause more lags and affect the user’s experience. For your convenience, I included the a JPG image around 58.5KB in size as our testing image. We will need the exact size of the image in Bytes in order to calculate the bitrates or byte-rates.

Simple Javascript Speed Detection
Insert this code snippet into the page where you want to conduct the speed test

var SpeedTest = function() {
  /*
  From:  http://techallica.com/kilo-bytes-per-second-vs-kilo-bits-per-second-kbps-vs-kbps/
  256 kbps            31.3 KBps
  384 kbps            46.9 KBps
  512 kbps            62.5 KBps
  768 kbps            93.8 KBps
  1 mbps ~ 1000kbps   122.1 KBps
  */
};
SpeedTest.prototype = {
  imgUrl: "speedtest.jpg"    // Where the image is located at
  ,size: 59917                // bytes
  ,run: function( options ) {

    if( options && options.onStart )
      options.onStart();

    var imgUrl = this.imgUrl + "?r=" + Math.random();
    this.startTime = (new Date()).getTime() ;

    var testImage = new Image();
    var me = this;
    testImage.onload = function() {
      me.endTime = (new Date()).getTime();
      me.runTime = me.endTime - me.startTime;

      if( options && options.onEnd )
        options.onEnd( me.getResults() );
    };
    testImage.src = imgUrl;
  }

  ,getResults: function() {
    if( !this.runTime )
      return null;

    return {
      runTime: this.runTime
      ,Kbps: ( this.size * 8 / 1024 / ( this.runTime / 1000 ) )
      ,KBps: ( this.size / 1024 / ( this.runTime / 1000 ) )
    };
  }
}

First, we need to specify where we would load the image from (in this case, the relative path to the page) and the exact size of that image in Kilobytes. Since we don’t want the browser to cache the image, which short-circuits our test, we append a random number to the original image’s url (line 19). Before we run the test, we check to see if we need to run the onStart() functions being passed in. The idea is that you can show on the page with some messages or pop-up to notify about the test is about to run, simply by passing in an inline function.

The crux of the detection is from line 23 to line 30. If you are new to JavaScript or you are not comfortable with Closure, you should investigate a bit more since Closure is a critical and extremely powerful part of Object-Oriented JavaScript. In its simplest form of explanation, Closure means an anonymous function from a different execution context can gain access to another function’s internal properties.

In this case, after downloading the image, the browser fires the image’s onLoad event at a later time in a different execution context (the context of the current browser) other than context of our SpeedTest object. Without a closure reference via the variable “me” (line 23), which holds the reference to the SpeedTest object, we can no longer access to the startTime and endTime properties of the current SpeedTest object.

Now to run the test and obtain the results, all you have to do is to initialize a new SpeedTest object, execute the run() method, and pass in your custom actions. Currently 2 custom events onStart() and onEnd() are supported:

var st = new SpeedTest();
st.run({
  onStart: function() {
    alert('Before Running Speed Test');
  }

  ,onEnd: function(speed) {
    alert( 'Speed test complete:  ' + speed.Kbps + ' Kbps');
    // put your logic here
    if( speed.Kbps < 200 )
    {
      alert('Your connection is too slow');
    }
  }
});

The Multi-trial speed test

As with any kind of measurements, one single measure is subjected to certain errors and inaccuracy. In our case, the degree of inaccuracy is much higher since connection speed fluctuates constantly. Other factors would affect the single-pass test is the initial DNS look-up speed and overhead, temporal network latency, slow browser, etc. Hence we can further improve our answers by taking multiple tests and average the results to get a more accurate number.

Below is a more sophisticated speed-detection code, which we can specify how many times the test is run. Typically 3 runs is good enough and it will not impact on the user’s experience.

var SpeedTest = function() {
  /*
  From:  http://techallica.com/kilo-bytes-per-second-vs-kilo-bits-per-second-kbps-vs-kbps/
  256 kbps            31.3 KBps
  384 kbps            46.9 KBps
  512 kbps            62.5 KBps
  768 kbps            93.8 KBps
  1 mbps ~ 1000kbps   122.1 KBps
  */
};
SpeedTest.prototype = {
  runCount: 3                 // how many times we want to run the test for
  ,imgUrl: "speedtest.jpg"    // Where the image is located at
  ,size: 59917                // bytes
  ,run: function( options ) {
    this.results = []; // reset the results
    this.callback = ( options && options.onEnd ) ? options.onEnd : null;
    this.runTrial(0, options);
  }

  ,runTrial: function(i, options ) {
    var imgUrl = this.imgUrl + "?r=" + Math.random();
    var me = this;
    var testImage = new Image();
    testImage.onload = function() {
      me.results[i].endTime = ( new Date() ).getTime();
      me.results[i].runTime = me.results[i].endTime - me.results[i].startTime;

      if ( i < me.runCount - 1 )
        me.runTrial( i + 1 ); // run the next trial
      else
      {
        // Execute the callback
        if( me.callback )
          me.callback( me.getResults() );
      }
    };
    this.results[i] = { startTime: ( new Date() ).getTime() };
    testImage.src = imgUrl;
  }

  ,getResults: function() {
    var totalRunTime = 0;
    for( var i = 0; i < this.runCount; i++ )
    {
      if( !this.results || !this.results[i].endTime )
        return null; // exit if we found no endTime.  --> test’s not done yet
      else
        totalRunTime += this.results[i].runTime;
    }

    var avgRunTime = totalRunTime / this.runCount;

    return {
      avgRunTime: avgRunTime
      ,Kbps: ( this.size * 8 / 1024 / ( avgRunTime / 1000 ) )
      ,KBps: ( this.size / 1024 / ( avgRunTime / 1000 ) )
    };
  }
}

Even though our test method has changed, you can still use the same exact code as above to run the speed-test. The answer will be much more accurate this time. Basically we run multiple trials back-to-back, each time measuring the runtime and store it into an array, then average and get the final result.

Another small difference is that we have to store the reference to the user’s custom onEnd() method as a property of the SpeedTest object so that at the end of the test, we can execute it and return the answer to the user.

Final words

I hope you enjoy this article and make good use of the code. I packaged the 2 versions and the image into a zip file so just go ahead and have some fun. Go speed-racer!

Download

• SpeedTest Package