Željko Filipin's Blog on Software and Testing

Random photo: Varaždin, Croatia.

To make it explicit: I am not affiliated with TestingBot or Sauce Labs.

A few days ago I wrote about how to run watir-webdriver tests at Sauce Labs OnDemand. I have also noticed that watir-webdriver tests are pretty slow at Sauce Labs OnDemand.

Jari Bakken suggested to use net-http-persistent gem to speed things up. Jochen Delabie suggested I should try TestingBot. So, I have decided to try both.

TestingBot is pretty much the same service as Sauce Labs OnDemand. There is also 200 minutes of free trial, and the prices are a bit lower than Sauce Labs OnDemand. If you have used Sauce Labs OnDemand, TestingBot will look familiar. If you have not used Sauce Labs OnDemand, read the last couple of articles I have wrote here.

So, what you need to do to run your tests at TestingBot?

If you already run tests at Sauce Labs OnDemand:

require "watir-webdriver"
caps = Selenium::WebDriver::Remote::Capabilities.firefox
caps.version = "11"
caps.platform = :LINUX
caps[:name] = "testing this cloud thing"
browser = Watir::Browser.new(
  :remote,
  :url => "http://username:api-key@ondemand.saucelabs.com:80/wd/hub",
  :desired_capabilities => caps)

you only have to change caps.version and :url. Depending on the platform and browser, check if you have to change other settings too. (Supported operating systems: Sauce Labs OnDemand, TestingBot.)

require "watir-webdriver"
caps = Selenium::WebDriver::Remote::Capabilities.firefox
caps.version = 11
caps.platform = :LINUX
caps[:name] = "testing this cloud thing"
browser = Watir::Browser.new(
  :remote,
  :url => "http://key:secret@hub.testingbot.com:80/wd/hub", 
  :desired_capabilities => caps)

Of course, replace key and secret with your credentials.

I have run pretty much the same script as in the previous post. The script opens the browser, goes to 9 sites and closes the browser. Actually, it does that with and without using net-http-persistent gem, on a local browser, at Sauce Labs OnDemand and at TestingBot.

(I could not figure out how to make the last two columns display rounded numbers.)

What do the numbers say?

net-http-persistent gem this time helped to reduce test run time up to 10%, but other tests showed up to 15% shorter times.

TestingBot is still about 2 times slower than running a local browser, but Sauce Labs OnDemand is about 4 times slower. So, running tests at TestingBot is about 2 times faster than at Sauce Labs OnDemand.

As usual, if you want to run the script yourself, here it is:

Random Croatian Countryside

That Sauce Labs OnDemand thing is pretty nice, but I have noticed something strange. A test suite that takes about 20 minutes to run at my machine, takes 5 hours to run there! Uh-oh. Not what I have expected.

After googling around, and I have found that you can speed up the tests by disabling video, screen shots and advisor. So, I have decided to test how fast are the tests with default settings and with everything disabled.

This simple script visits a few pages. In my test it visits 9 client sites. You can visit top 10 sites, for example, if you want to run the test yourself.

require "rubygems"
require "watir-webdriver"
browser = Watir::Browser.new
sites = [
  "google.com",
  "facebook.com",
  # add more sites
]
sites.each do |site|
  browser.goto site
end
browser.close

Run the script 10 times (or more, if you have the time) to make the results at least a bit statistically significant. Measure the time it takes to run the script at your machine and at Sauce Labs OnDemand, with various operating system and browser combinations.

I got this results.

The script that just opens the browser, visits 9 pages and closes the browser takes 14 seconds on average at my machine. Exactly the same script at Sauce Labs OnDemand takes on average 55-69 seconds with default settings, and 48-64 seconds with everything disabled. So, the script is a bit faster when everything is disabled, but unfortunately, just a bit.

The entire script is here:

You have to replace username and api-key with your username and API key.

The only solution I have found so far is to run the tests in parallel. Report coming soon.

Bug Hunter's Journal

For a couple of clients I have been running watir-webdriver tests using Sauce Labs OnDemand. Never heard about it? In short, instead of driving a browser at your local machine, you can drive a browser in the cloud. Sounds cool, right?

What changes should you do to your existing watir-webdriver scripts to run them in the cloud?

This would open Firefox at your local machine:

require "rubygems"
require "watir-webdriver"
browser = Watir::Browser.new :firefox

And this would open Firefox at Sauce Labs OnDemand:

require "watir-webdriver"
caps = Selenium::WebDriver::Remote::Capabilities.firefox
caps.version = "11"
caps.platform = :LINUX
caps[:name] = "testing this cloud thing"
browser = Watir::Browser.new(
  :remote,
  :url => "http://username:api-key@ondemand.saucelabs.com:80/wd/hub",
  :desired_capabilities => caps)

You have to replace username and api-key with your username and API key. If you do not have saucelabs.com account, you can get one for free. Free account has limitation of 200 minutes of driving a browser per month, but that is more than enough to try.

You can run the above code by pasting it into IRB, or paste it into a Ruby file (named cloud.rb, for example) and just run the file from command line.

After the browser is opened, you can drive it as usual.

Now go to https://saucelabs.com/jobs and there should be a test with status running and name testing this cloud thing.

testing this cloud thing

Click the test name and you will see live video of the browser running in the cloud. After the test is finished you will be able to view the video and see the screenshots that (I think) are automatically taken for every page that the browser opens.