Testing Selenium IDE with Mozmill

Dave Hunt

Mozmill tests can be written for any Gecko based application, and can therefore be used to test Firefox extensions (add-ons). Since October I have been working on a new suite of tests for the Selenium IDE extension in the hope that we will be able to discover any regressions in new versions of either the add-on or in Firefox itself. Another reason for creating the suite is to demonstrate the ease of which such tests can be written and to encourage add-on developers to create test suites themselves.

Once tests have been created they can be checked into the Mozmill Tests repository. We will soon be running these on a daily basis against nightly Firefox builds and making reports available on our public dashboard.

The Selenium IDE tests currently comprise of three major parts:

  1. The shared module (selenium.js) abstracts the tests from the location of elements, provides centralised methods for common tasks, and exposes properties based on the UI.
  2. The checks helper module (checks.js) provides methods for common assertions to avoid duplication across tests.
  3. The tests themselves.

There are currently just 20, which basically executes Selenium commands and check that they pass or fail as expected. Below is a guide to construct one of these tests if you weren’t using the shared module or checks helper module. The test verifies that the assertText command executes and passes as expected.

First, we open Selenium IDE using the Tools menu item:

  controller.mainMenu.click("#menu_ToolsPopupItem");
  controller = Utils.handleWindow("type", "global:selenium-ide", undefined, false);

Then we clear the current contents of the Base URL field by selecting all of the text in it and hitting the delete key, and then type in our test data. You will notice here a reference to the getElement method, which allows us to gather all locators in a single location for less duplication, and much simpler test maintenance:

  var baseURL = this.getElement({type: "baseURL"});
  var cmdKey = Utils.getEntity(this.getDtds(), "selectAllCmd.key");
  controller.keypress(baseURL, cmdKey, {accelKey: true});
  controller.keypress(baseURL, 'VK_DELETE', {});
  controller.type(baseURL, "chrome://selenium-ide/");

Now we add three new commands to our Selenium test case by selecting the next available row and typing into the various fields:

  var commands = this.getElement({type: "commands"});
  var command = this.getElement({type: "command_action"});
  var target = this.getElement({type: "command_target"});
  var value = this.getElement({type: "command_value"});
 
  Widgets.clickTreeCell(controller, commands, commands.getNode().view.rowCount - 1, 0, {});
  controller.type(command, "open");
  controller.type(target, "/content/tests/functional/aut/search.html");
 
  Widgets.clickTreeCell(controller, commands, commands.getNode().view.rowCount - 1, 0, {});
  controller.type(command, "assertText");
  controller.type(target, "link=link with onclick attribute");
  controller.type(value, "link with onclick attribute");
 
  Widgets.clickTreeCell(controller, commands, commands.getNode().view.rowCount - 1, 0, {});
  controller.type(command, "echo");
  controller.type(target, "final command");

With our commands in place, we click the toolbar button to execute the test and wait for the test to complete:

  var playTest = this.getElement({type: "button_playTest"});
  controller.click(playTest);
 
  controller.waitFor(function () {
    return !playTest.getNode().disabled;
  }, "Play test button is enabled");

Now that the test has run, we check that the suite progress indicator has the ‘success’ class:

  var suiteProgressIndicator = this.getElement({type: "suiteProgress_indicator"});
  controller.assert(function () {
   return suiteProgressIndicator.getNode().className === "success";
  }, "Suite progress indicator is green");

We also check that the run counts are correct. The total number of tests run should be 1, and the number of failed tests should be 0:

 controller.assertValue(this.getElement({type: "suiteProgress_runCount"}), "1");
 controller.assertValue(this.getElement({type: "suiteProgress_failureCount"}), "0");

Now we check that there are no errors in the log console:

  var logErrors = this.getElements({type: "log_errors"});
  controller.assert(function () {
    return logErrors.length === 0;
  }, "No error messages present - got '" + logErrors.length +"', expected '" + 0 + "'");

Because the command we are testing should pass, we also check that the final command was executed:

  var logInfo = this.getElements({type: "log_info"};
  var finalLogInfoMessage = logInfo[logInfo.length-1].getNode().textContent;
  var re = new RegExp("^\[info] (.*)");
 
  controller.assert(function () {
    return re.exec(finalLogInfoMessage)[1] === "echo: final command";
  }, "Final command was executed, got '" + re.exec(finalLogInfoMessage)[1] +"' expected 'echo: final command'");

Finally, we close Selenium IDE:

  controller.window.close();
  mozmill.utils.waitFor(function () {
    return !mozmill.utils.getWindowByType("global:selenium-ide");
  }, "Selenium IDE has been closed.");
  controller = null;

As there are a lot of things here that will be shared across several tests, you can see that there would be a lot of duplicated code. This is the reason we abstract the useful user interface interactions into the shared module and the useful checks into a helper module. A nice side-effect of this is the test becoming much more readable. Below is the same test as above but calling out to the additional modules:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
var selenium = require("../../../lib/selenium");
var checks = require("../../../lib/checks");
 
function setupModule(module) {
  controller = mozmill.getBrowserController();
  sm = new selenium.SeleniumManager();
  sm.open(controller);
}
 
function teardownModule(module) {
  sm.close();
}
 
function testAssertTextCommandPasses() {
  sm.baseURL = "chrome://selenium-ide/";
  sm.addCommand({action: "open",
                target: "/content/tests/functional/aut/search.html"});
  sm.addCommand({action: "assertText",
                target: "link=link with onclick attribute",
                value: "link with onclick attribute"});
  sm.addCommand({action: "echo",
                target: "final command"});
  sm.playTest();
 
  checks.commandPassed(sm);
 
  sm.controller.assert(function () {
    return sm.finalLogInfoMessage === "echo: final command";
  }, "Final command was executed, got '" + sm.finalLogInfoMessage +"' expected 'echo: final command'");
}

As mentioned, there are currently only 20 automated tests for Selenium IDE, and we need more! If you’re interested in helping out and you have any questions, then you can either get in touch with me directly, ask in the #selenium IRC channel on Freenode, or post a message to the selenium-developers Google group.