1-Click Firefox functional testing with Mozmill-Crowd

Henrik Skupin

Firefox 4, the next major version of Mozilla’s web browser, will include hundreds of new features and enhancements. A lot of those will happen in the background and will primarily not be visible at the front-end. A very good example for such a feature is the support for graphics hardware acceleration. It will drastically improve the rendering of pages, videos, or the Firefox UI itself. But noticeable it is only because it is that fast. Alongside those back-end features, also the Firefox UI has been vastly changed. Mainly you will notice the reduced amount of visible elements in the browsers chrome, or the newly introduced Firefox button on Windows and Linux at the top-left side of the window. To ensure that those new features or landed changes work as expected, Mozilla QA and all our testers have to constantly run manual tests for those areas which cannot be automatically tested by a Mochitest. To make the life of all of us easier, we have been actively working on getting even those areas populated with functional automated tests by using our own testing framework called Mozmill.

While all that testing with Mozmill is working fine on an already setup machine, we got a lot of feedback from testers that the process to create the test environment is kinda complicated. That’s why we have decided to create an extension, which will automate all those steps. The tester can lean back and do not worry about this process anymore. Alongside that important feature, we can now even distribute our tests across the globe and get it run by everyone. With the test results collected we have a lot of opportunities to work with the data. Regressions can be identified right at the time when they appear the first time, even if it is restricted to localized Firefox builds. Further we could analyze the performance of executed tests across all the different machines which could give us ideas of existent bottlenecks to fix. And there are even more areas we want to work on, once this process has been established.

How it works?

The Mozmill-Crowd extension is a simple wrapper around the Mozmill testing framework. It ships with its own UI to allow a customization of the test-runs. Testers can select the version of the browser to test, or choose one of the supported test-runs. For the execution the Mozmill command line interface is used. It’s implemented in Python and gets automatically installed as part of the testing environment, which the extension setup itself. The combination of the command line tool and the extension allows us to execute even tests which need a restart of the browser. All the tests are executed in another Firefox instance which uses a fresh profile. That means your daily profile is safe and will not suffer from any data loss caused by an running test.

Installation and Setup

To install Mozmill Crowd for Firefox simply open the add-ons page on addons.mozilla.org and click the Install button. After a restart of Firefox the extension is installed and its menu entry has been added to the Tools menu. Clicking on it will open the Mozmill-Crowd window as shown below:

The settings for test-runs has to be done at the top of the window. Already populated with default values you should see the currently running browser selected in the application dropdown, and the General Test-run as the to test-run to execute. You can switch the application as necessary by adding another installed Firefox versions. To switch the test-run select one of the following options:

  • General Test-run: Executes a good portion of our manual Litmus BFT tests (Basic Functional Tests). Those cover areas like the Password Manager, Tabbed Browsing, Search, and a couple of others. All tests can be also run in localized builds.
  • L10n Test-run: Tests especially created for localizers. It has checks implemented for multiple usage of access keys and cropped elements. At the moment only the preferences window is checked.
  • Add-ons Test-run: Tests written by add-on authors to ensure the add-on is working properly with Firefox.

Beside the above mentioned settings which will be changed most often, others are available in the preferences of the add-on.

To finalize the setup and to be able to run the tests, the test environment has to be prepared. At the moment this step is part of the test-run itself and will be executed as the first task when the “Start Test-run” button is clicked. If the setup hasn’t been run before a dialog will appear and ask you to select a folder. This selected folder is used for all the extension related files, including the test environment. Create a new one or select an existing folder on your hard-drive to continue. In the next step the test environment gets downloaded and prepared in the specified folder. Additional modules, i.e. Mozmill itself, will be downloaded and installed automatically.

Note: Due to current limitations the extension executes the process via a blocking call. That means the browser instance which has been used to start the setup process will not react and also not update the screen until the test-run has been completed. That’s intended at the moment but will be changed in the next version of the extension. The slow script warning can also be ignored.

Test Execution

To execute a test-run simply select its entry in the dropdown and also choose the application under test. A click on “Start Test-run” will start the tests. With the setup step already executed, a script from the test environment will automatically pull the latest version of the automation scripts and all available Mozmill tests before it starts the browser, and runs the selected test-run. Once all the tests have been executed, the application is shutdown.

Note: Similar to the environment setup step the browser itself will be blocked until the test-run has been finished.

Sending Test Results

The extension supports sending of test results to a document based database, i.e. CouchDB. Per default the sending is disabled, but can easily be enabled in the preferences dialog of the extension. We would really appreciate your willingness to send reports to our server. Therefore please obey the following two important information:

  • The application under test always has to be the front-most application. Don’t switch to another application while the tests are executed.
  • Don’t kill the application during the test-run. Due to a bug in Mozmill it would cause a lot of false positive failures in our database.

If you are curious about your own test results simply head to our Mozmill dashboard which lists all the uploaded reports for the general test-run. Your result should appear at the top of the list.

L10n Test-run

With the Firefox 4.0 RC1 release coming up soon our primarily focus is targeted at localizers at the moment. That means if you are leading an l10n team or are a member of one, please run our l10n test-run in your locale of Firefox. Even with only tests for the preferences window available, we already have covered a couple of bugs in different locales of Firefox 4.0 beta releases. If failures have been found, screenshots are saved in the temporary folder of your user account. Those can be used to easily identify the broken UI elements. As an example have a look at bug 614938.

Thanks everyone for your feedback!