To get familiar with Mozmill test scripts you can have a look at the exisiting Firefox tests from the Mozmill-Test repository. If you haven't pulled them on your local disk please read the former chapter about the test repository.

Depending on the method you have used to install Mozmill there are two different ways to execute Mozmill tests. You can run them from within the Mozmill IDE which is part of the Firefox add-on or by calling the command line application. The latter offers additional features like restart capabilities for Mozmill tests.

Note: You should always use a new profile to run those tests. Never run them in your daily profile because they will cause dataloss (e.g. all bookmarks are deleted).

Using the Mozmill IDE

After installing the extension the Mozmill IDE is available from within the Tools menu. It consists of several parts. From top to down these are:

• Menu bar: Offers access to all supported functions
• Tab bar: Contains tabs to switch between the editor, output, events, and shell panes
  • Editor: Create and modify Mozmill tests with the integrated text editor
  • Output: Test results when running a test are shown on this tab
  • Events:
  • Shell: Run and test Javascript in the Mozmill scope

Ok, lets go ahead and finally explain how tests can be run from inside the IDE. For this purpose three different ways exist which can be found inside the Test menu of the menu bar:

• Run Editor
• Run File
• Run Directory

The simplest solution is to use "Run File" which let you choose a test script and runs it automatically when you click the Open button of the file dialog. Any test results will be logged to the output panel. The second option is to use "Run Editor". To start the test you have to open the test script by yourself. Once its content is shown inside the Editor panel you can click on "Run Editor" and the test will be run in the same manner as with "Run File". Last but not least there is the third option called "Run Directory". Using this method Mozmill lets you specifiy a folder and will run all tests under this folder and its subfolders.

Using the Command Line

In comparance to the Mozmill IDE the command line application can only execute Mozmill tests. Editing has to be done with an external editor - just use your favorite one! To run a test, call the application with one of the options given below. A fresh profile will be automatically created so the test always runs in a clean environment. But keep in mind that if you want to run a bunch of tests inside a folder all those tests will be executed in the same profile. Beneath those normal tests you will also be able to run restart tests "e.g. extension installation".

How to Execute a Mozmill Test

Once you have installed Mozmill via setuptools you can run the mozmill command to get an overview about the available options:

$ mozmill --help

Usage: mozmill [options]

Options:

  -h, --help            show this help message and exit

  -b BINARY, --binary=BINARY

                        Binary path.

  -w PLUGINS, --plugins=PLUGINS

                        Plugin paths to install.

  -n, --no-new-profile  Do not create new profile.

  -l LOGFILE, --logfile=LOGFILE

                        Log all events to file.

  --report=REPORT       Report the results. Requires url to results server.

  -t TEST, --test=TEST  Run test file or directory.

  --showall             Show all test output.

  -D, --debug           Install debugging plugins.

  -d DEFAULT_PROFILE, --default-profile=DEFAULT_PROFILE

                        Default profile path.

  --show-errors         Print logger errors to the console.

  -u, --usecode         Use code module instead of iPython

  -s, --shell           Start a Python shell

  -p PROFILE, --profile=PROFILE

                        Profile path.

  -P PORT, --port=PORT  TCP port to run jsbridge on.

You will get a list of all available options which let you customize the Mozmill test-run. Most of the times you will use the -t option to specify a single file or a folder. But also -b is helpful to run the test(s) against a given version of Firefox. Otherwise the systems default Firefox installation will be used. In the following you can find some examples specific to our Mozmill test repository. Just change into the newly created mozmill-test repository to continue...

$ mozmill -t firefox/testPreferences/testRestoreHomepageToDefault.js

Starts the default Firefox application, executes the given test, and closes Firefox afterward.

$ mozmill -t firefox/testPreferences/

Starts the default Firefox application, executes all the tests in the given folder in alphabetical order, and closes Firefox afterward.

$ mozmill -t firefox/testPreferences/testRestoreHomepageToDefault.js -b "c:\firefox 3.5\firefox.exe"
$ mozmill -t firefox/testPreferences/testRestoreHomepageToDefault.js -b "/usr/bin/firefox"
$ mozmill -t firefox/testPreferences/testRestoreHomepageToDefault.js -b "/Applications/Firefox.app"

Starts the specified Firefox version (depending on the platform: as shown Windows, Linux, OS X), executes all the tests in the given folder in alphabetical order, and closes the browser afterward.

Note: On Windows and Linux the full path to the executable has to be specified when using the -b parameter.

If tests fail or error/exceptions appear it can be helpful to specify the --show-errors option too. The full error description will be logged to the console.

How to Execute a Mozmill Restart Test

When Mozmill gets installed the mozmill-restart application is automatically installed too. It allows you to run Mozmill tests with restarts in between. That allows you to run tests like installing an extension which needs a restart to finish. For restart tests you will always specify a folder as the test source. Inside this folder all the tests have the following names: test1.js, test2.js, ..., testX.js. That way you can run as many tests as you want in numerical order.

The mozmill-restart application expects the same options. To get a list of them call the application with the --help option. Lets show some examples:

$ mozmill-restart -t firefox/restartTests/testExtensionInstallUninstall/

Starts the default Firefox application, runs all the tests in the given folder by restarting Firefox in-between, and finally closes Firefox. The same profile is used for all tests.

$ mozmill-restart -t firefox/restartTests/

Starts the default Firefox application, runs all the restart tests in the given folder by restarting Firefox in-between, and finally closes Firefox. The same profile is used for each restart test. It will not be shared between the different restart tests.

$ mozmill-restart -t firefox/restartTests/testExtensionInstallUninstall/ -b "c:\firefox 3.5\firefox.exe"

Starts the Firefox version located in the given folder "c:\firefox 3.5", runs all the tests in the given folder, restarts Firefox in-between each test and finally closes the browser.

Note: On Windows and Linux the full path to the executable has to be specified when using the -b parameter.

Branches

In our repository the existing tests have to work on different versions of Firefox. Given the fact that things will change over time and new features are implemented the tests are stored in different branches. That means before you can run those tests you have to check if the following branch matches the to use Firefox application. If they differ you have to switch the branch:

Firefox 3.5.x: mozilla1.9.1

Firefox 3.6.x: default

Selecting a branch

To check which branch is currently selected run the following command:

$ hg branch
mozilla1.9.1

Given the output the current branch is mozilla1.9.1 and the tests will work with all versions of Firefox 3.5.x. If another branch is needed the command below switches to the default branch:

$ hg branch default

Run all BFT/FFT Firefox Tests

Once the correct branch has been selected the full suite of Mozmill tests can be run against Firefox. There are two command which have to be executed. The following gives an example for a Firefox 3.5 build installed in the given folder:

$ mozmill -b "c:\firefox 3.5\firefox.exe" -t firefox/
$ mozmill-restart -b "c:\firefox 3.5\firefox.exe" -t firefox/restartTests

Software Update Tests

As part of the release testing process QA has to perform software update tests on various update channels to confirm that the update process works flawlessly on all systems. Therefor software update tests written for Mozmill are available.

Run the Tests

To run the tests a Python script has to be called which is located under the scripts folder. It's name is update.py. A list of command line options can be retrieved by running the script with the help parameter:

$ python scripts/update.py --help
Usage: update.py [options]

Options:
  -h, --help            show this help message and exit
  --no-fallback         No fallback update should be performed
  -b BINARY, --binary=BINARY
                        Binary path.
  -i INSTALL, --install=INSTALL
                        Installation folder for the build
  -n, --no-new-profile  Do not create new profile.
  -l LOGFILE, --logfile=LOGFILE
                        Log all events to file.
  -c CHANNEL, --channel=CHANNEL
                        Update channel (betatest, beta, nightly, releasetest,
                        release)
  --showall             Show all test output.
  -P PORT, --port=PORT  TCP port to run jsbridge on.
  -t TYPE, --type=TYPE  Type of the update (minor, major)
  --report=REPORT       Report the results. Requires url to results server.
  --show-errors         Print logger errors to the console.
  -D, --debug           Install debugging plugins.
  -p PROFILE, --profile=PROFILE
                        Profile path.

As you can see most of the options are the same as when you run normal Mozmill tests. But there were added 3 more options:

• --channel - Specifies the channel the test should be run against
• --install - Install the build into the given folder before the test is started and uninstall it afterward.
• --no-fallback - Perform the direct update only without simulating a failure during the update
• --type - Selects which update type (minor or major) should be tested

Finally the builds to test have to be specified as arguments. It can be a list of all the builds or an existent folder. In the latter case the script will automatically collect all necessary files.

Lets show some examples how to run the tests:

• Run a normal update test (partial/complete) by using an installer build and uninstall it afterward

$ python scripts/update.py --no-fallback -i "c:\firefox" firefox_setup.exe

 

• Run a fallback update test (partial/complete) by using an already installed build

$ python scripts/update.py "c:\program files\mozilla firefox\firefox.exe"

 

• Run an update test (partial/complete) by using the beta channel and all builds inside the given folder

$ python scripts/update.py --channel="beta" -i "c:\firefox" "c:\builds\"

 

• Run a major update test on the default channel

$ python scripts/update.py --type=major -i "c:\firefox" firefox_setup.exe