Note: This is revised version of a previous blog post due to some important changes on running Firefox OS UI tests on the Firefox OS desktop build.
It’s still a little difficult to get your hands on a device that can run Firefox OS right now, but if you’re interested in running the UI tests a device is not essential. This guide will show you how to run the tests on the nightly desktop builds we provide.
Step 1: Download the latest desktop build
The Firefox OS desktop build lets you run Gaia (the UI for Firefox OS) and web apps in a Gecko-based environment somewhat similar to an actual device. There are certain limitations of the desktop client, including: it doesn’t emulate device hardware (camera, battery, etc), it doesn’t support carrier based operations such as sending/receiving messages or calls, and it relies on the network connection of the machine it’s running on.
You can download the latest desktop build from this location, but make sure you download the appropriate file for your operating system. Unfortunately, due to bug 832469 the nightly desktop builds do not currently work on Windows, so you will need either Mac or Linux (a virtual machine is fine) to continue:
- Mac: b2g-[VERSION].multi.mac64.dmg
- Linux (32bit): b2g-[VERSION].multi.linux-i686.tar.bz2
- Linux (64bit): b2g-[VERSION].multi.linux-x86_64.tar.bz2
Once downloaded, you will need to extract the contents to a local folder. For the purposes of the rest of this guide, I’ll refer to this location as $B2G_HOME.
If a profile is specified when running the tests (recommended), a clone of to profile will be used. This helps to ensure that all tests run in a clean state, however if you also intend to launch and interact with the desktop build manually I would recommend making a copy of the default profile and using the copy for your tests.
Step 2: Acknowledge the risks
When running against a device, there’s a very real risk of data loss or unexpected costs. Although it’s much less likely when running against the Firefox OS desktop build, there’s still potential for data loss. For this reason you must create a test variables file to acknowledge this risk. You can find more details for how to do this here.
Step 3: Populate your test variables
Now that you have a test variables file, you can (optionally) add test variables that might be required by certain tests. For example, if you want to run the e-mail tests, you must provide valid e-mail account details. You can read more about the test variables here.
Step 4: Run the tests!
You will need to have git and Python installed (I recommend using version 2.7), and I highly recommend using virtual environments.
First, clone the gaia-ui-tests repository using the following command line, where $WORKSPACE is your local workspace folder:
cd $WORKSPACE git clone git://github.com/mozilla/gaia-ui-tests.git gaia-ui-tests cd gaia-ui-tests
If you’re using virtual environments, create a new environment and activate it. You will only need to create it once, but will need to activate it whenever you wish to run the tests:
virtualenv .env source .env/bin/activate
Now you need to install the test harness (gaiatest) and all of it’s dependencies:
python setup.py develop
Once this is done, you will have everything you need to run the tests, using the following command:
gaiatest --app=b2gdesktop --binary=$B2G_HOME/path/to/b2g-bin --profile=$B2G_HOME/path/to/gaia/profile --testvars=/path/to/testvars.json --restart --type=b2g-xfail gaiatest/tests/manifest.ini
You should then start to see the tests running, with output similar to the following:
starting httpd running webserver on http://10.0.1.12:61016/ TEST-SKIP | test_clock_add_alarm_multiple_times.py | device=desktop, app=B2G TEST-SKIP | test_play_3gp_video.py | device=desktop, app=B2G TEST-SKIP | test_gallery_delete_image.py | device=desktop, app=B2G ... TEST-START test_settings.py test_get_all_settings (test_settings.TestSettings) ... ok test_set_named_setting (test_settings.TestSettings) ... ok test_set_volume (test_settings.TestSettings) ... ok ---------------------------------------------------------------------- Ran 3 tests in 5.246s OK
You see more skipped tests, and these are simply tests that are not appropriate to run on the desktop build.
We also have a subset of these tests running against the desktop build in Travis CI. Click the following build status image for details of the latest results. 
Step 5: Contribute?
Now you can run the tests, you’re in a great position to help us out! To contribute, you will need to set up a github account and then fork the main gaia-ui-tests repository. You will then need to update your local clone so it’s associated with your fork rather than the main one. You can do this with the following commands, replacing $USERNAME with your github username:
git config remote.origin.url git@github.com:$USERNAME/gaia-ui-tests.git git fetch origin git remote prune origin
You can now create a branch, and make your changes. Once done, you should commit your changes and push them to your fork before submitting a pull request. I’m not going to cover these steps in detail here, as they’re fairly standard git practices and will be covered in far better detail elsewhere. In fact, github:help has some fantastic documentation.
If you’re looking for a task, you should first check the desktop issues list on github. If there’s nothing available there, see if you can find an area that needs more coverage. Feel free to add an issue and a comment to say you’ll work on it.
You can also ask us for tasks! There are several mailing lists that you can sign up to: Automation Development, Web QA, and B2G QA. We’re also on IRC, and you can find us in #automation, #mozwebqa, and #appsqa all on irc.mozilla.org.