Automation Development Team and Q2 Goals

Henrik Skupin

Some changes to our team happened lately I want to quickly talk about before starting to dive into our team goals for this quarter. So what’s going on?

Automation is a key part when it comes to qualifying new and existing code. Over the last years it’s getting more and more important at Mozilla for developers and QA to proof the quality of Firefox on a fast pace without the hassle of manual testing. With the change in the Firefox release process to a rapid train model a ton of different builds have to be tested by QA on a regular basis. Doing all that manually is impossible. A test framework which has a high impact for QA is Mozmill and the tests and automation tools written around by the Automation Services team over the last couple of months help a lot. But at the current state it’s not enough. We have to be better and cover way more of the manual testing with automated tests.

To make that possible the Automation Services team is now part of the Automation and Tools Engineering team (aka A-team) and has been renamed to Automation Development. We, which means Dave Hunt and myself, are part of the crew who will be responsible for any kind of automation question. For now we can only cover the Mozmill and Selenium frameworks but with additional team members added hopefully soon, more and more harnesses can be covered. Another key aspect for our crew will be mentorship and education, which means that we are eager to bring you into the automation land and to know all the stuff to write automated tests on your own. Sounds interesting? It definitely is, and I like it a lot!

So what will we work on during the current quarter?

There is a lot of work which has to be done for Mozmill to finally get out version 2.0 and to update our automation scripts and Mozmill tests, including a new API to write the tests. So those will be our prioritized projects.

  1. Mozmill 2.0: A new major version of Mozmill has to be released to fix a lot of our problems we still have on the 1.5 branch, and which can’t be easily solved there. The development is already active for a lot of months but we were never able to finish it up. Given that this new version is a blocker for us to create more reliable Mozmill tests, lets get it hopefully done by this quarter. If you have Python or Javascript skills stop by and let us know. We could need any help we can get.
  2. Automation Scripts: Those scripts drive our Mozmill test-runs for different versions of Firefox. They have been written in the early stages around Mozmill 1.0 and need a complete refactor to work with Mozmill 2.0. Also a couple of features will have to be moved to MozBase so that even other harnesses can benefit from. This is all Python related work.
  3. Test API refactor: In parallel to the above mentioned projects we will also have to update our API, used by Mozmill tests, to make it easier to write new and maintain existing tests. This is absolutely important especially for new members who start with Mozmill. If you have Javascript skills and want to improve them, let us know too.

Beside those projects we will also continue our work in the following areas:

  • Mozmill CI: More and more tests are getting covered by our new Mozmill CI system, which I will talk about more in detail soon. So we have to move this instance to an ESX cluster to have a ton of machines for test execution, include handling of the DTC (default to compatible) test-run, and make sure that the upcoming Firefox Beta Nightly builds are handled too.
  • Mozmill Tests: We take a stab in handling and coordinating the creation of new Mozmill tests and fixing the broken ones. The reason for this change I will also explain in a separate blog post shortly. But to say in general we need more test coverage and skipped/failing tests over a long time are bad practice and counter-productive. So lets get all of them re-enabled again and new tests written.
  • Addons: There will not be that much time for us this quarter to continue developing the add-ons we own. But we will make sure to fix breakage for MemChaser and Nightly Tester Tools if those occur. If you want to help out, you are more then welcome!
  • WebApps Testing: While this is not on our official goals for this quarter we have to assist the WebApps QA team to figure out the right test harnesses to create automated functional tests for open web apps. It will be a challenge and we are kinda interested to see that upcoming.

As you can see there are plenty of projects to work on. Any of those are public and are waiting for your contribution. If you are interested in working with us and to raise your personal skills, please contact us via our public mailing list or #automation IRC channel.