QMO quality.mozilla.org (alpha)

User Management

• In the left-hand sidebar, click on the Manage Users link.
• Enter your search criteria in the search box and submit.
• Select the user you would like to edit from the list of returned users. Click on "edit" in the left-hand column to edit that user.
• Update the information for the user as necessary and then click on "Submit changes."

Flags and Values

• Enabled: if a user account is not enabled, they cannot login and run tests. User accounts are enabled by default.
• Is an admin?: Admin users can edit everything, including other users.
• Web services auth token: A web services auth token allows that user to submit testing results via web services (XML-RPC). In general, we should encourage users who are going to submit results via web services (likely automated results) to setup a separate account for submitting those results.

Categories

What are Categories?

"Categories" is a catch-all term that refers to the various components that are used to describe and delineate testcases, groupings, and results. Specifically, these include:

• Products;
• Platforms;
• Operating Systems; and,
• Branches.

Managing Categories

All the aforementioned categories have options to Add new entities, or Edit or Delete existing entities.

• Add: calls up a new form for entering the required criteria for that entity.
• Edit: calls up a form with existing values for that entity prepopulated. Edit as required, and then submit your changes.
• Delete: calls up a confirmation dialog to verfiy that you really want to delete the chosen entity.

Testcase Management

Adding a testcase

It is a good idea to check for existing testcases that can be re-used or cloned before creating a new testcase.

Fill in all of the required information for the new testcase. Javascript checks will make sure you have provided all the required information.

• Steps to perform/Expected Results: be as specific yet concise as possible. Remember that the people who will be running this test will not necessarily be as familiar with Mozilla processes and internals. If terms or concepts need clarification, hyperlink them to an appropriate information source rather than explaining them inline. This will help reduce the size of the testcase and hopefully make it less daunting to testers.
• Enabled: disabled testcases will not appear in test lists, except in the Manage Testcases interface or when accessed explicitly.
• Community Enabled: Only testcases that are community enabled can be seen by the community when tests are being run. This flag will not affect the view for admin users, i.e. admin users will always all tests regardless of this flag.

NOTE: Once a new testcase is added, you need to add it to a subgroup as a separate step.

Edit, Clone, and Delete

• You can use the filter criteria to narrow the list of displayed testcases.
• Select the testcase you want to clone or delete.
• Verify from the displayed information that the testcase you have selected is the one you want to clone/delete.
• Confirm your action in the resulting dialog.
• When cloning, be sure to edit the new testcase to distinguish it from the original testcase by more than ID.

NOTE: when cloning a testcase, all of the existing relationships, e.g. subgroup membership, will be propagated to the new testcase. You will need to edit the new testcase to remove or alter those linkages if they are not accurate.

Please see How to create a Testcase for more details.

Subgroup Management

Subgroups are the first level of testcase aggregation. Generally speaking, a subgroup will contain an ordered selection of testcases that test a particular area of the product at a given resolution.

There may be multiple subgroups for the same product area at varying resolutions if that area needs to be included in testgroups of differing resolutions, e.g. Smoketests vs. BFTs.

Adding a subgroup

It is a good idea to check for existing subgroups that can be re-used before creating a new subgroup.

Fill in all of the required information for the new subgroup. Javascript checks will make sure you have provided all the required information.

• Enabled: disabled subgroups will not appear in subgroup lists, except in the Manage Subgroups interface or when accessed explicitly.

Adding testcases to a subgroup

The list of "All testcases for Product/Branch" on the left is populated/filtered as you select a product and branch for your subgroup above. You can then select testcases to add to your subgroup, and move them across into the "Testcases in Subgroup" box using the arrow buttons. You can also change the order of the testcases within the subgroup using the up and down buttons on the right.

Edit, Clone, and Delete

• You can use the filter criteria to narrow the list of displayed subgroups.
• Select the subgroup you want to clone or delete.
• Verify from the displayed information that the subgroup you have selected is the one you want to clone/delete.
• Confirm your action in the resulting dialog.
• When cloning, be sure to edit the new subgroup to distinguish it from the original subgroup by more than ID.

NOTE: when cloning a subgroup, all of the existing relationships, e.g. testcase membership, will be propagated to the new subgroup. You will need to edit the new subgroup to remove or alter those linkages if they are not accurate.

Testgroup Management

Testgroups are the second level of testcase aggregation, and contain ordered collections of subgroups. Generally speaking, a testgroup will contain a selection of subgroups at a given resolution that combine to achieve appropriate coverage of the given product (or part thereof) at the chosen resolution.

There are usually multiple testgroups for the same product area at varying resolutions, e.g. Smoketests vs. BFTs.

Adding a Testgroup

It is a good idea to check for existing testgroups that can be re-used before creating a new testgroup.

Fill in all of the required information for the new testgroup. Javascript checks will make sure you have provided all the required information.

• Enabled: disabled testgroups will not appear in testgroup lists, except in the Manage Testgroups interface or when accessed explicitly.

Adding subgroups to a testgroup

The list of "All subgroups for Product/Branch" on the left is populated/filtered as you select a product and branch for your testgroup above. You can then select subgroups to add to your testgroup, and move them across into the "Subgroups in Testgroup" box using the arrow buttons. You can also change the order of the subgroups within the testgroup using the up and down buttons on the right.

Edit, Clone, and Delete

• You can use the filter criteria to narrow the list of displayed testgroups.
• Select the testgroup you want to clone or delete.
• Verify from the displayed information that the testgroup you have selected is the one you want to clone/delete.
• Confirm your action in the resulting dialog.
• When cloning, be sure to edit the new testgroup to distinguish it from the original testgroup by more than ID.

NOTE: when cloning a testgroup, all of the existing relationships, e.g. subgroup membership, will be propagated to the new testgroup. You will need to edit the new testgroup to remove or alter those linkages if they are not accurate.

Test Run Management

Adding a Test Run

Fill in all of the required information for the new test run. Javascript checks will make sure you have provided all the required information.

• Start/Finish Timestamp: these variables influence how long a a given test run is visible. Long-running test runs (e.g. the catch-alls) should have their finish timestamp set for some date far in the future, ideally to the point in time where we plan to stop supporting that branch. These values can always be adjusted later.
• Enabled: disabled test runs will not appear in test runs lists, except in the Manage Test Runs interface or when accessed explicitly.
• Recommended? recommended test runs are listed preferentially (i.e at the top) of the listing of available test runs and on the Litmus index page.

Edit, Clone, and Delete

• You can use the filter criteria to narrow the list of displayed test runs.
• Select the testgroup you want to clone or delete.
• Verify from the displayed information that the testgroup you have selected is the one you want to clone/delete.
• Confirm your action in the resulting dialog.
• When cloning, be sure to edit the new test run to distinguish it from the original test run by more than ID.

Reporting

Existing reports

The following reports are available in the footer of every page:

• Recent Failures
• Recently Unclear
• Most Common Failures
• Testcases Most Frequently Marked As Unclear
• Testgroup Popularity
• Results With Comments
• Recently Added Testcases
• Recently Changed Testcases

The statistics page reports totals for various elements in Litmus (testcases, subgroups, etc.).

Reports for previous testdays can be accessed from the Testdays page.

Do-it-yourself

The are two different ways to create your own search:

Search Results:

This is the simpler of he two searching options. It allows you to limit your results by some common fields and specify how many results you would like to see.

Advanced Search:

This search page allows much more customization, at the cost of being a little slower.

Limit By Field: these lists operate similarly to the auto-updating drop-down lists elsewhere in Litmus. This section allows you to limit your results to specific test runs, products, branches, testgroups, subgroups, testcases, platforms, operating systems, locale, and result statuses.

Limit By Date: you can either choose a predefined timespan like on the regular search results page, or you can define your own interval. The interval date math uses the Date::Manip perl module syntax, which allows you to submit strict timestamps (YYYYMMDDhhmmss), or something more elaborate, such as:

e.g.
Start Date: 8 hours ago
End Date: Now

This would return all the results submitted in the previous 8 hours.

The syntax is quite robust, allowing you to work with just business days, etc. Please see the Date::Manip documentation for more infromation.

Search By Field: This section allows you to do boolean and/or regular expression searching on many of the fields in the database.

e.g.
Field: Submitted By
Match Criteria: contains the word/string
Search Value: mozilla.com

This would return all results submitted by testers using a mozilla.com email address.

Sort Order: allow you to sort your results.

General Options: these are options available to all users, including probably the most useful option which is to limit the number of results returned. This section also allows you to search for automated test results specifically.

Admin Options: this section provides limiting criteria for specific users, trusted user status, vetting status, and validity.

Best Practices

Vetting Results

Depending on the volume of results being submitted to Litmus, it is a good idea to look at test results marked as either failed or unclear every few days. This practice of vetting test results gives us (and the commuity) more confidence in the results because we are actively checking submissions and following up on failures.

To vet a result, click on the test result ID link which will take you to the test result details page. From there, check whether the result looks valid (you'll have to use your own judgement here). Check the 'Valid?' checkbox and click 'Vet Result.'

If a failure is new and valid, make sure a bug is on file to track the failure. Once you have a bug number for the failure (whether existing or new), you can add the bug ID to the regression bug ID field for the testcase. That way the bug ID will appear with every test result automatically. For transient failures with known bugs or side effects from other bugs, it makes more sense to add bug ID references to the test result itself.

By default, only test results that are marked as valid are included in search results. My marking results as invalid, you effectively remove that noise from your reporting.

Fixing Testcases marked as Unclear

If we're lucky, the reporter will have left a comment explaining what he/she found unclear. Check the comments and see whether you agree with the reported. If more than one person marks a given testcase as unclear, we should update it ASAP or consider disabling the testcase. Often the best solution is to include a link to an external information source (such as the Knowledge Base) rather than trying to explain a feature inline.