How to create a Testcase
You want to write/update testcases for Litmus ?
Great ! Here are some things you might need to know:
How to write testcases:
Testcase steps
Sample Test Cases
How to submit testcases
Changing testcases
Other Mozilla QA Activities
This documentation can also be found under http://wiki.mozilla.org/Mozilla_QA_Community:Test_Case_Writing_Day_Guidelines
OVERALL CAVEATS FOR WRITING TEST CASES FOR LITMUS
- 1. Be concise with your wording
- 2. Make the steps easy to follow
- 3. Be descriptive if necessary
- 4. Don't get hung up on terminology
- 5. Have a clear set of expected results
- 6. Write your test cases so someone unfamiliar with a browser or email client could understand the
- 7. When writing, keep your audience in mind
The set of steps that you will create when writing a testcase are
similar to the "Steps to Reproduce" that you might use when filing a
bug (some of you might be familiar with this if you use Bugzilla). When
writing a test case, here are some helpful guidelines to use:
TEST CASE TITLE:
When creating a title for a test case, try to drill down and be as concise as possible.
Good examples would be:
- Adding a Bookmark
- Open Search Using Keyboard Shortcuts
- Restore Home Page to default
Bad Examples would be:
- Home Page
- Help
- Fonts
The bad examples don't give enough information about what part of
help, home page and fonts will be tested. A better phrase for both
might be:
- Setting a Home Page
- Opening Mozilla Firefox Help
- Increasing Font Size Using Keyboard commands
Testcases in general:
Litmus testcases consists of 2 parts (tests to perform and expected result)
* The minimal set of steps necessary to test the feature. Try not to be too wordy here - emphasis is on being concise!
* When writing, think about the step by step process that will get you to the end game.
* When I write test case steps, I often script out the basic steps first and then do several iterations to firm them up.
* When writing a test case for a new feature, it is best to test all
aspects of it first and get an understanding of how it works first
before writing the test cases. So make sure to launch all windows and
click all radio buttons to understand what is supposed to happen.
* You can storyboard the screens if that is helpful. One way it so to take screenshots and then write your steps below them.
* If there are special setup steps, make sure to include them at the
top of the test case. An example of this would be: If you are asking a
tester to test a feature that requires a fresh profile, make sure to
state that at the beginning of the test case. Otherwise the tester may
test using a profile that already exists.
* Remember that Firefox and Thunderbird products run on three platforms
(Windows, Mac and Linux), and there may be slight differences in
behavior between the various platforms. If you have the ability to
check the other platforms when writing test cases, that is great.
Otherwise, when you submit your test case note that it has only be
written for Windows and a QA team member can tweak it so the verbiage
works for the other platforms.
* What the application or feature should do when it is invoked by the user.
* Try to avoid using terms like "Expected behavior should be observed."
It is better to phrase the expected results so the person testing will
understand what is supposed to happen.
EXPECTED RESULTS:
- What the application or feature should do when it is invoked by the user.
- Try to avoid using terms like "Expected behavior should be
observed." It is better to phrase the expected results so the person
testing will understand what is supposed to happen.
EXAMPLE 1:
You are asked to write a test case to test the functionality of adding an icon to the Firefox toolbar.
When writing this test case, you will first need to think about
the series of steps you would need to take to add a toolbar icon. When
flushing this out, you also need to remember there are various ways to
invoke the "Customize Toolbar" User Interface, and you might need to
consider that fact when writing the testcase.
First, let's start with a minimal set of steps and expected results to get us to the additon of a toolbar icon.
Title: Add an icon to the Firefox toolbar
Steps:
- 1. Right click on the toolbar, and select "Customize" from the dropdown menu.
- 2. A "Customize Toolbar" dialog box containing icons/spaces should be shown.
- 3. Using your mouse, select an individual icon.
- 4. Drag the icon to the toolbar and place it to the right of
the URL bar (between the end of the URL bar and the beginning of the
search box). - 5. Click "Done" to close the Customize Toolbar dialog box.
- 6. Restart Firefox.
Expected Results:
- 1. The icon should be added to the toolbar in the place that you specified.
This is an example of a basic test case. But there are a number of assumptions that are not explicity called out there, such as
• other ways to get to the customize toolbar besides a right
click (it is important to test these as well - menu and keyboard
commands). It is not always necessary to include these in the test
case, but it is helpful since it widens the swath of what is tested
since users may come at the feature from different angles. Note that
there are test cases in Litmus that do cover the keyboard
functionality, so even if you don't include it in your test case all is
not lost.
Further, the test case stands on its own as a test to whether
you can add a icon, but what about the functionality of the actual icon
once it has been added (such as when you click the Print icon, does it
actually launch the print dialog)? It is often useful to write test
cases that "kill two birds with one stone", and simply adding a line
that says "Click on the toolbar icon you added and confirm that it
launches the expected functionality" will make the test case solid.
Note that this won't apply to every test case, but is just so happens
that in this example there is great benefit from the addition of that
extra line. Think about this kind of thing when you crafting your test
cases. You don't want to include everything in one test case, but at
the same time it behooves you to include something that will make the
test case useful in more ways than one.
Here is the test case, rewritten to include testing the functionality of the Print icon.
- 1. Right click on the toolbar, and select "Customize" from the dropdown menu.
- 2. A "Customize Toolbar" dialog box containing icons/spaces should be shown.
- 3. Select the Print icon using your mouse.
- 4. Drag the Print icon to the toolbar and place it to the
right of the URL bar (between the end of the URL bar and the beginning
of the search box). - 5. Click "Done" to close the Customize Toolbar dialog box.
- 6. Restart Firefox.
Expected Results:
- 1. The Print icon should be added to the toolbar in the place that you *specified.
2. Clicking on the Print icon should launch the Print dialog box.
Note that is is not necessary in include Step 6, Restart
Firefox. But the benefit of this step is verifying that after a browser
restart that Firefox is honoring the preference you have set.
Example 2:
Title: Drag/Select a Web Page
Steps:
- 1. View any web page.
- 2. Drag-select the page.
(Specifically, while holding down the mouse button, drag the mouse
pointer downwards from any point in the browser's content region to the
bottom of the browser's content region.)
Expected Results:
The window should scroll downwards. Scrolled content should be selected.
How this test case could be improved:
This test case is a good first cut, but could still stand some
improvement. For example, in the expected results, will a user
definitively be able to identify how content is selected? Usually when
the content is selected, it will be highlighted in a color. That would
probably be something useful to include.
Also, this test case demands a page that has a scroll window,
so it would useful to provide a URL for the user so they can navigate
quickly to a page that has enough content to be scrolled.
Creating a Variation of this test case
If I wanted to create a variation of this test case, I might do something like this:
Title: Drag/Select a Web Page using "Select All"
Steps:
- 1. View any web page.
- 2. Right click in the content pane and choose "Select All."
Expected Results:
All content on the page should be highlighted. Note: The color of the highlighting may vary depending on your system settings.
I wrote some testcases, how to i submit them to litmus ?
Please submit your testcases to testday@mozilla.org for review. Someone from the Mozilla QA Team will review your testcase and also checkin this testcase into litmus..
The Mozilla QA team is ready, willing and able to help out in our IRC channel if you need assistance
If you do testruns on Litmus and you notice a wrong/updated/ or a testcase with errors, you have 2 choices:
1.) you mark this testcase as unclear (please add a useful comment
2.) You can rewrite this testcases (see the examples above how to write testcases) and send this testcase to QA.
We appreciate every help for make Litmus and its testcases better.
Great you want to help to improve our testcases on Litmus.
Hopefully you can join also on our other Mozilla QA Community Activities
Mozilla Community QA Testdays : We test new releases and build. A great
place to test, to meet people from all other the world and to have some
testing fun. Our testdays are announced on the Mozilla QA Blog http://weblogs.mozillazine.org/qa/
Bugdays: Bug Days are an excellent way to contribute to Mozilla's QA efforts. It
doesn't matter if you've been involved with Mozilla for years or if
this is your first step into our amazing open source community. Anyone
can participate and be a valuable contributor. We gather together at
predetermined times to work on areas in Bugzilla. The exact area or
topic for Bug Day may vary week to week. But the general idea is to
work through some sort of bug list as a team.
