Improving Litmus
Litmus is our online test case management tool for manual test cases. We'd like to start a project to redesign the tool around the current needs of the disparate teams that use it. To do that we invite your feedback and your help. Everyone is welcome to join this project and help out. If you join the project, then you'll get special notices about progress, design reviews, design brainstorming sessions, and proof-of-concept availability.
Project Owner:
ctalbert
Project Details
Our goal is to have a test case management site that is useful for the current set of teams that use it, which are the QA Teams from various Mozilla projects and Localization teams on various Mozilla proejcts.
To ensure that we have all our concerns addressed, I invite people to contribute to the design process. We will be discussing the Litmus redesign on the Mozilla QA Newsgroup (mozilla.dev.quality), which can be found here or here. Discuss your ideas there. Then as we collect nuggets of user flows and design ideas, we'll be moving that information to the wiki pages below. You are also welcome to leave your ideas on the wiki pages themselves, but it's much harder to have a discussion about things on the wiki pages, so I encourage people to be brave and submit their ideas into the newsgroup first.
If you do add an idea to these wiki pages, be very very specific. Things like "improve feature x" are really not good enough. We need to know how to improve feature X, we need to know what it is about feature X that is sub-par, and we need to know what other nifty things should feature X provide. If you feel like your comment is too specific, then you are doing it right.
- Things I Love About Litmus - is there anything you just love about the current tool? Let us know so it doesn't get lost in the shuffle.
- Things I Can't Stand About Litmus - What should the new version do better than the old one? What pitfalls should the old version avoid?
- User Work Flows - What types of things should the user be able to do with Litmus?
- Adminstrator Work Flows - What kinds to things should a Litmus administrator be able to do?
- Localization Work Flows - What do Localizers of the tool need - for localizing the tool itself, for localizing the testcases? And for Localization testing, are there any special considerations different from normal User Work Flows?
- Other Stuff - Got a killer feature that doesn't fit into any of these categories? Put it here and we'll decide what to do about it.
- Mock-ups - Have a mock-up? Put it here.
How you can help...
Sign up on the project to get updates, join the discussion on the QA newsgroup.
Upcoming Events
| Dec 4 | Testday: QA Weave Beta Join the MozQA Community in testing Weave! Here's some of t… |
| Dec 11 | Testday: Testing SeaMonkey 2.0 b2 Join the MozQA Community in testing SeaMonkey 2.0 b2! ======… |
Feedback and Discussion
6 Comments
Post a Comment
whimboo
July 2, 2009
Mozmill
We need a field where we can specify a Mozmill test which covers this test. So we have a x-reference between both systems.
coop
June 26, 2009
Give users more incentive to do further testing
One possible model to emulate for the next iteration of Litmus is http://www.thesixtyone.com/.
This music site gives small, incremental rewards that are essentially meaningless but provide a way for users to increase in standing within the community and encourages them to continue using the site. Think achievement score.
This suggestion comes from beltzner, but I enjoy the site too.
We probably wouldn't want this sort of functionality to get in the way of focused testing, but it might encourage the more casual testers we've been hoping to attract for a while.
tchung
July 2, 2009
would it distract results?
thesixtyone.com is a great site, and i find myself wanting to bump my level. However, i sometimes find myself wanting to race through the song so i can hurry up and click that heart, instead of really trying to listen and enjoy the song snippet. My worries is similar impatience could lead to folks hurrily clicking the testcase "PASS" checkboxes, just to get their status up, which could potentially miss an actual testcase that may have failed. But some way to incorporate both ideas would be excellente.
coop
July 3, 2009
Potential for abuse; automated validation; establishing trust
We have always faced the possibility that someone *could* come in to Litmus and flood it with bogus results. In the 3.5 years that Litmus has been around, this has happened once, and even then it was a relatively small number of results and not immediately obvious.
It's a precarious line to walk. The more you open the system up to more casual testers, the more checks and balances (hopefully automated) you need to ensure the veracity and quality of the results. I wrote a similar test reporting system for a previous employer that was more focused on automated results. I cannot stress enough how important it is to have automated systems to validate results and look for potential abuse. Humans that sign up (or are signed up, natch) for this work burn out fast, speaking from experience.
The current system offers no incentive to casual testers, so I guess it depends on how badly you want to harness that resource. There are conscientious people in the community who will run testcases properly on their own, but a little enticement wouldn't hurt.
The way we avoid the abuse issue in the current system is codified in the concept of trust. Trusted testers (MoCo employees, Tomcat-wannabes) can be flagged as such so that their results can be queried separately from those submitted by casual testers.
Casual testers *can* be useful, but IMO their importance increases with volume so that some useful analysis can be done on their aggregated results, i.e. one failing result is less interesting than 10 failing results for the same testcases (just like crash-stats).
juanb
February 5, 2009
Existing Bugs
We could take into consideration the existing Litmus bugs in Bugzilla. Some of those look very useful, as they address some use case scenarios.
tchung
February 5, 2009
Link to those existing bugs
...which are found at http://tinyurl.com/4c2n6h. There are about 50 bugs in that query. Some are already covering the redesign ideas.