Posts from June, 2010

Last month I blogged about the future of bugdays. In this blog post, I detailed the reasons I believed Bugdays were valuable and why I thought Bugdays might be getting a bad rap as a potential waste of resources. This was followed up by very worthwhile discussion on the same topic. I received quite a lot of feedback, most on the side of keeping but tweaking Bugdays. I’ve now had a couple of weeks to digest all of that feedback and have come up with some conclusions and questions we need to be asking ourselves.

First of all, I’d like to thank everyone who participated in this discussion. If there is one thing this discussion has proven, it’s that there are a lot of people who believe in the community and believe in getting the community involved. Comments came from many different perspectives. We had veteran members of the Mozilla project, developers, management, marketing, QA, community members, and even people from other open source projects (GNOME and Seamonkey). It’s great to see everyone get involved and contribute their unique perspectives to a discussion.

Conclusions

Over the last couple of weeks mentally processed the discussion, feedback, and suggestions from everyone. Using this feedback and mulling over my own opinions, I’ve come to several conclusions about bugdays:

1. Bugdays have always been, since the beginning, an important and valuable entry point to Mozilla. Many of Mozilla’s successes have come from people coming up through the community. Some of these people originally became involved with Mozilla through Bugdays. It serves as a gateway for people to learn and hone their skills in testing, development, bug triage, and investigation. Some of these people become more deeply involved in QA or move on to development. In turn, these people become mentors within the community. It is a perpetual cycle. Bugdays have always been, and should continue to be, a part of this cycle.

2. Bugdays about “bugzilla maintenance” are boring and yield little value compared to the effort required. It comes as no surprise that we see the lowest attendance and least success from “help us clean up X” bugdays. Most people who come to QA want to help test and fix. People want to be a detective, engineer, or tester; not a janitor. We need to curb the practice of running maintenance bugdays.

3. A more holistic approach to Bugdays and Testdays needs to be adopted. As Tracy Walker (QA) puts it, “Testdays are about feature testing with a bit of bug investigation. Bugdays are about bug investigation with a bit of exploratory testing.” This has always been the root approach to community events. In the beginning this was a very successful model, but somewhere they lost there way.

4. We need to foster a dedicated group of helpers; a pool of people on which we can rely. These should be people from within QA, within Mozilla but outside QA, and within the community. There needs to be leaders, feature experts, mentors, evangelizers, and participants. This group of people will help with the planning, communicating, and participation of all bugdays and community events.

5. Bugdays have failed to evolve with the Firefox project and the community. Bugdays were certainly successful in the past but the project was very different back in the beginning. Bugdays have changed little over the years even though the project and community have changed radically; in size, scope, and needs. This fish needs to grow some legs and lungs and walk out onto the land. What we really need to do here is learn from the past so we can design a successful future. We need to see what worked and what didn’t based on the context of the project in the beginning. We need to see what works now and what doesn’t based on the current context of the project.

6. We need to evangelize the concept of “Bugdays as a Tool”. We want developers to come to us with features to test and bug fixes to verify. We want them to see QA as a resource available to them, not just a mystical entity between them and a Firefox release. Bugdays and Testdays should be thought of as part of a feature release. Consider these two scenarios:

“I’ve just landed feature_X. QA, can you organize the community to test this feature?”
“We’ve just landed these fixes. QA, can you organize the community to verify these fixes?”.

This is where we want to be, not:

“We’ve just landed feature_X.” [pass ball to QA]
“We’ve just landed these fixes.” [pass ball to QA]

7. We need to do a better job of defining Bugdays and gauging success. We have to rethink our goals, define our audience, who are the stakeholders, what are the wants and needs of those stakeholders, how do we define success, what are the accurate metrics to measure that success, how do we track and visualize those metrics.

Speaking to the metrics side of Bugdays, we currently plan and track bugdays on a wiki. This is essentially a braindump and not very useful in tracking and gauging performance.

Speaking to the audience and stakeholders side of Bugdays, I think we typically assume a broad audience; it is not very well defined. I think we tend to assume developers of a particular feature, people cc’d on bugs, and the community at large is our audience. While this may be true in most cases, it’s hard to design a successful bugday if you don’t know exactly who makes up your audience. I believe consideration of desired outcomes, skill-sets, knowledge, roles, and participation of all stakeholders can go a long way to designing better bugdays.

8. QA needs to do a much better job with it’s level of commitment to Bugdays. There has been a lack of planning, communicating and participating in Bugdays from QA’s side if the house. That’s not to say the effort has not been there. The success we have seen has typically due to the efforts of very few participants. In the most recent bugday QA participation was extremely low. In fact, I can’t think of a single member of the QA team who actively participated in the bugday, excluding myself. QA should do whatever it takes to achieve a high level of standard for planning, communicating and participating in all community events. If this means hiring a person who’s sole responsibility is Bugdays and Testdays, then so be it.

9. We need to identify barriers and tear down these walls. For example, anybody can create a bugzilla account and file a bug. However, a user needs elevated permissions to be able to edit bugs. This does not usually pose a problem when it comes to Testdays. It creates an obvious problem to Bugdays as the primary task is triaging bugs. Not being able to edit bugs makes bug triage a painful task. It essentially makes those with permissions a proxy for those who do not. An all-or-nothing approach is not very useful. Perhaps we need an intermediate permission level to allow editing of some fields and not others. Not being able to edit bugs is a big barrier, detrimental to the enticement for participation in Bugdays.

Questions

Given this discussion and the above conclusions, we need to go back to the drawing board. Here are some questions I think we need to ask ourselves:

1. Benefits:
i) What are the benefits we want to achieve?
ii) How do we prioritize those benefits?
iii) What about bugdays today helps/hurts us in achieving those benefits?
iv) What activities would yield the most benefit to the project?
v) How do bugdays compare to other activities considering those benefits?
vi) How do bugdays make product_X/feature_Y better?
vii) What topics yield the most value:effort?

2. Communication
i) What is the message we currently communicate to the community and Mozilla with bugdays
ii) How do we communicate that message?
iii) Is this the right message? If not, what message should we be sending?
iv) Is this message being communicated effectively? If not, how can we improve?

3. Stakeholders
i) Who are the stakeholders?
ii) What do they like about bugdays?
iii) What don’t they like about bugdays?
iv) What are their strengths, weaknesses, and needs, and how do they map to bugdays?
v) How do we identify people who can help?
vi) What roles can certain people play in fostering and mentoring the community?

4. Metrics
i) What are our current metrics for success?
ii) Are they effective metrics? Why/why not?
iii) Are there more effective metrics for success? What are they and how are they more effective?
iv) How do we track these metrics? Is it effective? Why/why not?
v) What are some ways we can improve tracking these metrics (reporting, trends, historical data)?

Suggestions

Finally, throughout the entire discussion, many great suggestions were given; both for communicating and valuable topics. In the near future, I would like to organize a Bugday Brainstorming session in the hopes of addressing some of the conclusions and answering some of the questions above, and to get some suggestions for the future. Here are a few from the participants of the discussion to get you thinking:

What makes a successful bugday:
* unconfirmed bugs confirmed or resolved
* bugs given reliable steps to reproduce
* bugs nominated blocking?
* bugs given investigative information (minimized test cases, regression ranges, crash data, etc)
* improved coverage across a broad audience and multiple platform configurations

Potentially untapped communication channels:
* Flash/Java game user groups and developer forums
* Video site forums
* Flash/Java/Silverlight plugin developer and support forums
* Banner ads on social networking sites (ie. “Tired of ads crashing your browser? Us too! Click me to find out more.”)

Topics
* Triage bugs from Core:General, Firefox:General into correct component
* Triage bugs with patches not flagged for review or flagged for an inactive reviewer, and find someone to review it
* Triage bugs which need STR
* Triage bugs which need a regression range
* Triage bugs which need minimized test cases
* Triage bugs which have patches r+ that were never checked in, setting the checkin-needed flag
* Triage new incoming bugs for large hot-off-the-press features (HTML5 Parser, OOPP, Add-on Manager) and identifying potential blockers
* Verify fixed bugs for large hot-off-the-press features (HTML5 Parser, OOPP, Add-on Manager)
* Evaluating performance
* Workshops for specific tasks (crash testing, regression ranges, bug triage process, how to profile performance issues, etc)

Well, that pretty much sums up my thoughts and conclusions from the bugday discussion. I’m sure many of you have your own thoughts and ideas. I hope I’ve articulated the thoughts and suggestions of everyone who took part in the discussion. I also hope I’ve articulated the thoughts and suggestions on the minds of those who did not take part in the discussion; those who still care about Bugdays and the community but remained on the sidelines holding their opinions to themselves. If you believe I’ve misspoken or misrepresented Bugdays (past, present, or future), I welcome you to engage in discussion via comment on this blogpost.

I’ve set up a brainstorm wiki page. Feel free to put any of your thoughts, ideas, feedback and questions on that page. I’ll be using that page in addition to feedback from this blogpost and the previous discussion to design a formal brainstorming session.

Thanks everyone for your valuable feedback.

I look forward to brainstorming bugdays soon and evolving them into something truly valuable.

Continue reading

The newly redone Quality.mozilla.org (aka QMO), has just rolled out and got a newer and cleaner facelift to the site.   Spearheaded by our very own Aakash Desai and the contribution of work and ideas from many others, we hope version 2.0 of the site will encourage more exposure and usage of those who have an […] Continue reading

So now into the 4th week of the fennec front end automated tests project. The previous week had has seen less activity, but the main highlight was the submission of the first patch related to the project. The patch was related to the basic panning of preferences, like checking out whether the preferences stays the … Continue reading → Continue reading