Late last year I asked the QA community to take part in some surveys to get feedback on their experience with contributing to Mozilla QA projects. The first survey was designed to identify “common knowledge” about becoming involved with Mozilla QA. The second survey was targeted toward people who had stopped contributing. My original intent was to just let the surveys run for a few weeks but I never “officially” wrapped them up. I thought it would be interesting to have a look back on the results.
As it turns out, people have continued to submit feedback, even as recent as a couple of weeks ago. We’ve managed to collect over 70 data points and I think it reveals some interesting data about our community that I would like to share with you now.
The Findings
1. 64% of respondents are “newcomers”, having participated in the community for the first time in the last few months. To me this shows we do a good job at getting people to come but a poor job at getting them to stay.
2. The most common contributions were running tests, filing bugs, and verifying fixes. The least common contributions were documentation, writing tests, and submitting code. To me this shows that we need to do more to showcase the ways outside of “testing” people can be involved with QA.
3. The most commonly identified teams were Desktop, Automation, and Mobile. The lease commonly identified teams were Services, WebQA, and Browser Tech. To me this shows that we need to do more to showcase that helping with QA is more than just Firefox.
4. The most commonly identified motives for joining the community were to support Mozilla’s mission, make Firefox a better product, and learning new skills or improving existing skills. We need to find a way to better utilize that passion; let it be the wind in our sails.
5. The most commonly identified motives for ceasing contribution was not having enough time, difficulty discovering what to do and where to get help, and a general feeling of being unappreciated. 1/3 indicated they had a “bad interaction” with someone in the community. We need to do more to identify, admit, and fix our failures.
6. The most commonly known entry paths were IRC, QMO, and the Firefox first run page. The least commonly known entry paths were newsletters, social media, and mailing lists. I’m inclined to think that we should put less effort into the channels that aren’t being utilized; use them to bring people to QMO and IRC. On the other hand, maybe their under-utilization is due to us being spread too thin and not being able to give them the attention they are due . If we put a little more effort here maybe we can convert them into community-driven resources. Maybe it’s even time to find a some volunteers who have displayed a prolonged commitment to the community to take over management of the lesser known channels.
7. A side-effect of the surveys was that it gave contributors (new and past) an open opportunity to give us constructive criticism about our tools. Here are some of the highlights:
- It’s too hard finding ways to get involved on QMO
- Bugzilla is intimidating and makes me fear of making a mistake
- Lack of responsiveness makes the mailing lists a cold place sometimes
- IRC can be a bit too quiet at times
- Litmus does not make it easy if I just want to run tests
Conclusions and Recommendations
I can’t thank the community enough for taking their time to provide us their experiences and feedback. They have enabled me to come to some personal conclusions and recommendations about how we can improve.
- Come up with a way to either make our tools more usable and less daunting, either by adopting new tools or add a “usability layer” to existing tools
- Do a much better job in terms of appreciating the work of our community
- Promote civility (and denounce incivility) in everything we do and give people the benefit of the doubt where possible
- Break our contribution opportunities down to a size that’s easier to swallow; especially for those which are targeted at bringing in newcomers.
- Promote the fact that QA is more than just testing, and Mozilla is more than just Firefox
- We are doing a good job at bringing new people in; but a poor job at keeping them interested and involved
- We don’t need to give people a tangible reason to contribute; the feeling of appreciation, that they are making a difference, that they can learn something new is often reward enough. We need to do that better and more often; showcase it to the rest of the Project and the rest of the world.
- Our less used channels (first run page, social networks, mailing lists, newsgroups, etc) should be used to draw people to our primary channels (QMO, IRC)
- Identify and enable those with a prolonged community commitment to take up stewardship of areas of involvement, communication, and contribution
All of this said, I think it’s obvious that there is a lot of room for improvement. I think the best way we can ensure success is to pick a couple of these to focus on and give our all. I think we fail if we try to work on all of these at once.
Call to Action
My call to action for those of us is to make a commitment to incorporate one of these recommendations into our projects, thereby making them more open to the community. Consider laying the ground work to enable community members to chart their own path from newcomer to active contributor to owner.
My call to action to those of you who’ve been around the community for a while, take the next step toward stewardship and leadership. If you are someone with the skills, knowledge, and experience to make it happen take that next step. When you see an area of need jump in, talk to people, and get involved.
My call to action for newcomers and those thinking about joining the community is to push us to do a better job by jumping in, asking questions, offering to help, and providing us feedback along the way (constructive criticism included).
My call to action for those who have left the community is to check back in from time to time. We’d love to have you back if you’ll give us another chance. If you feel you were done wrong in the past, let us know; your feedback will prevent us from making the same mistake and benefit the entire community.
Thank you to everyone for your continued participation, contribution, and feedback. I’m looking forward to growing Mozilla with all of you for many months and years to come.
Feel free to contact me if you have any questions and feedback, or if you just want to talk about community, getting involved, and Mozilla.