Another quarter has come and gone.
I’m writing today to report on the activity in the Mozilla QA community by way of testday statistics. Before I run down the statistics I would like to thank David Burns for setting up the testday bot in our #testday IRC channel. Gathering the data needed would have been far more labour-intensive, otherwise.
Q1 2012 Testday Statistics
A brief explanation of the methodology. The testday bot records the number of unique visitors and the number of messages they post in the #testday channel.
Bugs are tracked using a query. For every bug we file, resolve, verify, or otherwise comment on, we add a whiteboard tag [testday-YYYYMMDD]. This makes it easy to track after the testday in Bugzilla long after the event is over.
Active Testers are those people in the channel who are “active”. I define active as being anyone who has commented 5 times or more. This allows me to filter out nicks who autojoin and don’t really participate.
Success
I noticed the most successful testdays were events which focused on multiple new features; while our least successful testday was one where scope was too narrow and technical:
- Firefox Beta: 16 active testers, 18 bugs
- Firefox Affiliates: 15 active testers, 12 bugs
- Firefox Mobile: 12 active testers, 12 bugs
- Silent Update: 3 active testers, 0 bugs
I think this shows that people (especially newcomers) are drawn to testing something new and that testdays make a great entry point for those less technically inclined. It’s a great way to learn new, fundamental skills and to play with something new at the same time.
Growth
I’d like to make conclusions based on growth over time. Unfortunately, I don’t have really good data on 2011. I plan to continue recording this data. My hope is to be able to start doing some trend analysis in July. That said, we averaged 12 active testers and 5 bugs this quarter. Based on personal experience, I believe that is par for the course. I’d like to see if we can improve upon that in Q2.
What’s Next
I’m currently engaged with the Mozilla Metrics team to develop a web-based dashboard to give us greater visibility into the metrics of testdays. In the future, we will be able to generate reports and identify key focus areas for growing the community and participation. My hope is that a proof-of-concept can be set up by end of Q2.
A big thank you to all those who took time to contribute and participate in testdays this quarter. I look forward to working with more of you over the next three months.
As always, keep reading QMO to stay apprised of upcoming events. If you want to talk more about testdays, getting involved with Mozilla, or if you are a developer wondering how testdays can benefit the work you do; please contact me.