We hit another milestone this week. After 24 weeks, 12 Betas, and 3 RCs, Firefox 25 was tested, signed-off, and shipped to the general public. Since Firefox 10 I’ve been collecting data using the bugzilla status flags in an effort to determine what impact our policies and efforts are having on the quality of the product. I’ve decided it would be good of me to move that project over to my blog. For one, it makes this data a little bit more discoverable. For another, this will give me an excuse to do something I don’t have a good track record of following through on: blogging.
The Data
Before I go into the numbers let me state that I have no background in Metrics or Statistics, whatsoever. I will not, nor can I make any conclusions about the data. I am merely presenting it here to you, the community, in a way I feel to be interesting. Feel free to comment on this or any related posts if you have any suggestions about how I may improve my methodology or to make more accurate, informed conclusions.
With that, here are the numbers for Firefox 25. Anything in green is a success that we need to continue to build on. Anything in red is an area that needs greater attention as we move forward.
- 382 verified fixes (1% improvement)
- 649 unverified fixes (14% improvement)
- 439 unfixed bugs (14% improvement)
- 245 wontfix bugs (10% improvement)
- 129 unconfirmed bugs (34% degradation)
We’ve made continuous improvement to our processes around fix verification. This includes focusing more on high priority bugs in the pushlogs, verifying fixes earlier in the cycle whenever possible, documenting our processes, and trying to involve more volunteers. tells a story that I’ve seen repeated the last few releases. The addition of a third QA release lead (Tracy Walker) has allowed us to scale somewhat in the last couple of cycles (we are now spending more time on Aurora than we were before). I suspect the move to two Betas per week has also contributed here in that it’s allowed us to do more focused testing on fewer changes more frequently. Unfortunately we aren’t putting enough effort into unconfirmed bug triage, something I hope to improve upon as I switch focus to Firefox 28 next week.
The Data Visualized
Now I’d like to share the visualization of this data. Please keep in mind that I’m not drawing any conclusions here, I’m merely visualizing the data in ways I believe to be interesting. I hope you do too.
Breakdown of bugs by status per Firefox version
Number of fixes landed in each branch for each Firefox version
Percentage of fixes verified prior to release for each Firefox version
Active instances of Firefox compared to unconfirmed bugs shipped each release
Active instances of Firefox compared to number of fixes verified in that release
More to Come
That’s Firefox 25 from my perspective. I hope you have found this interesting. I will continue to share this data and update the visualizations every six weeks. You are encouraged to provide me feedback and ask questions on the data, the comparisons I’ve visualize, and my methodology.
Until next release, enjoy.