Today I finally finished reviewing the hundreds (yes, hundreds!) of endurance reports that were submitted on our Firefox 4 add-ons test day last Friday and on the days following. It was amazing to see so many reports coming in, and I would like to thank everyone that ran an endurance test run. By far the most active contributor was pxbuz, to whom I’m extra grateful!
Of all of the test runs, it turns out there are three major issues discovered:
The first is that we really need to improve the reporting system. Going through the results was a long and tedious job, so I will be thinking about how I can improve that experience.
Secondly, we need to come up with a way to dismiss any modal dialogs that add-ons might show on first run. There were a couple of these that resulted in what looked like memory leaks, but turns out would be impossible to replicate manually.
I saved the best for last – we found a memory leak when the Greasemonkey add-on is installed! It seems that when entering/leaving private browsing mode there is memory allocated but not released. A bug has been raised and hopefully it’ll soon be resolved. Greasemonkey is one of our most popular add-ons with a current average active daily usage of over 2.5 million users!
Below you can see how the memory leak was spotted. On the left is an example of an endurance test without any add-ons installed, and on the right is a test run with Greasemonkey installed. Those five spikes that start around the 500 checkpoints mark occur during the private browsing test.
You can see the actual reports