Thanks to the amazing efforts of the MemShrink project, Firefox’s memory usage is seeing some great improvements. In particular, Firefox 7 will be much more efficient with memory than the current version. As endurance tests monitor resources such as memory, it makes sense for us to work together to ensure that we’re moving in the right direction, and that we don’t regress in any of these areas.
At this point there are only five endurance tests, and although these can be run with many hundreds of iterations in order to seek out memory leaks in the tested areas, they do nothing to simulate a user. It was suggested that we have a special endurance test similar to Stuart Parmenter’s Mem Buster test.
Creating an initial version of this new test did not take long. Instead of opening sites in new windows I open them in tabs, and the number of sites opened is controlled by iterations and micro-iterations. I also increased the number of sites so we’d be hitting the same ones less often, and based this new list on Alexa’s top sites. Once I added in handling of modal dialogs that some sites were causing to be displayed then I was able to consistently get results.
This test would appear to be similar to Talos tp5 in that is loads sites from Alexa’s index, however we’re not measuring how long each site takes to load. Instead, we move onto the next site after a delay as specified on the command line. I have kept the same delay as the original Mem Buster test, which is 3 seconds.
After running the Mem Buster Endurance Test five times across five versions of Firefox, I found the results to clearly reflect the MemShrink efforts. Although the memory consumption varies somewhat for each run, the general downward trend is unmistakable.
In the following charts you can see the improvement in memory usage between Firefox 4 & 5. These can be directly compared as the endurance tests were measuring the same metrics (allocated memory & mapped memory).
In Firefox 6 there were several improvements to memory reporting, and the endurance tests were updated to record new metrics (explicit memory & resident memory). You can see in the following charts that explicit memory usage in Firefox 7 is rough half that of Firefox 6! It appears that this has increased in Firefox 8, which will require some further investigation. The resident memory has continued to decrease in each version.
You can follow the progress of the Mem Buster Endurance Test in Bugzilla. Full reports from the test runs used in this blog post can be found here.
Update: It appears that the explicit memory calculated for Firefox 7 on Mac was artificially low. This explains the slight increase in Firefox 8. If you’re interested you can read further details on Bugzilla.