The implementation of Media Source Extensions (MSE) to support YouTube videos running in HTML5 has begun. We have enabled MSE in Firefox Nightly for YouTube on Windows and Mac, and over the past couple of weeks Mozilla engineers have been … Continue reading
Posts from 2014
I’m writing to announce that the testday last Friday marks the final Testday for 2014. Due to upcoming holidays, we will not be organizing any more events for the remainder of the year. As the year wraps up we will … Continue reading
Hello everyone! As you already know, Friday, December 12th we held the Firefox 35.0 Beta 3 Testday. We’d like to take this opportunity to thank everyone for getting involved in the proposed testing activities and in general, for your hard work in helping us make … Continue reading
Hello Mozilla QA Community, I’m writing today to seek your assistance. Another year is coming to a close and we’ve seen no significant uptick in participation. Later this year we made a focused attempt to relaunch the Testdays program with … Continue reading
As a kid, all Craig wanted to do was draw comic books. He took art classes in high school and learned about printing at a vocational school. In college at Pittsburg State University, Pittsburg, Kansas, he first majored in Fine … Continue reading
WHAT DO YOU DO AT MOZILLA? I’m the resident pointy-hair, aka administrative overhead — and some people even call me “manager.” Our team’s most up-to-date and complete page is on the Mozilla Wiki, here: https://wiki.mozilla.org/QA/Execution/Web_Testing. As our team name implies, … Continue reading → Continue reading
From October 24-26 I participated in the Mozilla Festival in London. I was interested to attend the event for a few reasons, the first being I had never attended the event, and the second was to interact with the Mozilla … Continue reading
Hello everyone! Friday, November 10th we held a Firefox 34.0 Beta 7 Testday. Thank you for helping us on doing some verification testing and updating some of the bugs. More details about the work done can be found in the etherpad. Many … Continue reading
Firefox OS QA held a Bug Bash on Thursday, October 23 2014. Participants from all over the world were invited to test the latest build of Firefox OS v2.1 on their Flame devices and report any bugs found. Coordinators in … Continue reading
Many of Mozilla’s web projects have adopted concepts from continuous delivery.
Once a code change lands on a master branch it is shepherded to production with
little-to-no human intervention. Quickly getting new features out to users.
While the implementation details vary from project to project, the core concepts
tend to remain the same between projects. It is important for project stewards;
project managers, developers, testers, UX, community, etc to have a shared
The purpose of this document is a starting point for this conversation.
Why we do this
We find that being able to move quicker allows us to create higher quality
software with limited resources. This fits our goal of creating features that
our users want, working in the open, and experimenting with our development
Continuous delivery has worked for us because of the culture of our community.
We accept shared responsibility for the quality of our work, are strong
collaborators, and have a culture that is supportive of learning from mistakes.
Over time we’ve become more confident in the quality of our work. Moving quickly
allows us to take a more proactive stance towards the quality of the products we
create as well as mitigating risk when creating new features.
What are QA’s responsibilities
- Acting as a user advocate
- Leading discussions that identify and assign risk to user stories
- Driving exploratory testing efforts
What are the dev team’s responsibilities
- Developing new features for the project and implementing fixes for issues
- Developing unit tests
- Maintaining code quality via code review and documentation of standards
** project management, developers, qa, UX, l10n **
- Deep knowledge of the application and the ability to identify areas of risk
- Provide feedback on feature definition to the project manager and team
- Identifying, creating, and maintaining the end-to-end test automation
- Developing the appropriate mitigation strategies to lower risk of regressions landing in production
- Staying current on best practices and pushing the team to explore the applied relevancy
- Continually discussing and fine-tuning test coverage
- Opening avenues for community collaboration on the project
How we do this
The basic concept is that any code that lands on the master branch should be in
a shippable state. New feature work should be categorized by the
amount of risk the feature or change to the code base poses.
The team discusses and agrees upon a risk metric that they are comfortable with
assuming. Assumed risk isn’t a static conversation, but it is good to build a
skeleton that the team can agree on and enhance over time.
As a product should be broken into user stories so a team can apply a
hierarchy of risk. It is important to identify and engage as many stakeholders
as possible, each individual will have a unique and important point of view.
The list of stakeholders often include; project managers, developers, testers,
UX, IT, DBAs, the security team, etc. Sometimes the product has high enough
visibility in the organization that a proxy for the CEO is beneficial.
User stories and features are grouped into three categories
- Features that users should never see broken in production
- Features that can go to production broken but need to be discovered and fixed within a specific time period
- Features that can regress on production and will be fixed once a user identifies the problem
An example from mozillians.org
- The ability to create a new account should never go to production broken
- The ability to edit the profile of an existing account can break but we want to discover and patch the problem within 1 hour of introducing the regression
- CSS layout can regress on production and will be fixed when a user uses the in-site feedback form to notify us
Not a dogmatic rule, in general new features or defect fixes should be manually tested and verified. Exploratory testing is the rule. Depending on the risk assigned to a feature, projects have the option of using waffle flags. The use of feature flags allows us to hide features from the general public while allowing a team or a group of beta testers to test on production with real data.
An important note about feature verification
If a feature is deemed as low risk and it’s functionality is easy to verify, anyone on the team is
empowered to test and verify it. If it is a feature that requires deeper investigation and poses
high risk, the QA team is the group who verifies it. If feature set is big or and the use workflow
is changed it is important to engage the test team to flush out defects and usability concerns.
Feature work that falls into the category of “better never break” should see a heavy amount of exploratory testing before it is exposed to the public. Once the feature is baked and code churn has decreased, we seek an appropriate level of coverage using Selenium, Python’s Requests API, load testing, and fuzzers.
A component of the QA team automation strategy should also includes reviewing the amount of test coverage that follows a developers pull request as well as taking part in the code review. Often times the test team is able to influence and enhance the depth of coverage by simply asking for more developer level tests [Unit + Integration].
Sites should always have an easily discoverable method for users to submit feedback and file bugs. The team must be religious about responding to requests for help in a timely manner and do so with empathy. .. do not respond to a user with “invalid” or “works as expected.”