Posts by Matt Brandt

Continuous Delivery – a generic plan

Matt Brandt

Many of Mozilla’s web projects have adopted concepts from continuous delivery.
Once a code change lands on a master branch it is shepherded to production with
little-to-no human intervention. Quickly getting new features out to users.

Definition of a participant

While the implementation details vary from project to project, the core concepts
tend to remain the same between projects. It is important for project stewards;
project managers, developers, testers, UX, community, etc to have a shared
vocabulary.

The purpose of this document is a starting point for this conversation.

Why we do this

We find that being able to move quicker allows us to create higher quality
software with limited resources. This fits our goal of creating features that
our users want, working in the open, and experimenting with our development
practices.

Continuous delivery has worked for us because of the culture of our community.
We accept shared responsibility for the quality of our work, are strong
collaborators, and have a culture that is supportive of learning from mistakes.

Over time we’ve become more confident in the quality of our work. Moving quickly
allows us to take a more proactive stance towards the quality of the products we
create as well as mitigating risk when creating new features.

What are QA’s responsibilities

  • Acting as a user advocate
  • Leading discussions that identify and assign risk to user stories
  • Driving exploratory testing efforts

What are the dev team’s responsibilities

  • Developing new features for the project and implementing fixes for issues
  • Developing unit tests
  • Maintaining code quality via code review and documentation of standards

Shared responsibilities

** project management, developers, qa, UX, l10n **

  • Deep knowledge of the application and the ability to identify areas of risk
  • Provide feedback on feature definition to the project manager and team
  • Identifying, creating, and maintaining the end-to-end test automation
  • Developing the appropriate mitigation strategies to lower risk of regressions landing in production
  • Staying current on best practices and pushing the team to explore the applied relevancy
  • Continually discussing and fine-tuning test coverage
  • Opening avenues for community collaboration on the project

How we do this

The basic concept is that any code that lands on the master branch should be in
a shippable state. New feature work should be categorized by the
amount of risk the feature or change to the code base poses.

Risk-based assessment

The team discusses and agrees upon a risk metric that they are comfortable with
assuming. Assumed risk isn’t a static conversation, but it is good to build a
skeleton that the team can agree on and enhance over time.

As a product should be broken into user stories so a team can apply a
hierarchy of risk. It is important to identify and engage as many stakeholders
as possible, each individual will have a unique and important point of view.

The list of stakeholders often include; project managers, developers, testers,
UX, IT, DBAs, the security team, etc. Sometimes the product has high enough
visibility in the organization that a proxy for the CEO is beneficial.

User stories and features are grouped into three categories

  • Features that users should never see broken in production
  • Features that can go to production broken but need to be discovered and fixed within a specific time period
  • Features that can regress on production and will be fixed once a user identifies the problem

An example from mozillians.org

  • The ability to create a new account should never go to production broken
  • The ability to edit the profile of an existing account can break but we want to discover and patch the problem within 1 hour of introducing the regression
  • CSS layout can regress on production and will be fixed when a user uses the in-site feedback form to notify us

Feature testing

Not a dogmatic rule, in general new features or defect fixes should be manually tested and verified. Exploratory testing is the rule. Depending on the risk assigned to a feature, projects have the option of using waffle flags. The use of feature flags allows us to hide features from the general public while allowing a team or a group of beta testers to test on production with real data.

An important note about feature verification

If a feature is deemed as low risk and it’s functionality is easy to verify, anyone on the team is
empowered to test and verify it. If it is a feature that requires deeper investigation and poses
high risk, the QA team is the group who verifies it. If feature set is big or and the use workflow
is changed it is important to engage the test team to flush out defects and usability concerns.

Automate tests

Feature work that falls into the category of “better never break” should see a heavy amount of exploratory testing before it is exposed to the public. Once the feature is baked and code churn has decreased, we seek an appropriate level of coverage using Selenium, Python’s Requests API, load testing, and fuzzers.

A component of the QA team automation strategy should also includes reviewing the amount of test coverage that follows a developers pull request as well as taking part in the code review. Often times the test team is able to influence and enhance the depth of coverage by simply asking for more developer level tests [Unit + Integration].

User feedback

Sites should always have an easily discoverable method for users to submit feedback and file bugs. The team must be religious about responding to requests for help in a timely manner and do so with empathy. .. do not respond to a user with “invalid” or “works as expected.”

Continue reading

Mozillians Vouch Refactor – Testday

Matt Brandt

Welcome to the Mozillians project. We love working with old and new
contributors!

The new Mozillians.org

The latest feature set of the community phonebook has landed
and is ready for testing. The phonebook is a place to find, share, and
communicate with open web advocates throughout the world.

The community has worked to find creative ways to make the phonebook a
place where active and trusted memebers of the community can connect. The
latest set of features seeks to push new contributors and returning contributors to be
more active in shaping our community.

The new trust model:

  • Anyone can create a Mozillians account.
  • 1 vouch allows users to search for and view Mozillian profiles
  • 3 vouches are needed to vouch for new Mozillians

A test environment is live on our dev server – mozillians-dev.allizom.org
– and ready for your critical eye. We welcome you to explore the features and offer
your open, honest, and unabashed feedback.

Test Plan

There are two areas that you can get involved with helping test; feature
verification using exploratory testing techniques and test automation coverage.

Setup for manual testing

Here are the unverified features that needs to still need to be tested. Find
a bug that needs verification and begin testing.If you have questions reach
out in IRC – #mozwebqa or #commtools – and introduce yourself.
Myself or another community member will help you.

To get started you’ll need:

  • Disposable email addresses so you can create test accounts on dev I recommend free services like Mailinator or 10minutemail.
  • personatestuser.org is one of my preferred tools if you don’t need to verify receipt of email but it forces you to parse a json file.
  • Create an account on our dev server.
  • Some of the test scenarios will require a vouched account. You can automatically vouch yourself by appending /vouch to the url.
  • For example mozillians.allizom.org/u/your_account_name/vouch.
  • Alternatively you can unvouch yourself by appending /unvouch to the url.

Filing Bugs

  • Write good bugs that provide clear steps to reproduce the problem. Read this document for tips.
  • Use this form to file new bugs.
  • Bugzilla etiquette – be polite and treat people with respect, we are a friendly community.
  • IRC etiquette – same as Bugzilla; relax and have fun.

Setup for test automation

Creating test automation assumes that you have a bit of experience writing code and working with version control.

The new Mozillians.org

Getting started:

We really appreciate your enthusiasm and help with making the community
phonebook better. This is fully a community initiative and wouldn’t exist
without you.

I look forward to seeing you online!

Matt Brandt

https://mozillians.org/u/mbrandt

Continue reading

Commtools – Meetup

Matt Brandt

The Mozilla Community Tools team is hosting a Meetup this week, June 10 – 12,
2014. Prior to the Meetup the community brainstormed goals for the gathering
as well as a loose schedule. Ongoing meeting notes can be found here. If you
would like to partic… Continue reading

Commtools – Meetup

Matt Brandt

The Mozilla Community Tools team is hosting a Meetup this week, June 10 – 12,
2014. Prior to the Meetup the group brainstormed goals for the gathering
as well as a loose schedule. Ongoing meeting notes can be found here. If you
would like to participat… Continue reading

Java Posse Roundup – A winter trek

Matt Brandt

Crested Butte, Colorado

By several orders of magnitude this year’s Java Posse Roundup was more interesting and engaging than the previous year. In large part due to my increased comfort attending an OpenSpace event, primarily, as a participant it is up to me to infuse energy into the event even when I am not a subject expert, let alone a novice on a topic.

Definition of a participant

I’ve learned to respect the law of two feet, to be a participant in each discussion I attend or leave and find a session that does interest me.

Why attend:

I enjoy working with driven people who are drivers in technology who want to push the envelope on technical topics. Pushing the envelope often means making a lot of mistakes while moving forward and trusting the group your working with to support mistake making in order to make relentless forward progress. Each person while not an expert – remember the goal is to break new technical ground – adds a small piece of vital knowledge when working towards solving a difficult puzzle.

I’ve learned over the years that small gatherings of self-driven people are fair more interesting to work with than attending events where I’m spoken at. I find that if I’m going to travel to an event my time is often better invested in events like this one. Events where participants are being spoken at are great for leaning and being exposed to concepts/new tech/etc, however I can usually find the talks online and watch them at my leisure.

To get the most out the Posse Roundup you must participate. This is an event that enables the formation of relationships.

Discussions wanted to have:

  • Continuous delivery vs. continuous deployment: risk analysis and measuring quality
  • Community engagement strategies and measuring success
  • Host a hack session on building and testing Firefox OS apps
  • Learn what others are doing: test plans, manual testing, automation, tools, definitions of community
  • Get involved in other participants hack sessions and be exposed to new frameworks, tools, thought processes, etc
  • Work with the local high school to involve students in the hack sessions
  • Learn how to build a snow cave

Snow cave building

Take aways:

  • Docker – Very cool containers! I have thoughts on how Web QA can leverage this, Docker coupled with PaaS has some interesting potential. Future blog post to come.
  • Continuous Deployment – None of the organizations I spoke with know how to do this well. At best, groups are comfortable with continuous delivery – delivery of artifacts that are in theory ready for deployment at any time. Netflix has convened a new team devoted to exploring this space. Future blog post to come.
  • Rust – Much enthusiasm for this language. Can Mozilla do more to engage the community outside of locations like Silicon Valley?
  • Monads – Definitely wtf .. I have no idea what they are and what the people in the room were talking about, but it was interesting non-the-less.
  • OpenSpaces Board – Rails has definitely matured since I last used it although once we moved into the realm of needing to do interesting things with it – web sockets & click binding – it got interesting. View code we worked on – OpenSpacesBoard
  • What makes a good development process – this was one of my favorite sessions; future blog post to come.
  • Web Mapping & GIS – Storing and querying Lat/Long data is error prone and inefficient; GeoHashing isn’t. The Mozillians project will directly benefit from this. There are many good apis.
  • Building a snow cave – it’s not for the claustrophobic

Session I attended:

(full listing of session)

Monday:

  • Identify sessions and place them into time slots
  • Continuous delivery vs. continuous deployment: risk analysis and measuring quality
  • Take a break and build a snow cave

Tuesday:

  • Working remote; the good, the bad, & the ugly
  • What makes a good development process
  • I hate logins – modern authentication
  • Open Sourcing corporate code

Wednesday Hackathon:

  • OpenSpaces Board web app
  • Introduction to Marionette
  • Haskell hacking
  • Cross-country skiing

Thursday:

  • New Organization Structures
  • Branching out – Venturing beyond the JVM (Rust, Go, etc)
  • Monads WTF

Friday:

  • TDD Overrated?
  • Continuous Deployment: putting it all together – CI, Puppet, Chef, orchestration, metrics
  • Web Mapping & GIS
  • What now for the Web (HTML 5, CSS, Javascript etc)

More to come in future posts.

Continue reading

Mozillians Re-Design

Matt Brandt

Help test the new site redesign. Welcome to the Mozillians project – a
community phonebook to find, share and communicate with like minded
Mozillians throughout the world. I’m thrilled to announce the latest
evolution of the website.

The team has been feverishly redesigning the user experience. We’re at a point
of stability; design ideas have been expressed, debated, agreed upon, and those
ideas codified.

The new Mozillians.org

A test environment is live on our staging server – mozillians.allizom.org
– for your critical eye. We welcome you to explore the new site and offer
your open and unabashed feedback.

Test Plan

We are doing exploratory testing. Here are a few things you could check for:

  • Functionality – a feature does what it is supposed to do.
  • Layout issues – does the site look correct in different environments and screen resolutions?
  • Usability – does something not make sense? seem hard to use?
  • Security – XSS, encoding/escaping issues, etc.
  • Error handling – system fails gracefully and displays useful and appropriate error messages
  • localization (l10n)

Out of scope for this test cycle (don’t test):

  • the profile pages – work is still being completed on their design. For example here is my profile page
  • search functionality – we know that search and results it returns work sub-optimally and will improve it in the future

Environments to test against:

  • Linux, OSX, Windows
  • Android, iOS
  • Firefox (RC, Beta, Nightly), Chrome, IE 9+, Safari

Setup

To get started you’ll need:

  • Get a vouched Mozillians account on our staging server. Ask in #commtools to have your account vouched.
  • Disposable email addresses so you can create test accounts on stage. I recommend free services like Mailinator or 10minutemail.

Filing Bugs

Important tips for filing bugs:

  • Search Bugzilla to see if the defect has already been filed. Try not to file duplicates if a bug already exists.
  • Write good bugs that provide clear steps to reproduce the problem. Read this document for tips.
  • Use this form to file new bugs.
  • Bugzilla etiquette – be polite and treat people with respect, we are a friendly community.
  • IRC etiquette – same as Bugzilla; relax and have fun.

We plan on keeping this test cycle open from 2013-09-18 to 2013-09-25.

We really appreciate your enthusiasm and help with making the community
phonebook better. This is fully a community initiative and wouldn’t exist
without you. If you have questions or simply want to say hello reach out in IRC
#mozwebqa or #commtools – and introduce yourself. Myself or another
community member will help you. My online name is mbrandt.

I look forward to seeing you online!

Matt Brandt

https://mozillians.org/u/mbrandt

Continue reading

Mozillians a big thank you!

Matt Brandt

Thank You

Firstly, I wanted to profoundly thank the Mozilla community for the incredible
response and support that you expressed when we put out the call for help
to do exploratory testing on the Mozillians project.

Thank You

** The Mozilla Web QA team in San Francisco. From left to right: Raymond Etornam, Dave Hunt, Bob Silverberg, Matt Brandt, Stephen Donner, Zac Campbell, Rebecca Billings, and Krupa Raj **

The number and quality of bugs filed (54) within the short timeframe of a few
weeks has been nothing short of overwhelming. The community has helped identify
conversations that stress usability. These conversations are informing the next
generation of UX that the team is working on.

I’ve whole-heartily enjoyed speaking with you in IRC, email, and Bugzilla. To
be quite honest your feedback has swamped our small team! This is something to
be celebrated! Thank you!

Next Steps: Phase 3

The Mozillians project is entering Phase 3 which represents a massive
site-wide overhaul.

“Contributors and staff alike mention the difficulty for potential contributors
to get started in contributing to Mozilla. There is no easy way to find a
finite contribution to perform without having to go through, on average, 3
click-throughs via on normal contributor channels (i.e. Mozilla Newsletter,
Get Involved, etc.). The aim of this milestone is to deliver a set of
solutions to increase the likelihood of participation from potential and
new contributors.” ~ Phase 3

Roadmap - Phase 3

Phase 3 includes:

Over the course of the next several weeks the team will begin creating the next
iteration of Mozillians after which we’ll engage the community again for an
in-depth round of focused exploratory testing.

Staying Involved

Many of our web projects ebb and flow at varying rates. While the Mozillians
project is in a state of focused calm you can continue to stay involved; first
through manual testing on our other web properties and second by helping
with test automation.

Stay involved

There is an third option. If the Mozillians project runs strong in your
veins I am willing to mentor a single individual who may become the proxy
lead
. The project currently needs help with new feature testing as well as
assistance with the weekly releases.

I am looking for an individual who is willing to learn as well as actively help me
creatively push the project forward. This is something that only a person who
strongly enjoys being a QA (Quality Advocate) can do. You must be able to commit
3 – 4 hours per week.

If this interests you come talk to me in IRC in the #commtools or #mozwebqa
channels. My nick is mbrandt. I am typically online 8am – 5pm UTC/GMT -8 hours
Monday – Friday. You can also reach through email.

See you online!

Continue reading

Mozillians Testplan

Matt Brandt

Manual Testing

Welcome to the Mozillians project – A community phonebook to share, identify
and communicate with other Mozillians in our community. If this is your first
time working on a project at Mozilla or you’re a seasoned veteran this is a great
project to get involved on.

Mozillians.org homepage

Exploratory Testing

Exploratory testing is a great way to contribute to the Mozillians project. Many
of our integration and functional tests have been automated, but automated
tests are best at catching the bugs we can anticipate. Exploratory testing
catches the truly interesting bugs — the ones we haven’t imagined yet — and
gives testers an opportunity to learn about the Mozillians project and platform.

“The main advantage of exploratory testing is that less preparation is needed,
important bugs are found quickly, and at execution time, the approach tends to
be more intellectually stimulating than execution of scripted tests.”
Wikipedia

There are several schools of thought on what exploratory testing is. For
our purpose we’ll treat it as a testing approach where a tester through
creativity and curiosity seeks to find out how the software works. An
exploratory tester simply looks at and uses the software as a normal user might,
and asks questions on behalf of the users. This approach reveals places where
the software does not behave as expected — in other words, bugs!

Audience

The Mozillians project targets several different types of users:

  • Community members who want to connect with others in the community in order to learn more about them and/or form groups around interests and skills
  • Community organizers who want to reach out to groups of people that identify with particular interests
  • Developers who want to use the API to power their own applications

Testing

Explore!

Your mission if you choose to accept is to creatively explore the Mozillians
website, keeping in mind the users it is intended for. As you work through the
different areas of the application, apply a critical eye to the design, layout,
workflows, and different functionality of the site. If it helps you, keep notes
about what you find, questions you may have, and thoughts about additional
areas you’d like to test.

Here are few places to start:

  • Is the site behaving as you expect it to?
  • Does the site and user flow behave the same on different devices and web browsers?
  • What happens when you pass in unexpected values to the API?

To get started you’ll need:

Filing Bugs

On the Mozillians project all defects and feature requests are tracked in
Bugzilla as bugs. Yes! We treat everything as a bug. Not all bugs describe
problems in software; some bugs describe feature requests, and some ask for help.
We simply use Bugzilla as a ticketing system that helps facilitate discussion.

Bugzilla

As you are testing, the types of bugs that you’ll discover will likely fall into
three categories:

  1. Functional regressions in the application
  2. Usability problems,
  3. New feature suggestions.

Sometimes the issues are known or even deliberate.
If a feature request has been turned down in the past and you feel it should be
incorporated into the application, please file a bug (or ask that a closed one be
reopened) and advocate for that feature.

A few tips for filing bugs:

  • Provide as much information as possible so the team (devs & other QA’s) understands the problem or suggested enhancement that you are making.
  • Do your best to find and provide clear steps to reproduce to problem, including what browser you used, what URL you were visiting, where you clicked, what you saw, etc. Sometimes a screenshot is a great help.
  • Try searching Bugzilla for the defects
  • Use this form to file new bugs

You can always ask us questions if you aren’t sure if what you found should be
entered as a bug.

Please let me know how your testing went. I’d love to hear from you!

mbrandt@mozilla.com

Continue reading

Mozillians Testplan

Matt Brandt

Manual Testing

Welcome to the Mozillians project – A community phonebook to share, identify
and communicate with other Mozillians in our community. If this is your first
time working on a project at Mozilla or you’re a seasoned veteran this is a great
project to get involved on.

Mozillians.org homepage

Exploratory Testing

Exploratory testing is a great way to contribute to the Mozillians project. Many
of our integration and functional tests have been automated, but automated
tests are best at catching the bugs we can anticipate. Exploratory testing
catches the truly interesting bugs — the ones we haven’t imagined yet — and
gives testers an opportunity to learn about the Mozillians project and platform.

“The main advantage of exploratory testing is that less preparation is needed,
important bugs are found quickly, and at execution time, the approach tends to
be more intellectually stimulating than execution of scripted tests.”
Wikipedia

There are several schools of thought on what exploratory testing is. For
our purpose we’ll treat it as a testing approach where a tester through
creativity and curiosity seeks to find out how the software works. An
exploratory tester simply looks at and uses the software as a normal user might,
and asks questions on behalf of the users. This approach reveals places where
the software does not behave as expected — in other words, bugs!

Audience

The Mozillians project targets several different types of users:

  • Community members who want to connect with others in the community in order to learn more about them and/or form groups around interests and skills
  • Community organizers who want to reach out to groups of people that identify with particular interests
  • Developers who want to use the API to power their own applications

Testing

Explore!

Your mission if you choose to accept is to creatively explore the Mozillians
website, keeping in mind the users it is intended for. As you work through the
different areas of the application, apply a critical eye to the design, layout,
workflows, and different functionality of the site. If it helps you, keep notes
about what you find, questions you may have, and thoughts about additional
areas you’d like to test.

Here are few places to start:

  • Is the site behaving as you expect it to?
  • Does the site and user flow behave the same on different devices and web browsers?
  • What happens when you pass in unexpected values to the API?

To get started you’ll need:

Filing Bugs

On the Mozillians project all defects and feature requests are tracked in
Bugzilla as bugs. Yes! We treat everything as a bug. Not all bugs describe
problems in software; some bugs describe feature requests, and some ask for help.
We simply use Bugzilla as a ticketing system that helps facilitate discussion.

Bugzilla

As you are testing, the types of bugs that you’ll discover will likely fall into
three categories:

  1. Functional regressions in the application
  2. Usability problems,
  3. New feature suggestions.

Sometimes the issues are known or even deliberate.
If a feature request has been turned down in the past and you feel it should be
incorporated into the application, please file a bug (or ask that a closed one be
reopened) and advocate for that feature.

A few tips for filing bugs:

  • Provide as much information as possible so the team (devs & other QA’s) understands the problem or suggested enhancement that you are making.
  • Do your best to find and provide clear steps to reproduce to problem, including what browser you used, what URL you were visiting, where you clicked, what you saw, etc. Sometimes a screenshot is a great help.
  • Try searching Bugzilla for the defects
  • Use this form to file new bugs

You can always ask us questions if you aren’t sure if what you found should be
entered as a bug.

Please let me know how your testing went. I’d love to hear from you!

mbrandt@mozilla.com

Continue reading