Code and Exploratory example…


In high school, I had taught that we should have preconditions, what the function does and post conditions in every function/procedure that we write.  Not just for ourselves in the event that we forget what happens, but also for other people reading our code.  [side note : yes.  I had computer science classes since junior high school.  In college the same thing.  In practice, it’s not always done that way.

For those that aren’t familiar with coding, just think of a function as a lego block.  Now a days lego is all sorts of funky.  They have lego pieces that act as windows, doors, hindges, etc.  Each lego piece has some sort of function, and one side that helps it link to another piece to make it fit with another piece.  You can think of the little bumps as output variables and the other side with the holes, input variables.  When you put a bunch of lego pieces fitting all together, you basically have some sort of model, or artwork, or whatever the builder has envisioned (hopefully).

What it did teach me though, is how to do QA.  “How so?” you ask?  Simplifying it down it’s like this:

1) an application has a certain state before it goes through a procedure/function/etc.

2) functions have 0 to … however many inputs when the function is called

3) the functions does something

4) the function outputs 0 to however many outputs after the function ends up running

5) the application has a certain state after the function ran.  This could potentially be the same as before the application ran.

So what does this mean?  It means that when you encounter a bug, whether it be a UI bug or a crashing bug, usually there is a way to get to that through basic usage and translating that to code.  It may not be easy all the time, but basically the application got to a certain state, a function ran and either within the function or the outcome of the function caused an issue.  Understanding the state prior to where the issue occurred helps with the reproducibility of the issue.  Understanding the function itself and the weakness of the functions, helps find critical bugs faster.  This helps out with integration testing, and in some sense covers what unit testing cannot necessarily cover.

“Ok, blah blah blah… where’s the example?” you say?  Fair point.  Let’s take a look at mozillians web page incorporating browserid.  Linked is the staging server (a server where testers can test out what the dev have in plan and not affect the production [what’s served to the public] server) :

What I do know is that the sign in is using browser id which is one part on a different server, and another part that is mozillians web page.  I don’t really know too much about what’s going on in depth about either, but I do some exploring to figure out the ways into mozillians.  This basically means clicking on as many links as I can find in terms of the login or registeration.  I also do ways to explore what types of input browser id is taking in.  The next steps is to figure out what kind of variables I can use from mozillians to input towards the browser id.   The biggest thing that I found is methods in trying to click on a link into browser id as well as changing the language.

I do the same thing for output.  What does browserid send to mozillians?  What information does mozillians accept?  You have to experiment and think about what’s going on in the background as well as ask developers what might be happening in order to figure out how things are working.  What the developers say might not necessarily match, because it could be that they expect something to happen, but something else ends up actually occurring due to an unexpected event.  That’s called a bug.  ;)   In this sense, you’re basically being an editor to a writer.  You’re trying to catch all the loop holes for the writer of the program.  Here are the bugs that I written about mozillians.  [note : It was stated that I shouldn’t worry about duplicates, but I can’t help not looking at least once.  I voted on one that tchung wrote because I also ran into it.  This helps to cut down on the additional overhead of trying to manage duplicate bugs].  As long as you understand how you got to a certain state correctly, the bug should be reproducible.  In order to test that you are understanding the state, reset the state of the application to where you know what’s going on. (ie quit the browser, clear the cache etc.) and then try to reproduce the bug by getting to that same state you were in and then do the steps in which you were able to produce a bug.

In some sense, you can get to one off bugs easily by keeping track of some small things.  In Native Fennec, the way that the urlbar is handled is different from a text field in the content.  The urlbar is a widget, where as the text field is something that’s built into fennec.  So the behaviors might be different.  Another example is tabs.  What happens when you reproduce a bug in the current tab?  When it’s fixed, what happens when you try in a new tab in the background, and not the current tab?  Read various bugs and try to keep up with things that might end up making things “one off”.  From there, when there is a bug fix, try to figure out what things might end up being applicable to try to make a one off bug.  This is what QA folk call “testing around” the fix.  Testing the fix by doing the exact steps usually isn’t as hugely productive because that’s what the developer usually is trying to target (unless either a) the developer is testing the tester to see if they are checking things, or b) the developer made a fudge in the code which was missed by the review) and c) the developer didn’t test his fix.  [note : In some sense, when the developer is testing his fix, he might encounter a bug, in which developer have an advantage of uncovering stuff that wasn’t found before]  So when verifying a bug… check what the developer tried to fix and more so, test around the bug as well.

Filed under: QA, QMO