Using Specifications to find bugs…


The following is an expression of my own opinions and not necessarily reflecting any company:

Specifications and technical documentation allows everyone to be on track with what the feature entails. This includes other Developers, QA, and Tech Support as well as marketing in terms of what the feature will represent and what it possibly can do.  Even contributors for the case of open source material.  The value of this documentation is the ramp up time needed to figure out what’s going on with a project feature.  In my own experience, it took me 2 weeks from the start (1 week for procedures, equipment and setup and 1 week for actually reading the document and seeing how things are laid out) to ramp up with a well documented project versus 4 to 6 weeks to figure out how some of the basic things work because I’m hunting for documentation and finding outdated materials.  In every company that I’ve worked for the first question has always been “Have you searched for the documentation?”  then “What have you figured out or done?” To eliminate those questions as run arounds, I try to look for the answers first, and also observe what other people are doing in order to get an understanding of how things operate at a company.  This helps curve the ramp up time.  As a food for thought: would it be easier to spend 1 hour of reading through a documentation or 1 hour reading through code?  And how much farther do you get from reading 1 hour of code versus 1 hour of documentation?  How much will you remember what has and hasn’t changed after a year has gone by through reading 1 hour of code versus 1 hour of documentation?

In most cases though, in my opinion, technical specifications should have a rough draft at the beginning and should be finalized towards the later end of the stage, and not the beginning [Note: technical specifications means the details of the feature; a general end user specification should be drawn up towards the beginning with use cases; various scenarios on various level grades of end users… who’s the target audience, what’s the aim of the feature, etc.].  The reason being is that development needs time to prototype the feature, not just to get the feature working but also for integration purposes.  During the prototyping, the engineering design as well as the user interface may end up changing depending on the problems that occur (issues that may come up during the time frame that causes the specifications to change a lot.  If you’ve read some of the stories of the Steve Jobs at Apple, you may have read how many prototypes that they go through during this time frame of the design.  (Inside Steve’s Mind as well as some other articles that I have come across is what I’m referring to.)  You can also see some of the evolution in open source projects.  You can call this phase whatever you want, pre-Alpha, Alpha, Nightly, whatever, but all the engineering that I’ve seen are pretty much the same.  The feature isn’t fully testable until it’s “Feature Complete”.  Maintenance is a chore that cuts into time.  Also something to considering, during the feature building cycle, the QA, Dev, and Design teams are working together, but tech support, marketing , documentation team, localization doesn’t necessarily need to know at this moment in time.  Localization needs to wait til string freeze, the other team needs to know what goes in the final product not the path to the final product.  Why does technical specifications and engineering documentation need to really be drawn up at the beginning?  End User Specification should perhaps be drawn up so everyone is on the same page in terms of what to resolve.

Ok, so there’s my talk about technical specifications, and the need for them and when they should appear.  So getting on to how to use technical specifications for finding bugs.  The reason why QA would be looking at these documents are for many reasons.  Not just for figuring out if the technical specifications is achieving the goals of the end user specifications, but also trying to find out the weakness or flaws in the design and also the best method to test and where to test for critical issues.

UML diagrams are particularly good for this.  As well as state diagrams or flow diagrams.  It’s not just for looking at what’s there, but also what’s not listed.  The diagrams will show you want is accounted for and doing positive testing is easy in that regards, but in most cases will find you little or no bugs.  It’s more about what’s not accounted for and what are more critical to have for the end user.

Whether it be the process flow for UI for end user usability, or figuring out the where the critical issues might be.  Generally you can figure out the critical areas and where things might end up being an issue by studying the diagrams.  For example, if there’s two applications looking at one file, you have to worry about file locking and threads as well as what happens for a new user for the system (for checking for nulls) and all the permutations that could potentially cause an issue.  Depending on the situation, a combination could save some time; I do find though, that order does tend to matter in most cases.  An example of using permutations within fennec is using a tab versus not using a tab to visit a web site and changing orientations (landscape to portrait or vice versa).   Order does matter because you have have to be in a certain state to cause a bug to occur.  In essence, you can visualize which permutation can be an issue as you look at the diagram by visualizing how things would interact and work together to find bugs.

Knowing what type of architectural pattern also helps with testing.  I personally like the MVC for testing when the feature is complete.  Usually because the developers will end up making the UI portion last, so white box testing would be necessary as black box testing would not be possible during this time.  The reason I like this model better, is because of the separation of the data from the view, which allows for changing the ui in an easier fashion should the UI flow not work as well, but does not necessarily affect the data.  It also helps to understand design patterns and other terminology that a company uses helps for ramp up time.


Filed under: Uncategorized