bug


Striving for high quality bug reports.

As any QA manager can confirm, one of the greatest wastes of time there is in the software industry (although probably not the greatest waste of time) is badly logged bugs.

Failure to provide users with a methodical bug logging systems is a huge time leech: it hurts thousands of corporations, and wastes millions of man hours around the globe. One of my attempts to combat this big evil leech is a piece of software I wrote at work called bug.

To begin with, bug captures a lot of the behind the scenes information that is normally missing from bug reports.

Such as:

  • username
  • machine name
  • operating system
  • geographical location
  • etc.

It then holds the user's hand to ensure they enter information in a methodical manner.


Repro Steps - a case study

Repro steps list the steps an engineer needs to take to reproduce a bug in a system. Let's pick a case study everyone can (potentially) relate with, crashing a car. Let us also look at a low quality bug report, for crashing a car:

Hi, I got into my car this morning and drove, but I crashed. Please fix this issue!

This is the most engineer-unfriendly bug report there is. And believe it or not, it sadly looks like the vast majority of bug reports engineers receive from their users on a daily basis.

Let us now think of a better report from the driver after the crash. This is a list of what the driver did, step by step, which lead up to the crash:

  • open car
  • sit in driver's seat
  • turn on ignition
  • press down on clutch, accelerate gently, and reach biting point
  • release hand brake
  • accelerate, and change gears as appropriate
  • do not steer
  • bug

In this case we'll say the driver is a suicidal idiot, but in 50 years' time, this might be a true bug report for the automatic steering and braking systems failing to engage properly. Assuming the driver survived to be able to submit this report, the careful engineer might notice something missing: the driver never attached their seat belt.

The engineer therefore investigates, and to his horror realises that the automatic steering and braking system cannot engage themselves as long as the seat belt is not buckled!

It would have taken a lot of back and forth discussion between the engineer and driver, (probably via the engineer's boss, through the insurance company's law department etc.) to reach the information that the driver was not wearing his seat belt.

Let's look at some further information which can gather to further help us elucidate what went wrong.


Expectations

We all expect different things from different systems. Therefore, capturing what the user actually expected to happen is vital. Pardon the grim nature of this example, but for a driver trying to kill himself, the automatic steering and breaking system clicking on and steering the vehicle to safety is the bug! Whereas for the driver who wants to stay alive, the opposite is true.

A simple box asking what the user expected to happen adds so much valuable information to resolve the issue.

I expected the car to steer or brake automatically.

Aha! We now know exactly what was going through the user's head, and what lead to the crash. This is the kind of information you never get in standard emailed bug reports.


Actual behaviour

Ask the user to tell you, in his own words, what happened instead of his expectation. This is generally the behaviour in the system that needs changing according to this particular user, and where the bug truly lies.

The car did not brake, and fell straight into a cesspit filled with crocodiles.

(Whoever said crashes had to be boring).


Error message

Like car crashes have skid marks, or plane crashes black boxes, or muders finger prints and 911 calls, software systems always contain error messages. Printing out the current status of operations of a piece of software, into a window, or into log files, is just the way software is built in the first place. A box asking the user for an error message is the first step towards acquiring this information. Embedding your bug reporting software into your product enables you to gather this information automatically. It is indeed very hard to trust users to know where to find the error message, so you should do everything you can to salvage this information automatically.


As a template

I've worked in highly effective companies that did not even have such software, they just used an email template for logging bugs. If your users are smart and disciplined this could likely suffice.

Dear (Department or company name),

This e-mail is to inform you of a bug.

Workstation: (workstation name)
Location: (bucknall | mainframe | vancouver)
OS: (Linux | Windows | Mac)

Scene: (path to scene, if no scene involved then delete this field)

Show: generic

Repro Steps:
        - (repro step 1)
        - (repro step 2)
        - (repro step 3)
        - (repro step ...)
        - (repro step N)
        - *bug*

Expected Result:

(what you expected to happen)

Actual Result:

(what actually happened)

Error Message:

(error message from program or terminal)

Further Info:

(any extra information you think is relevant)

Regards,
-(reporter's name)

Coutesy goes a long way

Bugs are usually logged by angry users on a deadline. But it is no reason to forget our good manners, and it's usually a good idea to remain polite with the developers who'll be solving our problem. Dear, and Regards add a nice touch to any bug report and make developers' jobs more human and humane.

Every bug now looks the same

This is probably the most important point to take home: using a bug template means every bug now looks the same. Developers know where to look for what in the report. Their job is now streamlined.


Conclusions

Like everything system related, clear thinking brings clear results. Clear thinking by software developers leads to high quality usable software. But let us not forget that clear thinking from users is also required and even though such tools increase the quality of bugs logged by users, they are in no way a panacea.

A smart user who knows what he wants is able to log high quality bugs without the use of such software. Similarly you can throw all the tools you want at users with murky thinking, they are still be able to fill in bad bug reports.

It is crucial to minimise, or entirely cut down to 1 the input streams to your department for bug logging. This is extremely easy to do if your clients are external, and immensely hard to do if your clients are internal. If your users are still capable of logging low quality bugs using an idiot proof system, then you are probably dealing with an idiot. Do not waste time on low quality bugs. As software professionals we feel that it is our duty to respond to all the bugs and requests of our users equally, but in reality if it looks and quacks like a lame duck, it most probably is a lame duck. Focus on resolving bugs for your smart users, those who log high quality bugs and help move your product forward. For low quality bugs, write a quick response to the user with further guidelines on how to log a high quality bug, reject his bug and move on.

Wasting time with back and forth with users who are not able to log bugs properly wastes time that could be spent fixing high quality bugs, which harms the software development's team efficiency, which in turns harms your product, and in turn harms your customer, your turnover and their turnover. In short, it harms everyone.


comments powered by Disqus