| || |
I had been assigned with a task to find out standard ways of rejecting bugs by developers.
Some of my doubts are...
1. Are there any standard methods of bug rejection
2. Does it depend on organization to organization
3. Should it remain same for all projects in a firm
4. Will the QA team decide on these methods?
Can someone help me out by throwing light on this topic.
Thanks in advance
Re: Defect Rejection
0) I don't understand what you mean by "rejecting bugs by developers". What does the text of your assignment say? I'm going to assume that your question is, "What are the standard methods by which developers claim that something in not a bug, and how do we get around them?"
1) Two standard methods of bug rejection are
a) for the developer to say "it works for me". The antidote to this is to ask "If it doesn't work for /me/, for whom else might it not work?"
b) for the developer to say "no user would ever do that". The antidote for this is to ask "Have you thought of all the possible users of our software? What do you mean by 'no user'? Could a hacker do it? Could a user do it by accident?"
2) I don't know of anything that all organizations do. Of course things vary from organization to organization.
3) Every project in every firm exists in some context. If a practice or a process doesn't work for a given process, it's insane to use it.
4) I have no idea what this question means.
5) There's an issue related to research in these forums. Sometimes people are asking for answers to work assignments, sometimes to university assignments. In any case, the answers that I supply, that others in this forum supply, that you provide on your assignment, and that your boss or examiner expects are opinions based on a certain context.
When you give the answer to your assignment, it's important for you to report credibly. This means identifying your sources. Thus if you choose to quote or paraphrase me, you should say so: "Michael Bolton, writing on QAForums, believes..." If someone else provides another opinion, and you quote her, use her name too.
Identifying the sources for the answers you find, either in research or in testing itself, is part of being a good tester.
Re: Defect Rejection
I have no intentions to use the content above as my own. Ofcourse I know to respect your views and credit you if its used.IP of a person should be acknowledged.
The replies to question numbers 2 & 3 answers my question #4.
Re: Defect Rejection
Mr.Michael is big and strong and scares coders a lot, Lokesh. Take it from me, he is right, or at least very persistent. What I wonder is if you actually understood your mission, which may be to formally manage the bug reporting and resolution process, which is very much a QA responsibility. If development would be as punctual they wouldn't need QA to supplement the work they don't do. If you simply present a problem to a programmer they will tell you to take a hike and you engage in a dialog a.k.a. Bolton's scenario 1. However, if a formal bug report is drawn up with a screen shot or other supporting evidence and reviewed by the manager for assignment to a developer then dialog scenario 1 can be avoided. Bolton's postulation 2 simply means that the degree of formality depends on what is at stake. If you develop informal code for local use that is non-critical you would not have to establish the same level of audit-trail I would expect you to use for commercial code that is developed for clients. So in part the oracle has outlined the need for some self-examination, to make sure you formalize to the degree that you need it and not for the sake of formality (unless your organization is too rich and wants to buy a top-of-the-line problem management methodology). Bolton's theorem 3 is wisdom at its best, if you know it hurts then stop doing it. You implement a level of formality that ensures nothing remains forgotten and unfixed at the end of the project, unless there is a formal decision to postpone the change till a future release. You can document it as a resolution, so that a future project picks up where you left off. Only formal documentation of what your resolutions were will help with that because you may not even have the same people on a future project. Bolton's lamentation 4 shows he doesn't take crap from anyone, and therefore his position is that it is rediculous to think that anyone other than QA would decide that. I believe he implies that we friggin well impose it, hence his reluctance to even acknowledge the question, but I am sure that captures the sentiment. What about Bolton's rant 5? He really wants you to get on the mailing list for his newsletter so that he has a little more space to to make his views much more clear to many more people, and hopefully for him to get some real payback for distributing his brand of wisdom with endless patience over and over and over...
Re: Defect Rejection
It all started with one developer moving the resolution as 'not a bug'. There were no reasons why he decided so.
So an offline meeting happened with both QA/Dev teams present and all such bugs were present.
So at some point they asked 'On what basis do you feel its a bug' I supported my answer with feature requirements and Test Cases.
They were of the opinion that Customer wont do this n that n etc...
Then I pondered on this...If any standards are available for this 'not a bug' I would have less trouble to support my stand.
As such there wont be any functional requirements for all user specific scenarios [like he can do this, he cant do this etc]
Its good that we have started to develop one such document now.
Re: Defect Rejection
Actually, a couple of clarifications to Frits' comments.
First, I'm not certainly big, I'm not very strong, and I only scare coders who don't like looking good. I've met very few of these. I try to make it clear that by testing the product, I'm giving coders information that we can all use to make decisions about our products and our companies, ideally with the goal of improving them. Most coders are happy, most of the time, when I find interesting bugs, because not having them in production makes their lives easier. They'd probably rather find out about the bug from me than by a panic phone call at 6:00am on Saturday morning, or three weeks into production and one week before their annual review. /Especially/ when the bug wasn't really their fault.
Second, I genuinely did not understand question (4). I would very much like to answer any question that Lokesh might have; if he can help me to understand it, I'll take a stab at it. Frits misinterprets my reply, which simply meant what it said.
I do want to be clear on one thing, though: the testing organization should definitely NOT decide on whether a bug is fixed or not. Testers should not be put in the position of making business decisions, nor should they accept the position if it's offered. If they want that responsiblity, they should become project managers with all of the authority, information, other responsibilities, and salary that comes with the job.
Should we consider ourselves quality gatekeepers for the product? We shouldn't, because it's not realistic. We can be overruled. There are often business priorities that take precedence over our insistence that the product has to work Just So, and that's as it should be. The gate that we keep can easily be steamrollered by everyone from the project manager right up to the CEO. That's what they're paid for--to listen to the entire organization, not just the testers.
It's far better for everyone when they understand my role. I find out things about the product. I exercise the product and investigate it, seeking out vulnerabilities, risks, and other threats to the product and to the organization. I tell managers about what I've found. I try to think expansively of the risks and their consequences, and open people's eyes to possibilities if I think I've communicated in a way that has underplayed their significance. They take that information and all of the other information that they've got about the schedule, the budget, the market, the competition, the development department, their bonuses, the company's financial situation, the threat of layoffs, the 60 Minutes team at the door... and they (not I) weigh all those things together to make a decision. That's why they get the big bucks /and/ the ulcers.
Now: someone might say that Quality Assurance is different from testing in this respect. It is; QA is even more vulnerable if it believes itself to be the gatekeeper. I would go so far as to contend that a genuinely healthy organization needs testers, but not a QA department. In a healthy organization, everyone is responsible for assuring quality; everyone contributes to the design and execution of processes; and everyone behaves responsibly within that framework. That includes, by the way, the intelligence and the agreement of all to adapt the process when it's necessary to do so. Conditions and requirements change, and processes should be our servants, not our masters. But even in those healthy organizations, there is a CEO and (s)he has the final say.
Lessons Learned in Software Testing, and the people and precepts of the Context-Driven School of Software Testing, lay this out clearly. From personal experience (and the impetus from a course by Cem Kaner way back in 1996), I can say that life is much more relaxed for a tester as soon as he drops the ambition to be CEO-without-a-paycheque.
Finally, Frits also, alas, misinterpreted (5), but he couldn't have done that with a lot of help from me. Rant (5) was my first and hamfisted attempt to give a different kind of answer to a recurring issue in this forum: how do we answer a question that's clearly a school assignment? I've been uncomfortable in refusing to answer such questions ("it's an assignment; find out yourself!"), since one of the best ways to get answers to questions generally is to ask your community. On the other hand, I don't want my answer to appear verbatim on someone else's homework. So these days, I'm asking people to cite their source of their answers, which is not only ethical but entirely valid scholarship.
At the same time, people (when answering assignments) should not appeal to authority--"It's so because Bolton says it's so". They should instead by able to say, "Bolton says it's A; Beizer says it's B; Pettichord says it's C. I can demonstrate that it's mostly C, because Pettichord's research was thorogh, Beizer doesn't address this context and because Bolton's argument has these nine flaws..."
I'm not fishing for subscribers to the newsletter by asking for citations. If I'm fishing for subscribers, it's simply by posting stuff here /at all/, wouldn't you think? [img]images/icons/smile.gif[/img] But in any case, I hope that I provide value and stir up some thinking and discussion one way or the other. Oh--and just in case anyone was unclear--the newsletter is free (except for the time it takes people to read it), so it's not like there's a profit motive at work. Frits IS right, however, about spreading ideas being its own form of payback.
It probably complicates matters somewhat as I realize that Lokesh's question is clearly NOT a homework assignment. Ooops.
So... back to Lokesh's issue [img]images/icons/smile.gif[/img]
Re: Defect Rejection
Lokesh asked, "any thoughts", and wouldn't you know, I have a couple.
The first is not to get too hung up on documentation, but to continue to do a good job of explanation. That is, you can write all of the documentation in the world in advance of finding a bug, and you'll still find stuff that the documentation doesn't cover. So, to the extent that you feel it necessary to do documentation at all, keep it down to principles, and don't bother trying to capture every instance.
One of the reasons that I tout James' Heuristic Test Strategy Model http://www.satisfice.com/tools/satisfice-tsm-4p.pdf is that it's an example of something that's lean and very effective and that leads to good heuristics. The quality criteria, the "-ilities" cover most attributes for most products pretty well. No matter how much documentation you write, there will be /no/ discussion about some bugs, there will be /some/ discussion about most bugs, there will be /a lot/ of discussion about a few bugs. The latter kind of discussion happens just as easily when there are very detailed bug criteria as when there aren't. The reason for this is that requirements change in response to business needs; that they are all fundamentally ambiguous to some degree; that there are always new types of bugs that don't fit into categories neatly; and finally, because the testers will have different information and opinions from the developers, who in turn will have different information and opinions from the business people. Collaboration and informal agreements may be quicker than protracted, legalistic arguments.
Here's an example: our organization chooses to treat all spelling errors as Level 3 defects, where 3 is lowest and 1 is highest. Spelling mistakes just aren't that important. This is a fine policy until our client, the Cleveland Pubic Library, receives the first build. That is, "anything that might embarrass the company or make it look avoidably stupid = 1" is less precise but more sensitive to context than "spelling error = 3".
Finally, it's much easier when a single person--not a committee, makes the decision. It sounds to me like Lokesh did the right thing by explaining his oracles and his heuristics, and then letting the program manager decide. That's as it should be. It would be nice to pretend that we're part-decision-makers, but I'd say it's more realistic to consider ourselves as service providers who provide credible information the people who really make the decisions.
Re: Defect Rejection
Standard reasons to reject something as a defect:
1. <Something> in the system does not match the specification.
2. The system did not handle an error condition gracefully.
3. <Something> is inconsistent with operation in other parts of the system, or other corporate products.
4. <Something> no longer works/displays the way it used to work/display.
5. <Something> does not appear to work/display correctly to the tester
Of these, 1-4 should be accepted as likely defects by the developer. The only "gray" area is number 5. My advice to testers is always the same, and it is difficult at first, but liberating, once you get the hang of it. It's our job to find and report defects. We don't own them. We need to report the defect, defend it as best we can (passionately in some instances) and then LET IT GO. If the company wants to go ahead with that error in the code, the quality of the system is on the head of the manager(s) that made the decision. All we can do is contribute as best we can to make the product as high-quality as possible in our own corporate environment. Each company will be different in the level of quality it desires.
By the way, I consider any advice that recommends not getting hung up on documentation or "to the extent you feel it necessary to do documentation at all" to be irresponsible, unless you work in an agile test environment and on a very small team.
Re: Defect Rejection
Michael, I realized that Lokesh' requests was a genuine request from someone who was put in a bit of a bind, and I found your answers a little hard in places. I simply tried to soften it up a bit, and I also had a slightly different take on it, in part because I believe very much in QA as more than QC in a preventive role. In fact, many cases such as these are QA decisions, where the answer is based on making sure everything is in line. It may very well be that the requirements are wrong, but programmers cannot unilaterally fix them even if they think they know better. The environment that Lokesh describes seems like chaos, not much like the CMM-5 companies every Indian salesman brags about. I think I can agree with Linda's set of criteria, and I certainly agree with making it a priority to get the requirements right in the documentation so that future testing does not get in the same bind. I know we all have our unique views, and we agree to disagree, and I am glad I got you to reconsider your answer. Lokesh is much better off with the second version.
By the way, Michael, on question 4 we do agree it is not QA that makes the final decision. Lokesh asked if it is QA that decides on the methods and I pragmatically conclude that nobody else can be expected to do that. I also recommended that he simply establishes standards to demonstrate the basic facts for suspecting a bug, complete with a sufficient body of evidence, which is truly all a tester can do. That evidence should be properly catalogued, and testers should not be nervous if it turns out to be a tester mistake. Rule #1 is, if it looks like it might be a bug, document it.
Re: Defect Rejection
Frits:>Michael, I realized that Lokesh' requests was a genuine request from someone who was put in a bit of a bind, and I found your answers a little hard in places.
Well, I'm glad that at least Lokesh seemed to find them helpful to some degree.
Frits:>In fact, many cases such as these are QA decisions, where the answer is based on making sure everything is in line.
I disagree. Decisions about the project are project management decisions, not QA decisions (except to the extent that the actual assurance of quality is a management responsibility).
>By the way, I consider any advice that recommends not getting hung up on documentation or "to the extent you feel it necessary to do documentation at all" to be irresponsible, unless you work in an agile test environment and on a very small team.
I'm not sure I understand. If it's irresponsible for me to recommend not getting hung up on documentation, would it be therefore be responsible for me to recommend getting hung up on documentation?