zeebee
Newbie
Reged: 07/16/08
Posts: 7
Loc: Malmö, Sweden
|
|
I'm currently a trainee at a small company and my assignment is to integrate testing into their development process. The only directions I have is that I have to write test/use cases and then execute them myself. Since I'm the only tester I feel that it's unnecessary to document every step to execute it when I'm the one who's doing it in the end anyway. The requirements aren't very detailed and it's very hard to derive test cases from them.
I think it would be better to start with exploratory testing then start using test/use cases when the requirements are better detailed (I'll get to review them in the future) and when more testers get hired.
Please give me some more reasons why I should be using exploratory testing. (Or if I'm completely wrong with my reasoning, why not to use it.) 
|
Joe Strazzere
Moderator
Reged: 05/15/00
Posts: 10092
Loc: Massachusetts, USA
|
|
Perhaps they want you to write test cases and use cases so that the new hires will have something to work with?
--------------------
- Joe
I speak only for me. I do not speak for my employer, or for anyone else.
Visit my new blog All Things Quality
|
philk10
Advanced Member
Reged: 01/20/05
Posts: 578
Loc: UK
|
|
Some points your boss might raise, can you answer them...
"I've read that exploratory testing is best done by experienced testers, you are only a trainee "
"How do I know how many tests you've run and how much work you've done, wont you just sit there and click things at random ? "
"How will I know which areas of the program you've tested and in what depth"
"what if I want to assign extra staff to help the testing effort, they'll run the test scripts that you have written"
"If you say the requirements aren't detailed enough then how will your exploratory testing know if you've found a defect ? "
"how will we run regression tests if there are no test scripts"
--------------------
 and My Blog
|
Shane_MacLaughlin
Super Member
Reged: 09/22/05
Posts: 1496
Loc: Dublin, Ireland
|
|
While I haven't used the product, Test explorer seems to answer many of the above questions relating to the documentation side of things.
One other query that the Boss might add to the list;
"I understand that exploratory testing tests what the program does, we want you to test against client requirements"
--------------------
My LinkedIn profile
|
stoofer
Member
Reged: 08/11/08
Posts: 250
|
|
to add to philk's points, a question from after release:
"How did this bug get through testing?" Easier to answer if everyone signed off on the test plan & test cases, harder to answer if you performed solely exploratory testing.
I use a hybrid approach, as I am sure many others do (even without realising it), with both scripted and exploratory testing - although I have taken it a step further and instituted 'exploratory testing in [specific area] for [fixed amount of time] using at least functions [X, Y, Z]' as a test case that is part of my overall suite.
I use a prioritising system for test cases so that those which are proven to uncover bugs get run more often. This sort of tracking helps to 'prove' the concept of exploratory testing, manage it, and still maintain scripted tests (which are simply better than exploratory testing for some things - e.g. ensuring every button has been clicked).
|
zeebee
Newbie
Reged: 07/16/08
Posts: 7
Loc: Malmö, Sweden
|
|
Quote:
Perhaps they want you to write test cases and use cases so that the new hires will have something to work with?
I was just told I'll be part time employee after this trainee period which ends in 3 weeks. Then a full time employee when my next trainee period starts after 1st January. I'll get to hire testers when I start full time. Since the software will change a lot until then it would be very time consuming updating the cases.
Quote:
Some points your boss might raise, can you answer them...
"I've read that exploratory testing is best done by experienced testers, you are only a trainee "
True, but I'm here to learn right? Using the software as much as possible will teach me more than writing a lot of documentation that might not even be correct.
"How do I know how many tests you've run and how much work you've done, wont you just sit there and click things at random ? "
I would write down in what area I've been "clicking" at as well as writing down a description of what I've been testing and how long I spent doing it.
"How will I know which areas of the program you've tested and in what depth"
I've written down a plan of what areas and some basic functions that needs to be tested.
"what if I want to assign extra staff to help the testing effort, they'll run the test scripts that you have written"
The test cases will most likely be outdated by then.
"If you say the requirements aren't detailed enough then how will your exploratory testing know if you've found a defect ? "
Error messages, wrong data being shown compared to the db, etc. It's just as hard as knowing if the test cases I've written passed or not. Or?
"how will we run regression tests if there are no test scripts"
Not possible...  But we aren't using and automated scripts yet. 
I really appreciate all your comments!
|
Myrtle
Awsome Member
Reged: 06/12/06
Posts: 660
Loc: Spain
|
|
It's not necessary to use automated scripts to perform regression tests although that's automated testing's strength, IMO. It's perfectly possible to perform manual regression testing but for that to work some form of documented script is required to ensure the regression cycle covers areas that have previously been tested.
|
brentpaine
Veteran
Reged: 03/09/07
Posts: 3754
Loc: Waterloo, Ontario, Canada
|
|
Quote:
Since I'm the only tester I feel that it's unnecessary to document every step to execute it when I'm the one who's doing it in the end anyway.
This is a good reason NOT to do exploratory. If you believe that exploratory testing is a good fit for you because it requires the least amount of documentation, then you're either going to be in for a surprise or you're going to fail miserably.
Exploratory testing actually requires quite a bit of documentation, in the form of logging your work. Also, this documentation changes with each and every test cycle, so you're never avoiding it. Without spawning a debate on the subject, prior to it being known as Exploratory testing, ET was known as Ad Hoc testing. In comparison, I believe, it is very similar to Exploratory testing. However, there were huge misconceptions about it, also. For this reason, and possibly from poor implementation, Ad Hoc got a bad name because people weren't documenting properly.
Exploratory was developed from it's ashes as a new methodology, which explicitly promotes the documentation of your testing, in an attempt to bridge that weakness.
So how can you sell this to your boss? Well, Exploratory is a great means to test an application in a manner that closely mimics the way a client would use the application. This allows us to get away from linear test cases which might not force the user to interact with a function the way a real person would.
As a side note, if this application is being developed for a specific client, don't be surprised if they ask for things like a test plan, test cases, and/or test results. If the client doesn't fully understand your methodology then you could be in trouble here too because it can cause a deal to fall through if you don't have explicit test cases, as useless as they might be.
--------------------
Brent
--------------------
9 out of 10 people I prove wrong agree that I'm right. The other person is my wife.
--------------------
What is Holistic Testing?
|
Walen
Active Member
Reged: 05/09/01
Posts: 1064
|
|
Most, not all, instances where I have used ET, it has been in conjunction with scripted testing. At my current location, I've been using ET techniques in parallel with scripted testing. There are some instances where there are fundamental issues that must be proven, or cases to be made on what has been tested, with what results - precisely what the results were compared to what was expected.
In regulated industries, it would be extremely unlikely to pass an audit without a remarkably detailed description of what has been tested, the results of each and every iteration, what remains to be tested, and what, precisely, is the purpose of each test and what the expected results are.
As Brent mentions above, I have also worked in shops where detailed test plans are part of the deliverable to the client. Often, these were due before testing is complete, or, sometimes, before testing has begun. When there is significant money involved, "Trust me." does not pull much weight for test plans.
--------------------
P. Walen
|
DebbieStPierre
Newbie
Reged: 09/11/08
Posts: 7
Loc: Ottawa, Canada
|
|
I will also add my 2 cents worth here.
We know you are a beginner when you say exploratory testing will do when you don't know the requirements/functionality of the system.
By writing out your detailed test cases you will learn the system/application and be able to understand/explain/ justify your results. Otherwise, how will you determine what you have tested or what is still to be tested.
Negative and positive.
If, for instance, you say you've tested this screen with
- date
- given name
- surname
- address
- birthdate
- back arrows to previous screen, forward arrows
on it and it all passed ok, what does that say..
Did you, for example, test 13 for month, 32 for day, what about 2 for day or month or 90 for year with no month or day or.... I could go on to 50 different test cases for date alone. How will you know what you've tested and what you haven't?
That is just one field, wait until you start integration and then bring in other systems feeding to and from and then the internet..
cheers deb
|
zeebee
Newbie
Reged: 07/16/08
Posts: 7
Loc: Malmö, Sweden
|
|
Thanks guys, I didn't expect to get this great feedback! I'm indeed a beginner but we all have to start somewhere and I'm eager to listen and learn from you.
I was a bit stupid thinking that only not well documented ET would be sufficient enough. You're right that it's impossible to explain and justify my results and coverage of the tests without having detailed documentation.
Would applying ET to the areas where most test cases have been executed in the end of every day be a good approach?
|
brentpaine
Veteran
Reged: 03/09/07
Posts: 3754
Loc: Waterloo, Ontario, Canada
|
|
I wouldn't say it's stupid, I know when ad hoc testing first came into some popularity, there were plenty of people moving over to it because they thought it was a great way to side-step the lengthy documentation process. I don't think these people are stupid, it's just how it appears to people from the outside looking in.
ET is a great complimentary testing method. Where I am, we actually have a full test suite of manual, scripted test cases, and a complete suite of exploratory test cases. On a new build we will generally run a full test cycle of the manual test cases and then follow it up with the ET tests. We have actually proven the value of ET by doing this. In our original pilot of this method we were near completion of a smaller project. We had reported close to 400 bugs and all was looking good. We decided to run a quick ET cycle since marketing wasn't done on their end and we blew up the app. We probably reported another 300 bugs over the next few weeks, including 70 showstopper issues. It was a dream, really. No need to sell it anymore.
The difficulty will still be selling it until you have proven it to be applicable in your organization. One way might be running though some quick ET scenarios at the end of the day, sure.
A way that I actually argue ET to people who are very structured and want to ONLY to manual scripted testing is this. I pick up a pen and ask them what they think is going to happen if I drop it. Duh! It's going to fall on the ground. I then ask them if I dropped the pen 100 tiems if they thought it would drop every time. Well, yes, of course. Why? Because of gravity? Exactly! Nothing has changed with gravity in the past billion years or whatever it might be, so why should we expect that the pen all of a sudden won't drop? So do you think it's reasonable for me to test something 10 times that hasn't been changed since the first time I tested it? Do you think that's a waste? Do you think there is something more I could be doing with my time? Sure, testing everything else! A test passed will never fail unless there is code change, or unless a variable is introduced that creates an exception within that code, in which case, we might not catch it anyway.
That's basically my sales pitch.
--------------------
Brent
--------------------
9 out of 10 people I prove wrong agree that I'm right. The other person is my wife.
--------------------
What is Holistic Testing?
|
stoofer
Member
Reged: 08/11/08
Posts: 250
|
|
Quote:
...so why should we expect that the pen all of a sudden won't drop?
because you caught it. Minority Report 
|
philk10
Advanced Member
Reged: 01/20/05
Posts: 578
Loc: UK
|
|
Just read a blog post about Is exploratory testing only for senior testers?
--------------------
and My Blog
|
Gaurav_Pandey
Member
Reged: 06/18/08
Posts: 216
Loc: The Land of Snake Charmers
|
|
Shouldn;t exploratory testing be performed when the requirements are not provided? You would have no basis to create test cases.. then it makes sense to perform exploratory testing!!
|
brentpaine
Veteran
Reged: 03/09/07
Posts: 3754
Loc: Waterloo, Ontario, Canada
|
|
Quote:
Shouldn;t exploratory testing be performed when the requirements are not provided? You would have no basis to create test cases.. then it makes sense to perform exploratory testing!!
Yes and no. That's a little like saying, I have no idea how do perform open-heart surgery, but I guess I could wing it.
The fact of the matter is that the software is being written for an audience. The size of that audience may vary, but chances are they all have one thing in common, they are all looking for your software do perform a specific task. Each may have little differences in the way they use it, but there will be a huge common knowledge base and set of expectations for the software. If you don't know that then ET will be useless.
What makes you think that you can just blast away at a software if you really don't know how it's going to be used? This is what the requirements are meant to do. Not only does it inform us on the deliverables of the product, but it tells us what to expect and what the system should do.
However, I suppose there is the possibility that you are already a domain expert in the industry your software is designed for. In this case, maybe you already know what it should do. In this case, sure, you might get away without requirements. However, for the people who don't have that information, I would recommend, still, sourcing out information on what the requirements of the software are before you do anything.
Written requirements aren't always necessary, but a general understanding of what software should do IS. This is one of the main objectives of ET, use the software as a user would. Blasting away at the software is probably going to lead to poor quality, missed defects, and ET will likely get squashed at your shop.
--------------------
Brent
--------------------
9 out of 10 people I prove wrong agree that I'm right. The other person is my wife.
--------------------
What is Holistic Testing?
Edited by brentpaine (09/18/08 09:02 AM)
|
Walen
Active Member
Reged: 05/09/01
Posts: 1064
|
|
Quote:
Written requirements aren't always necessary, but a general understanding of what software should do IS. This is one of the main objectives of ET, use the software as a user would. Blasting away at the software is probably going to lead to poor quality, missed defects, and ET will likely get squashed at your shop.
Or - as has been said many times in many ways and places - A formal requirements document may not exist, however, somewhere, there is a list of expectations. If there isn't, what did the developer/dev team create? I've seen requirements in .txt files, emails and (my favorite) on cocktail napkins/serviettes. The was a reason why software was developed - find that, and you'll find the "requirements."
No one thing will be right in all circumstances. No practice will be a "best practice" always, everytime. Context wins.
--------------------
P. Walen
|
Limin
Newbie
Reged: 09/01/08
Posts: 13
|
|
After I read the above info, i am interested in the below question:
(1) When will be the best time to do exploratory testing?
(2) What can we expect from the exploratory testing?
Many thanks!
|
SiriusDG
Member
Reged: 06/03/03
Posts: 314
Loc: Tampa, FL
|
|
Certainly, a lack of requirements docs strengthens the case for exploratory testing, but it is not required. You could have all the documentation you like...I still want to do exploratory testing. To begin with, what makes you think the application adheres to the requirements doc...and when it does not, is the app wrong, or is the doc wrong? Software is a solution...if it solves your problem, then it is working well. If it does not, then it is not...even if it agrees with the requirements doc.
I also am not sure I like the open-heart surgery comparison. That is a very very specialized and high risk field. Here is a comparison I would like better.
I have never driven a Ferrari. Or a Bentley. Or a Maserati. Or a Rolls Royce. But I have driven cars, and I know how to drive. I'll bet I could test any of those cars. Now, as soon as I sit in the seat, I may not know exactly what all the buttons and gadgets do...but I'll bet after I explore a little, I can figure it all out. And I'll bet I can drive them all instantly. So somewhere between the common elements I know, and the highly specialized elements I do not, there is a knowledge gap. The exploration fills that gap. If I had never seen a car in my life before, it would be very difficult for me to test them, and any proponent of exploratory testing would agree, experience counts. If I am a mechanic, so much the better...my testing will have additional depth and dimensions. But if your only looking for a drive ability test, it is not required. So without reading the owners manual, the "doc", while driving, I begin playing with the specialized doo dads...when I am all done, I tell you I love it all. Except for the on board GPS...and I give you half a dozen things it failed to meet my expectations on. Not that I asked it to make me tea and crumpets...just things I expect an onboard GPS to be able to do in this day and age. You respond that it is working exactly as designed. Who is right? Who is wrong? I am the user, and it is not meeting my expectations. Had I read the "doc" and only tested to the "spec" you would never know it failed to meet my needs. This is the beauty of exploratory testing...it is less concerned with "Does it work right?" and more concerned with "Is there a problem with it?" If you really want to sell that very expensive car to your very picky customers, you better put your doc on a shelf and meet their expectations.
--------------------
David Gilbert
dgilbert@sirius-sqa.com
Test Explorations (my blog)
Sirius Software Quality Associates
My LinkedIn Profile
|
Shane_MacLaughlin
Super Member
Reged: 09/22/05
Posts: 1496
Loc: Dublin, Ireland
|
|
Quote:
If you really want to sell that very expensive car to your very picky customers, you better put your doc on a shelf and meet their expectations.
There's a major problem bringing this analogy to software, or any complex product for that matter, which is this. If your product doesn't meet client expections as you 'test drive' it, you're already way too late. In my experience, and as Brent and Pete Walen have already alluded to, understanding and managing client expectations is the single most important part of any major software project. Do it wrong, and the project is likely to fail. I'd much rather do the bulk of the exploratory analysis at requirements gathering phase in conjunction with all the stakeholders. Leaving it to the tester just before the product is delivered doesn't make much sense to me. If you're using an iterative development process, with small iterations, you can cope with this problem to some degree, but it still doesn't detract from the fact that changing a design after it has been implemented is a very expensive way of developing a system.
If I have to test a product that has no requirements, my first step is to establish those requirements. This can be by exploring the product to see what it does and make assumptions on what it should do based on my own experience. Personally, I'd rather get a user / domain expert on board at this point and explore the product with them to gain the context of their expectations. Anything else assumes that I am a domain expert which is often not the case. If I was developing and new sports car, I would want a professional driver who had driven a Ferrari and a Bentley and a Masserati to test it. Saying that the guy who drives a Lada to work didn't like the GPS doesn't equate to smart testing in my book, and could result in seriously endangering the driver.
And for my money, as I strive to understand the requirements, I would tend to document them seperately from the test details and results. The thinking here is that the requirements tend to be more concise and are independant of the implementation. The "doc" that you are putting on the shelf may already contain much of this information. Boring as it may be, I'd strongly recommend reading it; it doesn't preclude you from doing exploratory testing and may even help.
--------------------
My LinkedIn profile
Edited by smacl (09/19/08 02:39 AM)
|
Previous
Index
Next
Threaded
Edit
Reply
Quote
