SPONSORS:






User Tag List

Thanks Thanks:  0
Likes Likes:  0
Dislikes Dislikes:  0
Results 1 to 10 of 10
  1. #1
    Member
    Join Date
    Apr 2002
    Location
    Zwolle, The Netherlands
    Posts
    57
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Automated testing in eXtreme Programming

    Does anyone of you have experience in automated testing in a project that was agile / using eXtreme Programming (XP)?
    Would like to hear from you!


    ------------------
    **********************
    XP - for eXtreme Programming
    Test - for being a professional tester
    NL - for Netherlands
    “None of us is as smart as all of us” - Gerald Weinberg

  2. #2
    Advanced Member
    Join Date
    Jan 2002
    Location
    ma
    Posts
    531
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Automated testing in eXtreme Programming

    What would you like to know? Extreme programming is a broad discussion and I wouldn't know where to begin.

    First of all, testing in XP is all about unit-testing. Most people in QA do not do unit-testing. Even at functional level, it's rare to find talented people in QA who can actually code. These guys are very rare, and hard to find. Also, QA people (at this level) think things in a different angle. Developers think in "I designed to do x, so I need to make sure x works". Where as QA people often think in "Customer wants to do A, so how can I make X do A. What happens with customer wants B? Then how does X react?". These may seem to be the same, but are quite different.

    Also, functional tests at this level are often not executed at the system level... Unless you have SDK type of software where API functionality is exposed, it may not make sense to do this. (Though XP suggests that test harness is built into the code, and you can execute them at various points.)

    So, what do you want to know about XP and QA?

    ------------------

  3. #3
    Member
    Join Date
    Apr 2002
    Location
    Zwolle, The Netherlands
    Posts
    57
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Automated testing in eXtreme Programming

    Well , first of all it seems to me that you have to be an experienced automated tester because the requirements can change fast, so your automated test can change while you're working on it. I haven't read anywhere yet such an assumption. Is it true?

    Second of all is the test strategy. You mentioned that developers do the unit tests. And in your automated tests you most if all use Use Cases. Is that enough? Don't you need more test techniques to fully cover the system? To me it looks like you don't test the actual functionality. In traditional development -the V-model- you have programmertests, system tests, functional tests, integration tests and acceptance tests. It looks like XP doesn't cover it all. But then again: is that necessary?

    And I'm also interested in the "Do's en Don'ts" of automated testing in XP-projects.

    Anko

    ------------------
    **********************
    XP - for eXtreme Programming
    Test - for being a professional tester
    NL - for Netherlands
    “None of us is as smart as all of us” - Gerald Weinberg

  4. #4
    Senior Member
    Join Date
    Jul 2002
    Location
    Palm Bay, FL USA
    Posts
    2,346
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Automated testing in eXtreme Programming

    I don't claim to be an XP expert, but I will tell you that if you are looking for someone other than the client to do real functional testing - and automating it, that sounds like a VERY robust XP effort.

    The important thing to remember about automation is that it takes longer to automate a script than it does to manually test the site. Rational says that you souldn't automate unless you will be executing the test at least 5 times. I would argue that number reduces to 2 or 3 for an automation expert, but either way, how many tests do you KNOW will be executed that many times before you start in your XP development effort?

    I would use that as my guide - "will automating save me time?"

    As far as unit testing vs. functional testing. If you do have an "expert automater" at your disposal, it is not overly difficult to build a little test harness for automating unit testing, but once again, I ask if it's really worth the time.

    What do you hope to gain through automation?

    ------------------
    Scott Barber
    NOBLE(STAR
    Sr. Performance Engineer
    sbarber@noblestar.com
    http://www.noblestar.com
    Scott Barber
    Chief Technologist, PerfTestPlus
    Executive Director, Association for Software Testing
    Co-Author, Performance Testing Guidance for Web Applications
    sbarber@perftestplus.com

    If you can see it in your mind...
    you will find it in your life.

  5. #5
    Member
    Join Date
    Apr 2002
    Location
    Zwolle, The Netherlands
    Posts
    57
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Automated testing in eXtreme Programming

    <BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>Originally posted by RSBarber:

    What do you hope to gain through automation?

    <HR></BLOCKQUOTE>

    Well, I just wasn't sure if you had to automate. XP tells you to do so, but I'm glad you make some remarks. You have to be sure that it pays back.

    We are currently in the situation that we want to make our process more agile -so that is where XP comes around. On the other hand we are introducing a test tool for the legacy code.
    I realize that we don't have to mix things up - and take one step at a time. For new functionality we do it the old way (manually), for the legacy code we set up the tool. When the new functionality has been delivered we will start automating the scripts. Or maybe we try to automate the new stuff, but not making it part of the critical path. But that will probably take some time because we first have to get to know the test tool.

    ------------------
    XP - for eXtreme Programming, or at least agile
    Test - for being a professional tester
    NL - for Netherlands (do you actually know where that is?)
    “None of us is as smart as all of us” - Gerald Weinberg

  6. #6
    Senior Member
    Join Date
    Jul 2002
    Location
    Palm Bay, FL USA
    Posts
    2,346
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Automated testing in eXtreme Programming

    I think that is the approach I would follow in the situation you describe. Manually test as quickly and efficiently as possible to get the code ready for production, then automate for regression testing whenever you find yourself not "behind the eight-ball", so to speak, to keep up with development.

    As you get more experience with the test tool, you may find that you automate most things on the spot.

    My believe is that development should never have to WAIT due to test automation.

    ------------------
    Scott Barber
    NOBLE(STAR
    Sr. Performance Engineer
    sbarber@noblestar.com
    http://www.noblestar.com
    Scott Barber
    Chief Technologist, PerfTestPlus
    Executive Director, Association for Software Testing
    Co-Author, Performance Testing Guidance for Web Applications
    sbarber@perftestplus.com

    If you can see it in your mind...
    you will find it in your life.

  7. #7
    Advanced Member
    Join Date
    Jan 2002
    Location
    ma
    Posts
    531
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Automated testing in eXtreme Programming

    Is it tough?
    If you don't have the experience, JUnit is tough. It is a coding effort. You have to call a method, catch exceptions, and write additional code to test the code. Unless you know how to program, I recommend that you avoid it. Though if you need to test SDK, and API, it is better then writing your automation frame-work. If you are new to this, it may be better to stick to macro-recoding stuff at first.

    Test cases vs. Unit tests:
    Given time, QA should focus on various test cases. That is because unit tests (unless you are talking about a SDK/API that is exposed and used by a customer) exercises internal code that is invisible to the end-user. You should know how the internals of the product know so you know what to test, and what types of tests you need to conduct. Though you will realize that sometimes, testing the code is a lot easier.

    Do's and Don't-s:
    I'd recommend downloding the j-unit and reading the documentation. Besides, this is a rather big topic since xp covers from coding all the way to relase. There are books on this topic. If you are interested in this stuff, I recommend 'Code Complete' and 'Rapid Development'. They are pretty thorough and since they were written by the same author, they cover almost from end to end.


    As for genearl test strategy, here's my opinion:
    The use of test strategy differs greatly by the team/individual's experience. Some groups think of test effort in product release level. (Version 1.0, 1.1, 2.0), while others view things down to granurality all the way down to view/branch, which is even before merging the code. Remember that there may be 1 release to QA, but within a development effort in a large group, you may have 20-40 developers coding in parallel and each have their branch. In our company, one group has an automation tool that developers HAVE to run prior to submitting the changes to the main branch. This guarantees that their code changes will not introduce any regression. XP is another tool that allows for such thing. Look at ANT and you will see how the XP model all fits in.

    Level of testing involvement depends on experience/expertise of development/QA staff. More seasoned and experienced the tester becomes, his/her involvement with development becomes more synergetic. Few people have had the chance to work with talented testers who can improve quality throughout. These are rare and for those, automation is not just for finding bugs. Most people think finding more bugs is all there is to it, but that's what a 'tester' thinks. QA is about improving product quality. Quality can not be tested at the end. It has to be 'built-in' from the beginning. To do this, you have to implement various things. (Too long to discuss here)
    As the test automation matures and defects can be detected, the next stage is identification. By inserting automation into various points of development cycle, you can detect when the defect was introduced. The goal here is to identify the root-cause of the defect and to identify at what point the defect was introduced. This improves development cycle greatly by reducing development's effort to track bugs down. Often, defects are introduced and not detected for a long time. This costs hundreds of hours to track down, and identify the origin of defects. Eliminating this process stabilizes the code greatly. Third stage is prevention. As automation is accepted, then the tool can be used to prevent defects from entering into the code. This is 'where' the defect is introduced. Often, this can be achieved through the process of validating the code prior to merge. Also, developers can find out from various logs and steps if their code merges were executed.

    All the above stages can be tied down to various tools, such as defect tracking, build, to assist in development reliable software. Also, it helps QA do it's job by focusing on adding test cases, and trying to break the software, rather then making sure things work.

    So why don't Segue, Rational, Mercury and RSW do not talk about these things? (Not to pick on these guys but...)
    Remember, these are companies selling their solution. In fact, you would think that Rational would be able to provide a template which allows for things to plug in. But all of these guys have one very common (and deadly) requirement flaw. That is 'platform independence'. For those who say 'It doesn't matter', then ask why languages like 'java', 'perl', and even 'c++' dominates the market. These languages were championed for their 'platform-neutrality'. Unfortunatly, for all the test companies, they had to pick a platform since a.) It costs too much to support multiple platforms b.)Until the past 2-3 years, there was no single platform solution that was acceptable and they can't pump out a product that fast. c.)Not much dememand. Though automation is claimed to be popular, manual test is still the most popular form of test. and d.)There are better tools out there. Couple of big ones include perl and ant.

    With that in mind, you need to discuss and think what you need in the future. Having a seasoned staff is important. Avoid contractors. (sorry guys, but this is my opinion) They are great, but they don't have the passion since they are looking for the next job. Hire a full time person with good amount of experince, who is interested in doing the job, and not selling you some idea. Hiring/finding someone is tough, since there are many who pretend to know these things. (Maybe I'm one of them? :-))

    Good Luck!

    ------------------

  8. #8
    Senior Member
    Join Date
    Jan 2001
    Location
    Grandville, MI, USA
    Posts
    201
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Automated testing in eXtreme Programming

    "Does anyone of you have experience in automated testing in a project that was agile / using eXtreme Programming (XP)?"

    Yup!

    I've been working in an XP environment for a year and a half now and when we follow XP to the letter of the law (if there is such a thing), the developers do the Unit testing (really, they do!) and I do the integration (GUI/usability) testing.

    We use automation all along the way, with some good results and some not so good results. You do have to be fast and flexible... but when you're working off of story boards or cards, requirements are plain, simple and straight forward. If we can't test those requirements on the cards, we rip them up and write new ones.

    I've even gone so far as to write the requirements for one of our projects! Just push the marketing people aside and write them yourself. They give me all of the inputs necessary to create the stories and I break them down into manageable tasks (1-4 hours worth of development work).

    Rotate teams frequently and hit the ground running.

    With XP, not only do you get some very well designed requirements, you can pick and choose which requirements you implement during each iterration. Usually the underlying features first and then move on to the most important features for the customers.

    Manual test each of the features (directly from the story cards) and make sure they work. Then automate the same tests to be used for regression testing with the next build. If you find a problem, list in a tracking package and automate it immediately to run it in the next round of regression tests.

    You're almost guaranteed not to let the defect slip out to the "real world" if there's an automated test looking for it.

    As far as the automation is concerned, I usually run one ittereation behind the development team. So no product is ever more than 2 weeks out of my site... and after each 2 week iterration we have something that has been tested, verified and is ready to ship!

    You've got to be on your toes to pull all of this off and you have to work side by side with the development team to get this to work. There's no wall to toss stuff over since we go through the daily stand-up meetings and air out all of our dirty laundry... they know when they've checked in some bad code because I'm allowed to tell them.

    They also know when they've checked in some great code because I tell them that too!

    I think XP is a great system for us... is it right for you? Only time will tell.

    ------------------

  9. #9
    Senior Member
    Join Date
    Dec 2001
    Location
    Greensboro
    Posts
    544
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Automated testing in eXtreme Programming

    If there is one thing you should notice from all of these responses, and this true regardless of whether you are working in an XP environment or not, it is that automation really is aimed at regression testing. That's for a very simple reason...why build a test that you know will probably not work the next time around due to AUT changes? It doesn't pay. Point is - you need to pick what makes sense. Find those areas of the app that are 'stable' and work those tests. XP is really no different than the basic QA model in my opinion...the only difference is that if you are going to claim that you doing XP then you have no excuse not to get QA involved right off the bat. Places that aren't doing XP are basically places that don't realize the need for the integration of QA with development and who see it as the last stop on the assembly line. XP corps just kinda have it right....

    ------------------
    www.turbotester.com

  10. #10
    Member
    Join Date
    Apr 2002
    Location
    Zwolle, The Netherlands
    Posts
    57
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Automated testing in eXtreme Programming

    <BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>Originally posted by broop:
    Manual test each of the features (directly from the story cards) and make sure they work. Then automate the same tests to be used for regression testing with the next build.
    As far as the automation is concerned, I usually run one ittereation behind the development team. So no product is ever more than 2 weeks out of my site... and after each 2 week iterration we have something that has been tested, verified and is ready to ship!

    I think XP is a great system for us... is it right for you? Only time will tell.

    [/B]<HR></BLOCKQUOTE>

    Thank you Broop for your reply! I already thought that I must have misunderstood that from XP: automate all tests in the same iteration as you work on. Seemed to me like an impossible job. So okay, we manually test first en then automate it in the next iteration.

    If XP will work for us: we have two problems.
    1) We don't have *one* customer, we have 400 customers (we're selling our system on the free market). So we can't ask "the customer" what he wants. That makes it hard to clearly define the cutsomers role.
    2) We are not developing in Java or any OO language, but in Progress 4GL. A lot of the tools for XP (such as JUnit) are aimed at OO. We can't do anything with that - there isn't even a unit test framework for Progress, and according to some of our developers it's quite hard to make one up.
    Also: because we develop in progress, it quite difficult to automate the tests, none of the vendors do support the platform. We are currenlty "investigating" QARun and it looks like that -with some remarks- we can set up automated testing.

    ------------------
    XP - for eXtreme Programming, or at least agile
    Test - for being a professional tester
    NL - for Netherlands (do you actually know where that is?)
    “None of us is as smart as all of us” - Gerald Weinberg

 

 

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Search Engine Optimisation provided by DragonByte SEO v2.0.36 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Resources saved on this page: MySQL 9.38%
vBulletin Optimisation provided by vB Optimise v2.6.4 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
User Alert System provided by Advanced User Tagging v3.2.8 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
vBNominate (Lite) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Feedback Buttons provided by Advanced Post Thanks / Like (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Username Changing provided by Username Change (Free) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
BetaSoft Inc.
Digital Point modules: Sphinx-based search
All times are GMT -8. The time now is 03:59 AM.

Copyright BetaSoft Inc.