SPONSORS:






User Tag List

Thanks Thanks:  0
Likes Likes:  0
Dislikes Dislikes:  0
Results 1 to 4 of 4
  1. #1
    Senior Member
    Join Date
    Jun 2011
    Location
    London, England
    Posts
    243
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Performance Testing as part of CI? Discuss...

    It's all about Continuous Integration these days, but it seems to me integrating performance testing into a CI process is young. I've had this as a side project for a while and now it has got picked up and turned into a project itself. It's an attractive sounding concept and the people I work for are keen to adopt it.

    So I've been having these conversations with people about CI and it strikes me people recognise it as a very functionally orientated concept. Everyone likes the idea where an email is sent to the developer who broke the build, or better yet have their avatar flash up on a screen for everyone to see. And dashboards? Man, these are like IT porn.

    But how to achieve this for performance testing? Can this be done? My current train of thought is that to try and run perf. tests 'continuously' sounds...well, it would use a lot of disk space if nothing else. Perhaps it is better to keep full performance runs limited to a nightly build?

    But then I start to think it might be useful to have limited component tests run as part of CI and that full scale performance tests are better kept hands on, or if they run every night then this is just automation and not really continuous anything.

  2. #2
    Moderator
    Join Date
    Sep 2001
    Location
    Boston, MA
    Posts
    4,348
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Performance Testing as part of CI? Discuss...

    great topic.. Perf/CI integration has been on my mind as well..

    I run nightly tests (scheduled with Jenkins CI-server) against a staging environment to get continuous performance tests.

    Once a test is created and modeled, it goes into my performance harness and gets added to the nightly scheduler.

    I watch for performance trends or regressions over time.


    I did a screencast showing my setup:

    "Performance CI - Jenkins and Multi-Mechanize"
    http://www.youtube.com/watch?v=HatB6IlCLEs


    Corey Goldberg
    Homepage: goldb.org
    Twitter: twitter.com/cgoldberg
    Google+: gplus.to/cgoldberg

  3. #3
    Senior Member
    Join Date
    Jun 2011
    Location
    London, England
    Posts
    243
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Performance Testing as part of CI? Discuss...

    "Trends." Exactly. This is very much the reason why I started playing around with this.

    I often find myself building spreadsheets on the fly to analyse results from multiple test runs over time. But this quickly gets messy so I decided I needed a database with all my result sets in that I could query at will. Once you start thinking like this then automating the data import is a natural step, and from there automating the actual test execution is not much extra effort.

    So I put everything into a shell script and ran it as a cron job. After that I created a front end dashboard that polled a php script reading from mysql every x minutes, sat back, and felt rather smug.

    Then I realised our build server could call the shell script just as well as using cron and suddenly I had designed a CI system!

    But now I am close to folding in all the scripts and turning it on I suddenly find there are a lot of questions about how this thing should be setup.

    Things like frequency of execution and also what should be executed are strangely open.

    For example, I am torn between:

    1. Making the test rich. Folding in monitoring and having this recorded in the same db so each (nightly) run is a standard full test, with memory, threads, cpu, etc. etc.

    2. Keeping things simple. Only having component tests, this gives me delta results over time but further investigation would need to be done manually.

    The solution needs to be 'useful' without costing too much time, so I'm thinking option 2 is wise, but my inner geek thinks option 1 would be more fun...

  4. #4
    Moderator
    Join Date
    Mar 2002
    Location
    Tampa Bay
    Posts
    7,673
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Performance Testing as part of CI? Discuss...

    Solution 1 sounds like what my boss (several years ago) only-half jokingly referred to as the "big red button." (i.e hitting the button tests everything.)

    I thought of constructing one for him with the circuit inside designed to display "passed" as a Christmas present. But I didn't get around to it before I was laid off. (No sour grapes - I knew it was coming, and got a great severance package.)

    I love the concept, but I admit to the impracticality.

    Footnote: When I was studying for my Masters in the early-mid 1990's, I was struggling to come up with a thesis topic. (I am too lazy to study for a comprehensive axam, but I will happily spend hundreds of hours locating, reading and quoting several dozen papers in a document that only ends up 80 pages long.) My early choice was "The Universal Interface." i.e. something that everyone could use, that interfaced with everything (more or less electronic) known to man. Perhaps fortunately, my major professor talked me out of it!

 

 

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Search Engine Optimisation provided by DragonByte SEO v2.0.36 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Resources saved on this page: MySQL 12.50%
vBulletin Optimisation provided by vB Optimise v2.6.4 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
User Alert System provided by Advanced User Tagging v3.2.8 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
vBNominate (Lite) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Feedback Buttons provided by Advanced Post Thanks / Like (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Username Changing provided by Username Change (Free) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
BetaSoft Inc.
Digital Point modules: Sphinx-based search
All times are GMT -8. The time now is 05:49 AM.

Copyright BetaSoft Inc.