The online community for software testing & quality assurance professionals
 
 
Calendar   Today's Topics
Sponsors:
Lost Password?

Home
BetaSoft
Blogs
Jobs
Training
News
Links
Downloads



Software Testing >> Performance & Load Testing

Pages: 1
Oliver_Lloyd
Member


Reged: 06/04/11
Posts: 246
Loc: London, England
Trending - What tool or method do you use?
      #700957 - 03/09/12 10:01 AM

This is a bit of a bugbear with me, trending performance results over time, from test to test.

I did a lot of looking around and in the end wrote my own custom solution but I was curious how others have approached this issue?

--------------------
http503.com


Post Extras: Print Post   Remind Me!   Notify Moderator  
Corey_GModerator
Veteran


Reged: 09/14/01
Posts: 4281
Loc: Boston, MA
Re: Trending - What tool or method do you use? [Re: Oliver_Lloyd]
      #700962 - 03/09/12 10:29 AM

Jenkins performance plugin is what I'm using now.

https://wiki.jenkins-ci.org/display/JENKINS/Performance+Plugin

It consumes JTL files (JMeter-compatible/XML).. so I made my tool output that format. The trending/graphing is minimal, but works well.

I'm planning on doing some work in this area also, so please let us know what useful solutions you come up with.

--------------------
Corey Goldberg
Homepage: goldb.org
Twitter: twitter.com/cgoldberg
Google+: gplus.to/cgoldberg

Edited by Corey_G (03/09/12 12:00 PM)


Post Extras: Print Post   Remind Me!   Notify Moderator  
James Pulley
Moderator


Reged: 08/01/01
Posts: 5551
Loc: NC
Re: Trending - What tool or method do you use? [Re: Corey_G]
      #700975 - 03/09/12 12:00 PM

Depends upon the tool. if HP LR, then I just upload all of the table data from Analysis to a common MYSQL/SQLServer/ORACLE/... instance and use the unique test ID as a grouping item in any queries where I am looking at test to test performance.

Post Extras: Print Post   Remind Me!   Notify Moderator  
Oliver_Lloyd
Member


Reged: 06/04/11
Posts: 246
Loc: London, England
Re: Trending - What tool or method do you use? [Re: James Pulley]
      #700981 - 03/09/12 02:15 PM

James, this is the way we went in the end, stick the data in a db and create a presentation layer over it. LR has the most rocking Analysis package of all the tools for drilling into a single test but I am actually *more* interested in how my test results look when I compare them to the last build, or last week or even last year.

Corey, yeah, that plugin is a tidy fit for a simple CI solution (and probably the best option out there) but still, only one, single average, min and max value is presented per test run - you end up with a strong urge to drill down...

--------------------
http503.com


Post Extras: Print Post   Remind Me!   Notify Moderator  
Corey_GModerator
Veteran


Reged: 09/14/01
Posts: 4281
Loc: Boston, MA
Re: Trending - What tool or method do you use? [Re: Oliver_Lloyd]
      #700985 - 03/09/12 03:03 PM

Quote:

Corey, yeah, that plugin is a tidy fit for a simple CI solution (and probably the best option out there) but still, only one, single average, min and max value is presented per test run - you end up with a strong urge to drill down...




agreed. from my position, I can either enhance the Performance plugin (in Java [throws up in mouth])... or I can write my own solution that is decoupled from the CI system.

I'd like to save my results in either XML/JSON, and use a document database (nosql) to get trend data. (time series graphs of run-over-run metrics). I have the stack in mind... just need the time to start working on it.

-Corey

--------------------
Corey Goldberg
Homepage: goldb.org
Twitter: twitter.com/cgoldberg
Google+: gplus.to/cgoldberg


Post Extras: Print Post   Remind Me!   Notify Moderator  
Oliver_Lloyd
Member


Reged: 06/04/11
Posts: 246
Loc: London, England
Re: Trending - What tool or method do you use? [Re: Corey_G]
      #700995 - 03/10/12 04:29 AM

Is nosql the way to go for this sort of thing? I get the impression aggregation is messy and aggregation is the core of what you will be doing. But then I only played with indexeddb which is horrible for anything more advanced than looping over cursors.

I wanted to use client side storage for what I did and because indexddb is naff at the moment ended up going with Web SQL (despite it being depreciated) but I guess that road is a dead end so I might have to change my thinking eventually.

Re CI. Agreed. It makes no sense to integrate perf. tests with CI. I think regular (automated) testing is key but after every single build? Realistically it takes time to run a perf./load test and time (human time) to analyse the results...and herein lies the problem, speeding up that analysis requires a decent presentation layer.

--------------------
http503.com


Post Extras: Print Post   Remind Me!   Notify Moderator  
Corey_GModerator
Veteran


Reged: 09/14/01
Posts: 4281
Loc: Boston, MA
Re: Trending - What tool or method do you use? [Re: Oliver_Lloyd]
      #701000 - 03/10/12 10:40 AM

Quote:

Is nosql the way to go for this sort of thing? I get the impression aggregation is messy and aggregation is the core of what you will be doing.




I'm thinking specifically of nosql document databases like mongodb or couchdb, that are made for storing/querying json documents. So you save your results in json and just push them to the nosqldb, and write views for trending/querying across document data. A relational db is a trusted choice as well, but document db's (i think) were made for this sorta thing.

Quote:

Re CI. Agreed. It makes no sense to integrate perf. tests with CI. I think regular (automated) testing is key but after every single build? Realistically it takes time to run a perf./load test and time (human time) to analyse the results...and herein lies the problem, speeding up that analysis requires a decent presentation layer.




totally agree about speeding up analysis... but I think there *is* benefit of coupling perf with a CI system (though maybe not your primary build/CI setup). while you may not analyze results from every perf run triggered from a build... you can have the reports automatically, so they are at least *there* if you need to dig into issues. What I actually do is have a separate CI system that runs my perf tests. So I have a Jenkins setup that basically just uses the task scheduling and reporting features. Perf tests are triggered to run nightly against a staging environment. and reports are automagically generated.

Now just need the ability to analyze build-over-build or time-based trending.

--------------------
Corey Goldberg
Homepage: goldb.org
Twitter: twitter.com/cgoldberg
Google+: gplus.to/cgoldberg


Post Extras: Print Post   Remind Me!   Notify Moderator  
Oliver_Lloyd
Member


Reged: 06/04/11
Posts: 246
Loc: London, England
Re: Trending - What tool or method do you use? [Re: Corey_G]
      #701004 - 03/10/12 04:13 PM

OK, I looked very briefly into mongo and yes, that is easily up to the job. I never really got started with investigating it because right from the start I gave myself the requirement of using a client side database and so my view got tarnished by indexeddb (which only recently got a count function, sheesh).

Anyhow, I'll send you a link to what we have so far; I'll probably put it out there in a few weeks (I'll update this post with the details then).

PS. Yes, triggering perf. runs using the CI system is good, totally agree, just not after every commit! Nightly runs is just enough to perk me up in the mornings but not so much that I get buried in data.

--------------------
http503.com


Post Extras: Print Post   Remind Me!   Notify Moderator  
James Pulley
Moderator


Reged: 08/01/01
Posts: 5551
Loc: NC
Re: Trending - What tool or method do you use? [Re: Oliver_Lloyd]
      #701130 - 03/12/12 09:03 AM

Oliver, As you have LR. Have you notices that you can store all of your analysis sessions to a Microsoft SQL Server instance and then recall them back through the Analysis tool? While analysis is good for a cross compare for a couple of tests what I really find useful is once the data is in the DB I can run all sorts of queries against the source to pull out what I need for long term trend analysis.

Ironically there are some indexes that need to be added to the schema used for storing the data in order to make the DB work more efficiently (read, no long table scans) when storing the data in the DB. As this isn't a LR forume, per se, I just thought I would put this out there in case you were/were not using it. Feel free to hit me privately on the schema changes to improve the efficiency of the Analysis tool as well as your own queries against the data.

You would think that a company that makes performance testing tools would provide a schema which was tuned for performance. By the way, the other tool vendors have been guilty of this one as well....side effect of designing a lot of these tools when the data sets used in analysis were far smaller and the delays only minimal for poor schema designs.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Pages: 1



Extra information
0 registered and 10 anonymous users are browsing this forum.

Moderator:  AJ, RSBarber, rstens, Corey_G 

Print Topic

Forum Permissions
      You cannot start new topics
      You cannot reply to topics
      HTML is disabled
      UBBCode is enabled

Rating:
Topic views: 1569

Rate this topic

Jump to

Contact Us | Privacy statement SQAForums

Powered by UBB.threads™ 6.5.5