Oliver_Lloyd
Member
Reged: 06/04/11
Posts: 246
Loc: London, England
|
|
This is a bit of a bugbear with me, trending performance results over time, from test to test.
I did a lot of looking around and in the end wrote my own custom solution but I was curious how others have approached this issue?
--------------------
http503.com
|
Corey_G
Veteran
Reged: 09/14/01
Posts: 4281
Loc: Boston, MA
|
|
Jenkins performance plugin is what I'm using now.
https://wiki.jenkins-ci.org/display/JENKINS/Performance+Plugin
It consumes JTL files (JMeter-compatible/XML).. so I made my tool output that format. The trending/graphing is minimal, but works well.
I'm planning on doing some work in this area also, so please let us know what useful solutions you come up with.
--------------------
Corey Goldberg
Homepage: goldb.org
Twitter: twitter.com/cgoldberg
Google+: gplus.to/cgoldberg
Edited by Corey_G (03/09/12 12:00 PM)
|
James Pulley
Moderator
Reged: 08/01/01
Posts: 5551
Loc: NC
|
|
Depends upon the tool. if HP LR, then I just upload all of the table data from Analysis to a common MYSQL/SQLServer/ORACLE/... instance and use the unique test ID as a grouping item in any queries where I am looking at test to test performance.
|
Oliver_Lloyd
Member
Reged: 06/04/11
Posts: 246
Loc: London, England
|
|
James, this is the way we went in the end, stick the data in a db and create a presentation layer over it. LR has the most rocking Analysis package of all the tools for drilling into a single test but I am actually *more* interested in how my test results look when I compare them to the last build, or last week or even last year.
Corey, yeah, that plugin is a tidy fit for a simple CI solution (and probably the best option out there) but still, only one, single average, min and max value is presented per test run - you end up with a strong urge to drill down...
--------------------
http503.com
|
Corey_G
Veteran
Reged: 09/14/01
Posts: 4281
Loc: Boston, MA
|
|
Quote:
Corey, yeah, that plugin is a tidy fit for a simple CI solution (and probably the best option out there) but still, only one, single average, min and max value is presented per test run - you end up with a strong urge to drill down...
agreed. from my position, I can either enhance the Performance plugin (in Java [throws up in mouth])... or I can write my own solution that is decoupled from the CI system.
I'd like to save my results in either XML/JSON, and use a document database (nosql) to get trend data. (time series graphs of run-over-run metrics). I have the stack in mind... just need the time to start working on it.
-Corey
--------------------
Corey Goldberg
Homepage: goldb.org
Twitter: twitter.com/cgoldberg
Google+: gplus.to/cgoldberg
|
Oliver_Lloyd
Member
Reged: 06/04/11
Posts: 246
Loc: London, England
|
|
Is nosql the way to go for this sort of thing? I get the impression aggregation is messy and aggregation is the core of what you will be doing. But then I only played with indexeddb which is horrible for anything more advanced than looping over cursors.
I wanted to use client side storage for what I did and because indexddb is naff at the moment ended up going with Web SQL (despite it being depreciated) but I guess that road is a dead end so I might have to change my thinking eventually.
Re CI. Agreed. It makes no sense to integrate perf. tests with CI. I think regular (automated) testing is key but after every single build? Realistically it takes time to run a perf./load test and time (human time) to analyse the results...and herein lies the problem, speeding up that analysis requires a decent presentation layer.
--------------------
http503.com
|
Corey_G
Veteran
Reged: 09/14/01
Posts: 4281
Loc: Boston, MA
|
|
Quote:
Is nosql the way to go for this sort of thing? I get the impression aggregation is messy and aggregation is the core of what you will be doing.
I'm thinking specifically of nosql document databases like mongodb or couchdb, that are made for storing/querying json documents. So you save your results in json and just push them to the nosqldb, and write views for trending/querying across document data. A relational db is a trusted choice as well, but document db's (i think) were made for this sorta thing.
Quote:
Re CI. Agreed. It makes no sense to integrate perf. tests with CI. I think regular (automated) testing is key but after every single build? Realistically it takes time to run a perf./load test and time (human time) to analyse the results...and herein lies the problem, speeding up that analysis requires a decent presentation layer.
totally agree about speeding up analysis... but I think there *is* benefit of coupling perf with a CI system (though maybe not your primary build/CI setup). while you may not analyze results from every perf run triggered from a build... you can have the reports automatically, so they are at least *there* if you need to dig into issues. What I actually do is have a separate CI system that runs my perf tests. So I have a Jenkins setup that basically just uses the task scheduling and reporting features. Perf tests are triggered to run nightly against a staging environment. and reports are automagically generated.
Now just need the ability to analyze build-over-build or time-based trending.
--------------------
Corey Goldberg
Homepage: goldb.org
Twitter: twitter.com/cgoldberg
Google+: gplus.to/cgoldberg
|
Oliver_Lloyd
Member
Reged: 06/04/11
Posts: 246
Loc: London, England
|
|
OK, I looked very briefly into mongo and yes, that is easily up to the job. I never really got started with investigating it because right from the start I gave myself the requirement of using a client side database and so my view got tarnished by indexeddb (which only recently got a count function, sheesh).
Anyhow, I'll send you a link to what we have so far; I'll probably put it out there in a few weeks (I'll update this post with the details then).
PS. Yes, triggering perf. runs using the CI system is good, totally agree, just not after every commit! Nightly runs is just enough to perk me up in the mornings but not so much that I get buried in data.
--------------------
http503.com
|
James Pulley
Moderator
Reged: 08/01/01
Posts: 5551
Loc: NC
|
|
Oliver, As you have LR. Have you notices that you can store all of your analysis sessions to a Microsoft SQL Server instance and then recall them back through the Analysis tool? While analysis is good for a cross compare for a couple of tests what I really find useful is once the data is in the DB I can run all sorts of queries against the source to pull out what I need for long term trend analysis.
Ironically there are some indexes that need to be added to the schema used for storing the data in order to make the DB work more efficiently (read, no long table scans) when storing the data in the DB. As this isn't a LR forume, per se, I just thought I would put this out there in case you were/were not using it. Feel free to hit me privately on the schema changes to improve the efficiency of the Analysis tool as well as your own queries against the data.
You would think that a company that makes performance testing tools would provide a schema which was tuned for performance. By the way, the other tool vendors have been guilty of this one as well....side effect of designing a lot of these tools when the data sets used in analysis were far smaller and the delays only minimal for poor schema designs.
|
Previous
Index
Next
Threaded
Edit
Reply
Quote
