-20-something web services to be tested
-Around 400 concurrent virtual users
-Application is deployed in ec2 cloud
-JMeter to be used in ec2 for the testing.
-Management does not agree to procuring reserved instances so we have to go with on-demand instances. Therefore JMeter installation and data will be lost each time I terminate the instances.
I would like to ask for your suggestion - Which approach will be best among these two? -
1. Go for manual configuration of Jmeter master-slave setup in the cloud. If I go for Large Windows ec2 instance (4 x 1.2 GHz CPU) then i will probably need 1 master and 4 slaves for the above user load, considering that Web Services scripts are particularly resource hungry.
-This will involve a lot of setup overheads. Launching each instance manually every day, installing JMeter everytime in all the instances, configuring properties file and all the associated setup, Opening of ports(?), mapping with elastic IP and all. This alone might take upto 2-3 hours before every test run.
However I will have complete control on my installation and master and slave machines.
2. Go for the "JMeter in Cloud" AMI that is available in Amazon. (Amazon AMI)
This is a free AMI written by Jorg and his team and allows for automatic setup for most of the above activities.
However this approach also has some risks involved.
This AMI is private- that is, we will be able to use this AMI only if Jorg includes our account ID in his security group. Also, his AMI mandates that we go with Linux, Small, On-demand instances only. Small instances have 1.2 GHz CPU. So for simulating 400 users, we need 15-16 slaves. When there are so many slaves, and around 20 WebService scripts running from one master (which is also Small)... it will mean a HUGE risk of JMeter crashing suddenly/ OOM exceptions/ loss of results data/ repeated rework effort.
Also, I am not sure whether we can install extra JMeter plugins in the AMI which comes with pre-installed JMeter.
Cost-wise, the manual approach will cost around $100 more for 2 weeks of test execution window.
With pros and cons of each approach, i am really at a loss which one to go with.
Please help with your insights.
Without bogging you down- There is an actual testing cloud focused on JMeter(and is 100% compatible with JMeter scripts), you just upload your current scripts.
BlazeMeter as the testing cloud is called, has on-demand servers, all the data is stored in your account and it's available all the time. Also- offering a free trial (no credit card needed) with ten credits which provides 1 JMeter Console, 1 Medium Server and a Headless Configuration
Each credit (of the ten) gets you 1 medium sized server hour (which drives between 100-300 users per hour).You can also opt for automatic scripting. BlazeMeter.com. Let me know if I can help further. Daniela