What Skills Performance Testers Need ?
From time to time I see questions on different forums asking what skills are necessary for performance testers. There were pretty interesting discussions like http://www.sqaforums.com/showflat.ph...ge=0&vc=1. Looks like most experts agree that performance testing requires more skills than just knowledge about how to create a script for a particular load testing tool. While it is still possible to imagine a performance tester in a large corporation with deep specialization who only creates scripts and mechanically runs them while other performance experts monitor the system and analyze results, I don't see many perspectives neither for this person, nor for the approach. Systems become so complicated now that the sum of specialized expert views doesn't give the whole performance picture.
Thinking about skills necessary for performance testing I'd specify the following areas in addition to load testing per se:
- What is going on with the system? Monitoring and Performance Analysis.
- You got results. What if
? Modeling and Capacity Planning.
- We see the bottleneck. What to do? Tuning and System Performance Engineering.
- Tuning doesnt help the bottleneck is in the application. Go back to Software Performance Engineering.
- And write, present, communicate, organize all the time.
You probably need to know about tuning and have understanding of how applications should be designed to perform well (Software Performance Engineering). You don't need to be an expert, for example, in database tuning most companies have DBAs for that but you need to speak to a DBA in his language to coordinate efforts effectively. Or raise concerns about performance consequences of the current application design. Unfortunately it is not easy you need to know enough to understand what is going on and communicate effectively.
The question is how to get such skills. Constant self-learning and gradually getting experience? Yes, of course, but that takes a lot of time. Moreover, many areas are pretty hard to jump into from a scratch. You need to get some basic understanding before you will be comfortable enough to learn further yourself. Go to a class? Definitely for a performance testing class and your main tool. But for many other different products you are working with? Usually there are several week-long performance-related classes for each product. They are developed for specialists making a living tuning this particular product. You don't have time to go to all classes and usually you don't need to dive so deep. Talk to an expert? Sure, if you find one around. Performance experts are sparse and busy, so you'd better have well-prepared question(s) hard to do if you know little about the subject.
When you get well enough along the road you get into another trap. You already know enough that basic training won't be beneficial. Still there are almost no advanced classes at all for performance testers when you go beyond basics, details of environments, tools, systems, application, etc. become so different that it makes no sense to make a class for specific combinations. You know areas where you need more information, you need to verify your approaches and practices against other experts, you need more advanced tips and tricks, and you need to find somebody you may discuss your problems with.
I believe that a good conference is a solution in both cases. Somebody digests information and presents it back to you. Not that it is an absolute ideal quality of presentation and presenters vary but probably the most effective way when you need to jump into many different topics. At the moment I am aware about one open and practical conference covering all performance-related aspects: Computer Measurement Group (CMG) http://www.cmg.org/conference/cmg2007/. With over 150 sessions related to performance, it allows everybody to find something interesting. Let's look at the coming CMG conference http://www.cmg.org/conference/cmg2007/ from a performance tester point of view.
CMG has a Load Testing focus track this year with great presentations:
- Performance Testing: A Heuristic Approach by Scott Barber, PerfTestPlus
- Software & Systems Performance Assurance Providing Value Beyond Performance Testing by Andy Bolton, Capacitas
- Automating the Analysis of Load Test Results to Assess the Scalability and Stability of a Component by Dr. Andre B. Bondi, Siemens Corporate Research
- Performance Testing IS a Spectator Sport: Accelerate Testing Cycles with Collaborative Testing by Dan Koloski, Empirix
- An Approach for Accurately Recreating Web Workloads from Production Data by
Suresh K. Khemka, Nikhil Venugopal, Gaurav Caprihan, Infosys
- Normalizing Load Testing Runs - What to do When Load Testing Runs are Inconsistent by Yori Lavi, OpTier
- Software Performance Lifecycle at a Large National Bank by Amit Patel, Total Performance
For Monitoring and Performance Analysis, you have five half-day classes under the CMG-T umbrella - real classes starting from the beginning and giving what you actually need to know. From leading experts in the field: one generic class, one class per major platform (Windows, Unix/Linux, mainframe), and one for network.
- The Art and Science of Measurement by Mark B. Friedman, Microsoft
- Unix/Linux CMG Quick Start Course by Adrian Cockcroft, Netflix
- Windows System Performance Measurement and Analysis by Jeffry Schwartz, Integrated Services
- Introduction to z/OS Monitoring, Tuning, and the Workload Manager by Glenn R. Anderson, IBM
- Introduction to TCP/IP Performance Management by Nalini Elkins, Inside Products
Another class reviews what you need to know from statistics to be comfortable in the performance field:
- Statistics for Performance Analysis & Capacity Planning by Ray Wicks, IBM
Plus there are many presentation addressing specific environments or issues. A large variety of topics is covered from:
- Java Performance Analysis 401 by Peter Johnson, Unisys
- Windows Server 2008 Performance, Scalability and Tools by Bradley M. Waters, Microsoft
to very specific topics like
- Performance Optimization on Game Consoles by Bruce Dawson, Microsoft
Too much to list here see preliminary agenda http://www.cmg.org/cgi-bin/agenda_2007.pl for the full list.
Two other CMG-T half-day classes cover Modeling and Capacity Planning:
- Capacity Planning Boot Camp by Dr. Neil J. Gunther, Performance Dynamics
- Modeling and Forecasting by Dr Michael A. Salsburg, Unisys
Capacity Planning is a core area of CMG so there are many presentations investigating the subject in detail. Starting with the first part of the keynote presentation:
- Is Capacity Planning Still Relevant? by Jerred Ruble, TeamQuest Corporation
A caution here is that many modeling and capacity planning presentations are pretty advanced for an unprepared listener. While I strongly believe that a performance tester must understand the basic concepts of modeling and capacity planning (to be able to do "back of the envelope" calculations), serious mathematical modeling may be pretty complicated and is rather optional for performance testers.
Another CMG focus track is Software Performance Engineering (SPE). There is a great tutorial to start from:
- Software Performance Engineering: A Tutorial Introduction by Dr. Lloyd G. Williams, PerfX and Connie U. Smith, Performance Engineering Services
The following presentations are directly related to performance testing:
- How to Select Significant Workloads in Performance Models by Paolo Cremonesi, Politecnico di Milano
- Multiple Dimensions of Performance Requirements by Alexander Podelko, Oracle
- Performance Management Top 10 Traps by Megh Thakkar, CPT Global
A must-listen is the presentation about typical mistakes implementing performance projects (while it is about SPE, almost everything is true for performance testing too):
- Top 10 Ways to Kill an SPE Initiative by Dr. Connie U. Smith, Performance Engineering Services and Lloyd G. Williams, PerfX
One more great type of presentation involves overviews of different technologies for "performance" professionals: some insights that you need to know and probably won't find in marketing materials. There are so many new buzzwords that you can not go to training for each of them (and shouldn't - usually at least 95% of what is included in training is irrelevant to performance testing). One hour presentation discussing what this new thing may mean for you can give you a pretty good overview, good enough for what you are doing. Just to list a few examples from the CMG agenda:
- Internet 3.0: Performance and Productivity Issues of the Internet and a Proposal for the Next Generation by Prof. Raj Jain, Washington University in Saint Louis
- Going Green: Energy Efficiency Strategies for the Data Center by Tom Clark, Brocade
- Millicomputing: The Coolest Computers and the Flashiest Storage by Adrian Cockcroft, Netflix
-The Right Mix of Utility, Consolidation and Virtualization for Optimum Cost-Capacity-Performance by Adam Grummitt, Metron
See preliminary agenda http://www.cmg.org/cgi-bin/agenda_2007.pl for the full list of conference presentations.