| || |
Tips for Running SilkTest 24/7
Over the years, I have seen numerous messages asking for suggestions to improve SilkTestís ability to perform testing on a 24/7 basis. As part of my responsibilities while at Broderbund, I tested the image libraries that shipped with the PrintShop, PrintMaster and American Greetings products. This entailed opening and verifying approximately 15,000,000 images during a three month development cycle. In order to accomplish this, I ran three SilkTest host computers each controlling 10 agent machines running the AUT.
These machines needed to run 24/7 or as near to it as I could manage. I was able to achieve continuous testing approximately 95% of the time by following the guidelines I have outlined below.
First let me say that I donít believe it is possible to achieve continuous testing 100% of the time. This is due to a number of reasons. Pre-alpha, alpha, and even beta versions will contain memory leaks, debug code, and quick programming fixes that are going to cause testing crashes. And it isnít too difficult to prove that even final shipping versions will crash your testing if left to run long enough. Whether the problem is in your software, SilkTest, the operating system or a combination of all three (what I refer to as the Grand Alignment of Bugs), it will crash your tests.
By breaking the testing into sections and either 1) exiting and restarting the AUT after every section or 2) performing a restart of the computer running the AUT after each section, you can dramatically improve your testing success. I chose the restart as it is just as easy to perform; it ensures that all memory has been released and you are starting from a known condition. This is easy to accomplish when running the host and agent on different computers; it can be done on a single machine as well.
First, you will need to evaluate when you need to perform the restarts Ė every X number of test cases or as needed after certain troublesome test cases. This may require some trial and error. If you are using a test plan, create a test case that contains a script as described below and insert it into your plan at the desired locations. If not, create a function and call it as required.
This scheme requires that the Agent automatically starts when the computer reboots, (place a link to the Agent in the Startup folder) and if necessary, that the computer automatically logs into your network (I use TweakUI to do the log in). If you are on a single machine, your script will have to generate a batch file in the Startup folder that starts SilkTest and runs your script or test plan.
What you want to do is:
(S = Single Computer running both Host and Agent)
(H/A = Host and Agent on separate computers)
</font><ul type="square">[*]<font size="2" face="Verdana, Arial, Helvetica">(S & H/A) Drive the AUT to a condition where the computer can be restarted Ė close all files, log out of data servers, etc.</font>[*]<font size="2" face="Verdana, Arial, Helvetica">(S) Generate the batch file to restart SilkTest and run your script or plan file.
Note where you want to start testing after the restart Ė write this data to an .ini file that you read when your script starts
If you use plan files, do this instead:
- have your script make a copy of the plan file
- delete all completed test cases from the copy
- rename the original as a backup
- rename the copy to the original.</font>[*]<font size="2" face="Verdana, Arial, Helvetica">(S & H/A) Perform a restart from the Start Menu (or use one of the many available Restart executables to immediately reboot the computer).</font>[*]<font size="2" face="Verdana, Arial, Helvetica">(H/A) Disconnect from the Agent</font>[*]<font size="2" face="Verdana, Arial, Helvetica">(H/A) Sleep for 60 seconds</font>[*]<font size="2" face="Verdana, Arial, Helvetica">(H/A) When back, attempt to connect to the Agent Ė if it fails, sleep for 15 seconds and retry the connect</font>[/list]<font size="2" face="Verdana, Arial, Helvetica">If you have selected your restart points carefully, performing the above should improve your continuous running time to 95+%.
Since leaving Broderbund and starting my own consulting firm, I have been able to improve the performance to over 99% by using a USB controlled power strip that I have developed. This device turns off power to the computer running the AUT for a pre-determined time by making a single SYS_Execute call from SilkTest. By using inexpensive USB hubs, a single host can power cycle up to 256 agent computers.
I plan to market this product and expect to have a website available in January, 2005 with shipments to follow shortly thereafter. Feel free to contact me if you would like information before then.
This method is certainly not going to work for every testing situation, however you may be able to modify the concept to work for you. I hope you find these suggestions useful.