I am faced with a strange challenge. I work for a firm that writes softwares for a lot of pharmacutical companies. My job is to automate their testing. These companies follow very strict guidelines set by FDA to test their software. So their they (pharmacutical companies)send their auditor to see our testing practices.
To cut the story short, one question that constantly comes up is that how do we know that Silk Test works and it is testing the AUT proprely as they have mentioned 'you do sit there and watch it so how can you be sure?'
Any answer to this question would be highly appriciated.
For any test action performed there is an implied expectation of some result - a response received from the application being manipulated. Every test case is coded to setup a testable situation, initiate some action and then verify that an expected result occurred. If the expectation was met, then the actions required to obtain it must have occurred. If it didn't an error is logged.
So, where are the limits? Silk can be used quite effectively to verify the mechanical human interactive aspects of an application. It does not however (without added effort) directly know about or care about the appearance characteristics of the application being tested. For example: if the appearance is corrupted in some way but the objects being manipulated are unaffected, functional tests will still pass. Whether appearance is an issue for your application is up to you, but it seems unlikely. If you were testing a drawing application that requirement might be crucial.
Coming from a s/w background where total verification was paramount (avionics), I would guess that:
* the only way to really 'know for sure' would be for Segue to test the hell out of SilkTest and publish their results. This would probably slow down their progress to a crawl.
* (next best for you) develop automated tests, and then cross check them 100% manually. hopefully, you can convince then the fda that future runs of the same tests will produce equally valid results.
Chris R Mead
The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.
I encourage our coders to be exceedingly verbose in their _Print, _Print2 and _Print3 statements.
</font><blockquote><font size="1" face="Verdana, Arial, Helvetica">code:</font><hr /><pre style="font-size:x-small; font-family: monospace;">_Print("Entering this test/URL/Page."
_Print2("Performing this action, passing this value and looking for this result."
_Print3("Got this back."
_LogError("Failed to get expected result"</pre><hr /></blockquote><font size="2" face="Verdana, Arial, Helvetica">Then we have a boolean bVerbose in a related ini file. If this is set to true, you get enormahuge results files with literal walkthroughs of where the test is going, what it is looking for and what it found. If bVerbose is False, we see only the errors.
At some point, your own professionalism has to speak for itself and we take a certain amount of life on faith.
How do we know that the Pharm company got the right amounts of each ingredient in each pill?
How do we know that the machinery is accurate?
How do we know that the Pharmacist's license is valid?
How do we know that he/she didn't grab the wrong bottle off the shelf when filling our prescription?
How do we know that he could accurately read the physician's scribblings?
How do we know that MS Winders is driving Silk properly?
How do we know that there isn't another mathmatical calculation error in the Intel processor?
At some point, you have to look at the pile of evidence (your verbose results file) and say, "You know, it looks like KS's tests are just pretty thorough. We are comfortable with these results."