OS testing question
Fairly new poster here, so forgive me if this has already been discussed.
In a meeting yesterday, I was challenged with why it is important to test a web app on different Operating Systems. The person understood testing across browsers and between Mac and PC, but questioned how many times I had found bugs on a Win98 machine that worked fine on Win2K. I couldn't recall a time (in my relatively short QA career) that I actually had found this to be a problem.
My response was that different OSs might have difficulties with certain modules that a developer might use to run the app. For instance if another app is running in the background, an old OS might not run it properly. Another option I considered is that testing multiple OSs may be a holdover from non-web based software apps.
This all came about because I'm preparing to test a new client internal app on Win 98, Win 2k and Win NT4.0 all on IE 5.5 and 6.
Looking forward to all your comments on this one.
No hugging or cat-napping while court is in session
- actual sign in traffic courtroom
Re: OS testing question
<BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>Originally posted by qamatteson:
My response was that different OSs might have difficulties with certain modules that a developer might use to run the app. For instance if another app is running in the background, an old OS might not run it properly. Another option I considered is that testing multiple OSs may be a holdover from non-web based software apps.<HR></BLOCKQUOTE>
Speaking to the statements here, to some degree it is a hold-over but only in the sense that it is a hold-over that can make sense. The problem is that people often waste a great deal of time testing configurations that do not need to be tested because they assume any operating system can cause different failure modes. On the other hand, it is possible to not test a configuration that you really should. This speaks to risk analysis and looking at potential failure modes, which means looking at the operational profile of the Web application in question. The operational profile should, to an extent, determine the possible failure modes and that should point the way to your operating system test configurations.
As an example of where the Windows variant can make a difference, some places I worked used DLLs in the URL that requested information from the operating system in a very proscribed manner. In this case, if care was not taken to handle the different ways that WinNT stored information from that of Win9x, we did get errors, and this was regardless of the browser. It solely had to do with the operating system but, even within that, it was because of poor coding.
In general, good design for Web applications is making them workable across all browsers and all operating systems. In practice, this is not always possible. For example, the same version of Internet Explorer on Macintosh and on Windows can display things differently because of layout issues. However, generally, this will not be the case (in terms of display issues) between Win9x and WinNT. However, sometimes if you have a Java application (like an applet within the browser), how this is served up (client-side) can be affected by the OS but that often has more to do with the Java Virtual Machine (JVM) that is present than it does with the strict operating system itself. Server-side logic, like servlets, has helped ease some of those problems because, of course, the code runs at the server and then normalizes the output that goes to the browser, thus operating systems make no difference.
So, to a certain, extent it depends on the nature of the "Web app" that you are talking about. At the very least, you should, if you have the resources, try something on one of the Win9x OS's (Win95, Win98, Win98SE, or WinME) and on one of the NT OS's (NT, 2000) and then maybe the hybrid OS's (like XP). However, do a failure mode analysis first to determine the potential differences between the [NT/2000, XP] scenario.
Now, let us say that you use ActiveX controls. Well, first off, right there you have a problem in that UNIX users cannot use ActiveX controls at all. In this case, the operating system matters. Macintosh, as another example, does support ActiveX but there have been noted problems between the two OS's - even with the same browser. The problems have been much less, however, between Windows variants because ActiveX uses the WinAPI to determine which operating system is being run on. However, if you consider this in relation to using a Netscape browser, Netscape does not support ActiveX at all and, as such, the operating system makes no difference - it will not work simply because Netscape does not support it.
It is possible to waste a great deal of time testing on operating systems that will show no difference at all. However, it is also possible to miss a full subset of your users and some glaring issues if you have potential failure modes that are dependent upon the operating system. This is part of risk analysis that you should do. As a sidelight, most operating system differences that will have a problem in the Windows variants come from Windows NT. In other words, you will find that Win9x and Win2000 cross-issues are much less than issues between one of those variants and Windows NT.
Just some thoughts. Hopefully some of that helps at least in seeing how I look at the issue.
Re: OS testing question
That depends on what you are really testing. For example, many individuals assume that all 'java' is running on local machine. This is not true. Servlets are often only generating html. So, does it really matter what browser you are testing?
It all depends.
Personally, I think the #1 problem in QA industry (in my humble, yet bloated opinion) is that people only think in application. In reality, isn't Microsoft or Netscape following a spec to say how http works? How do all these browser and web servers with various version and platforms work? They follow a spec. There is a well defined standard, and you can read it at: http://www.w3.org/Protocols/HTTP/1.0/spec.html.
So I follow a spec, what good does it do?
In theory, if you test against the protocol spec, and your software meets it, all others who meets the spec should work, right? In case of html stuff, you can say this is quite true. HTML standards are followed pretty well. So, why test through the 'browser'? Why not test against http? Too difficult? It really isn't. Getting a web page is as simple as open a socket, request, and poll. That's all!
...So, what happens if a well known browser has a bug:
You end up spending lots of time testing the 'browser' on you company's buck for Microsoft and Netscape. If your servlets are returning a valid http 1.0 code, you can assume 99.9% that IE, and Netscape will handle it. If not, it's a bug on their part, and they WILL fix it. Browser testing in my personal opinion is rather foolish for many cases.
Why does one work and not the other?
First of all, this is a historical issue. In the earlier years, http 1.0 spec was kicking around, lots of people didn't follow it. Buy hey, it's been around for almost a decade, and people have 'agreed' on it. So, some of the stuff that people say is different isn't really that different. The major difference is how browsers handle errors. For example, all headers are supposed to have a \r in it's lonesome to denote the end of a header. Often people forget these things. (I found this bug in our code). IE handles this error fine. Netscape chokes on it, once in a very while, depending on timing. That's the difference. We fixed the header problem, and the 'mysterious' problem disappeared...
So why shouldn't bother testing the browser?
People don't read http protocol. They see it through the brower's point of view. You do want to test the aspects of usability. But you can't automate these things easily. So, do these things manually. Let the test tool handle all the 'technical' details.
1. What about cookies and header?
Cookies are psudo headers and not part of http 1.0 spec but it's stabilized and is pretty well define now. Headers are very well defined in the specs.
2. Doesn't the browser OS pass any thing?
Yes, and it's called the agent type header. If your servlet is reading it, that's fine. I send bogus junk to see if servlet chokes, and our stuff ignores it. Chances are, your application will ignore it as well. You don't need a 'browser' to test this out. You can even hand massage a packet if you don't want to code.
3. What about making sure browser works?
Don't you think there's plent of people out there doing this every day at work? I'm pretty sure that Bill Gates needs all your company's help to make him more money. (He sure needs that extra Trillion bucks). You should focus on the the 'usability' aspects of things.
If you test against http 1.0 spec, you'd find how many bugs crawl out. Most applications do not follow the spec, and rely on the browsers to handle the errors. This is a bomb waiting to explode. Why not just follow the stupid spec? It's so easy, it's not even funny...
Also, running through html validation tool for tag will save you tons of time. Try running Bobby against your page sometime. See how many things it will tell you is wrong with the page...
Re: OS testing question
igglue, your comments make little sense based on the question that was asked. Maybe I'm misreading here? You said that "In theory, if you test against the protocol spec, and your software meets it, all others who meets the spec should work, right?" That doesn't even make sense. Following the protocol is just the transfer protocol. It has nothing to do with the actual format of your HTML, which could be a problem. Beyond that it says nothing at all about something like Jeff brought up about Java applets or ActiveX which is what might bring up operating system testing.
I think your forgetting that there are different specs based on what's being done. HTML is a spec and so is HTTP but one is a transfer protocol and the other isn't. To say you can test one and largely ignore the other is crazy unless I'm misreading you. So you can test against HTTP but you can also test your HTML for layout problems. Having bad layout problems due to, say, a style sheet is not going to be revealed by just testing the HTTP protocol. That's why I can't even fathom the simplicity of the testing you propose. You've basically limited the discussion to just servelets and that's the easiest thing in the world to consider because everything runs at the server. The question about operating systems also has to take into account other things that are passed from client to server and how they execute.
Re: OS testing question
Based on the developer's question, I was making a big assumption about the level of testing. Well, as a QA person, I'd say this was a big boo-boo. Assumption leads to many mitakes. But where would the fun be? In short, I'd have to agree with ya all that it's absolutely important that correct level of testing is applied. Thus the answer remains the same and that is "It depends".
As for confusion the description, I think it can be made even more confusing (or maybe asnwer something) by the following example.
Provided that Node A and Node B is designed to talk to each other using protocol Z. If Application A validates that they meet the Node A requirements for protocol Z, then Applications A should be able to talk to any application that meets the Node B requirements for protocol Z. Now this has one big assumption that developers interpret the protocol specs in a identical manner. HTTP has been around for some time that I think you can say that most things have been pounded out.
If you are just intersted in the content, and client side scripting needs to be evaluated, then you should test different browsers. Now, how far you take this is an open question. I'd tend to agree that one is testing the browser's compatability between different OS if that individual tests same version on multiple platforms. But that would depend on who your customers are and what your company's quality level is...
Mr. Owl, how many licks does it take to get to the center of the tootsie pop?... ...The world may never know...
Re: OS testing question
<BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>Originally posted by igglue:
Provided that Node A and Node B is designed to talk to each other using protocol Z. If Application A validates that they meet the Node A requirements for protocol Z, then Applications A should be able to talk to any application that meets the Node B requirements for protocol Z. Now this has one big assumption that developers interpret the protocol specs in a identical manner.<HR></BLOCKQUOTE>
But your HTML is separate from the protocol! It doesn't matter how your HTML developers interpret the spec because the protocol is simply utilized as a transport layer. Your HTML does not contain transport "logic" of any sort. By this same logic, you could just say that you really should test TCP/IP because that's what HTTP is going to be relying on. I mean I agree with you about the headers but that is up to the communicating clients and servers. That's usually the most minimal of testing. I agree if something screws up with the headers or the MIME, that's a problem but that's a problem with the browser. If the browser sends incorrect MIME types, looking at the protocol will tell you nothing that means you've got a browser bug. The only thing that I could see that would make sense to what you're talking about is RFC 1738 and RFC 1808 if you're doing something like changing the URL semantics. Other than that, we're talking about totally different levels of abstraction.
<BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>ASP/JSP/HTML
If you are just intersted in the content, and client side scripting needs to be evaluated, then you should test different browsers.<HR></BLOCKQUOTE>
Yeah, but ASP and JSP are not client-side scripting. Those are server-side scripting. So testing different browsers for that and testing different browsers for HTML (which is client-side) are totally different things. And even that's different than testing on different OS's because you're more going to want to consider OS configuration testing based on the nature of how a given client-side technology interacts with the OS, if it even does this. (Like Jeff said, Java applets, since the JVM resides on the client, and ActiveX, which can't even run on some OS's.)
Re: OS testing question
Protocol testing is a valid area but it is not one that you will often be doing unless (a) you are creating a protocol yourself, (b) you are changing the nature of how the protocol is used to communicate. If you, as just one example, are creating or using a media type, keep in mind that HTTP is meant to provide open and extensible data typing. Thus you would assume that if you define the media type correctly, you would be okay. Not quite true. First of all, the browser does have to recognize what to do with it. It does that by having been programmed to do so and it would only have been programmed to do so if your media-type values were registered with the Internet Assigned Number Authority.
The bigger thing also comes in when you utilize extensions to HTTP. For example, consider using SOAP headers in particular. Here you can extend HTTP for new header entries that are applicable to things like authentication, transaction management, etc. However, that really has nothing to do with the operating system because it relies on the client-implemention for recognition of those protocols. Now, if, for some reason, a given client on a given operating system could not recognize the protocol but that same client on a different operating system could recognize the protocol, you have an interesting issue where configuration testing (in terms of operating systems) is relevant to the protocol-level testing.
As another example, consider XML. The idea of the XMLP (XML Protocol) has been around since at least early 2001. The schema is the set of requirements that you can test against if you wanted to test the protocol itself. However, this mainly applies if you are writing a new application to utilize the schema. This is a separate question from asking if the browser supports it. For example, prior to Netscape 6.x, the Netscape browsers did not support XML rendering at all so, regardless of how well you implemented the schema and the protocol, and regardless of what operating system you were on, the content would not show up in a Netscape browser. But going back to the initial point with that, the XML protocol defines a set of requests and the format of the responses to each of these requests, specified through, in this case, an XML schema. So protocol testing, in this case, would mean testing conformance such that you are testing if your application (in given clients) responds to each of the protocol requests with a response that validates with what is stated in the schema.
So there are actually, as we see, three fundamental points to consider overall and while this has probably gotten away from what the original poster was looking for in some sense, this is all still valid in terms of considering all of your options when testing. I agree, however, that, in general and all things being equal, protocol testing is not something you are going to spend most of your time doing, particularly in terms of a strict transport protocol (like TCP or IP) or a strict application protocol (like HTTP).
Re: OS testing question
You are not provding enough information here to answer the question.
What is the App ?
WEB based ?
(Even web based apps seem to have significant client issues )
If you are referring to a simple HTML, CFM or ASP based web application then it should be reasonable to assume that testing should be OS independant (although a simple quick check test still should be done).
Also you may want to do research into the user base (examine IIS logs to see what types of browser are being used ) or stick in a trojan sniffer to gather OS data aswell ( I never suggested this BTW ) to determine the OS profile for risk analysis.