Ad server and testing
The company I'm with has implemented the Open AdStream Ad Server. Basically, I am attempting to come up with testing whether ad campaigns are being delivered as they are set up to be. The software offers reports and logs but the company doesn't know how trustworthy they are. Does it make sense to verify that the reports are accurately representing the number of impressions displayed and then use them as the proof that campaigns are functioning properly? How could I run isolated tests to test the reports?
Re: Ad server and testing
This seems like a really tough one. As you say, you could make a test page, then make random hits yourself from various machines you have access to and see if these are accurately reported. A manual approach probably wouldn't give you much confidence that the results would scale to real traffic. Beyond that though, there seems to be a generic issue with trying to treat this as a pure black box situation. I think I would try to get the vendor to provide a little information on what's under the covers of the reporting engine and see if that gives you the ability to craft more realistic tests. Alternatively, you might just try to find a discussion group somewhere that focuses on software of this type and perhaps leverage others' research.
Best of luck.