I found that writing my own was the best choice. This was a few years back, and of course this was on unix, but if you're good at WSH (or whatever they call is now) it should be pretty easy. Especially if you can get some stubs or wrappers from development. If you want I'll see if I can dig up the code and post it. It was ksh/perl mixed with java. Basically it was just timing the execution of a search function by using remote execution and parallel processes. I just created a csv file with timing data and I used excel to do the graphs. I also used a DB and app server "monitoring" tool to see if I could find issues with these, or ways to speed these up.
The main reason I went this route was cost. The cheapest I could find a good one for my task was $40k. I wrote mine in 4 weeks and was paid 1/10 of that.
I just checked out Jmeter... wholly shnikes that has come a long way. I would suggest trying that or one of the others before trying to develop your own, especially if you're lacking in the scripting department.
"Imagination is more valuable than knowledge." (AE)