as jason_m points you need a book, in fact many books have been written on the subject, and there are forums and conferences where hackers meet to discuss new attacks.
however a good starting point for the basics that you can put into practice fairly quickly is the book: How to break security software.
Agile Testers of the World UNIT!
You start with threat model: This should be a joint work between dev, PM, and QA.
Given the threat model, you work on penetration plan. This plan should address all possible ways one can exploit the weakness and possible mitigation plans.
Take all areas in penetration plan that is "high threat" without satisfactory mitigation plan, and file a BUG. Make sure you look at the weakest point of your system - as any hacker would - when you do this. It's always too easy to concentrate on most complex area of your system and neglect the weak link.