So over the past week or so I've started to ponder what should be in place in order to have a successful team of people who can properly ascertain the quality of an app's security. A few things have come to mind but I welcome feedback in regards to where I might be on the right track and where I'm really off-track in my thinking + would love to hear cool and effective ideas and processes you've put in place over the years for security testing but realize one would never learn such a technique from a book. Some things I wrote as comments and others as questions......everything is fair-game though. Randomly pick things to comment on if you don't have the time to review everything.....by the end of the thread I'll probably end up with comments/issues/agreements to many of my questions and ideas.
1. Assign certain members of the team to stay abreast of security loopholes in the libraries that are compiled into your app and/or the loopholes contained in the apps that integrate with the app you are testing. Once we have a decent understanding of a vulnerability we'd then do a little research to understand just how serious this threat is. I suppose this could also be true of OS's and the interaction between your app and the OS to understand if a particular OS loophole would actually cause issues in your app.
2. Backdoors....do some of you have processes in place or code inspections that attempt to identify any backdoors that malicious developers wrote into the code? Or do you operate under the premise that well-known and effective security 'test types' so to speak will find these backdoors? The reason I ask is that it would seem that an incredibly inventive developer could easily code in a backdoor that wouldn't be detected by the current set of security tests in place. Or could potentially create a method to access an app which basically no one in the industry/test team/etc has ever come up with. In such a scenario does this simply end up in the Risk section for the project or have you and your team put in place a few things to attempt to locate such an issue? In this situation it'd seem to me you'd almost need to have a developer working in the QA team because so much knowledge about coding is needed.
3. Staying up-to-date with trends in the hacking community. Do you normally rely on companies who perform research on hacking trends to ensure you're knowledge is up to date? Or, for instance, do you actually have ties with hackers who day in/day out are successful? It would seem to me that if one could ever gain access to an underground website dedicated to the hacking community this could prove to be quite beneficial. Many of us have heard countless news stories about online predators and many police depts take this seriously enough they hire people to focus on this area of law enforcement. Have you ever heard of people taking the same approach in the software industry? i.e. Hire one or more people, create fake identities for them, ensure they can assume these identities to the point it's 2nd nature to them to role-play and attempt to learn vulnerabilities and hacking techniques during the period these techniques are, for instance, just starting to make their rounds throughout the hacking community. Or are you in the situation where although you'd like to have access to such info it's just not possible and thus you find keeping up to date with hacking techniques occurs after the technique has already been successfully utilized?
4. For apps where security is essential and you've got an app on your hands that contains an SDK do you make it a point to ensure developer documentation doesn't contain too many details about implementation? I would expect that extreme care must be taken to ensure what interfaces are exposed so the SDK documentation and exposed interfaces don't simply turn into the guidebook of choice for anyone wanting to illegally gain access to your app?
5. I take car and home security a bit to the extreme but one of the first things I learned many many years ago is how important it is to have 'layers' of security and not rely on one particular preventative measure. It'd take someone getting through 7 layers of security in roughly 20 minutes to actually steal my car. At around the 20 minute mark cops would arrive as a result of my security company alerting the police to an alarm. In the software world I'd expect this type of information is not generated randomly in an off-the-cuff manner such I just did with my analogy but rather is generated and analyzed throughout a project using tried and true mechanisms to obtaining accurate data?
6. Let's say I worked for Citibank as the VP of Development......the dev teams identifies an approach to security testing that will take 5 person years to develop and test costing US 1 million dollars. Another approach requires 10 person years and costs twice as much. If we go with the latter approach I'll need to propose the Development budget increases by roughly 400-500k because the capacity of the dev team is currently maxed out. By default I'll know though I'd better have some solid justification for why our budget needs to be increase to this level. It would seem that if I were ever in this situation the first thing I'd want to know is the cost that could be potentially associated with a breach in security and use this as a partial gauge as to what level of security is appropriate and the cost associated with an appropriate security strategy for one's app.
7. Is time spent to assign a priority to the particular breaches of security that may occur or put a different way assign a priority to the features that if breached would result in trivial, important or catastrophic consequences? The reason I ask such a question is I'd want to utilize a risk-based testing strategy for security testing just as we'd normally do for any other test type or functional area within the software. I suppose this could also be a data point for understanding an appropriate level of security at a feature or component level?
8. When assigning priority to security defects do you normally sit down and evaluate how many people do I think have the knowledge to exploit this hole once we release the software? Is this a tried-and-true universally known technique to gain illegal access to an app or something Johnny, my next door neighbor, just invented last night? Likewise, if we ship with this security issue how quickly do we think such information would be common knowledge within the hacking community and a potential exists that our current set of happy customers would quickly turn into a group of highly unhappy customers? The reason I ask this question is to understand if it's appropriate to have in place a process in which the QA team prioritizes all defects upon entering a defect. I won't go into all the reasons why we recently changed and started using this approach but I'm not so sure I'd feel comfortable having this process used for security defects simply because in many cases it seems so much knowledge about coding and implementation details are needed. These are some of the things that went through my mind as some things one would want to know in order to properly prioritize.....other suggestions or comments?
9. Excluding all the obvious and well-known measures used to determine if one is junior, mid-level or senior do any of you have potentially non-obvious mental security-specific checklists you use to determine if "You've got to be a really good hacker to find this security hole" or "This is so obvious anyone could identify and exploit this security hole". From your experiences have you ever noticed trends in the one's mindset and approaches to hacking as one progresses from junior to mid-level to a guru hacker? I'd probably assume the answer to this question may not differ too much than if I were to replace "Hacker" with "Developer" but I thought I'd ask it anyway just to see if there in some cases are differences based on your experiences.
I will mention my experience in security testing borders on nil. I'm about 3/4 the way through the book "How to Break Web Software" which has been a good introduction but so far hasn't struck me as being "comprehensive". Why am I doing all this? Something tells me in the next year or so I'll be in a position where proper assessments of security will be a key goal for myself, the team and various projects aside from the fact it's really annoying to realize I know nothing about a key area in the QA industry [img]/images/graemlins/smile.gif[/img]
Reserve a few months every so often and preview retirement throughout your career. You won't regret that a 35 year career was reduced to 34 years to take vacations measured in months in order to remember what a stress and care-free life is all about.
Books and hard work will get you anywhere you want to go.