System Call testing
Can some one throw some light on testing system calls in kernel level (not API testing).
Esentially, I am testing intrusion detection (IDS) application which intercepts system calls made by an application and blocks/allows based on the nature of the call.
I would like to know how to go about testing such an application. if I want to automate some of the testing how to go about doing it.
Re: System Call testing
Perhaps there are logs kept by the O/S that can tell you what your IDS should tell you. I would look for that and use that as a reference. (This of course assumes the O/S logger is bug-free!)
Process or task lists can be useful as well.
For the how to (I've never done this so don't take this as gospel), I would consider these items at a minimum:
1) Vary the execution permissions of the intruder(s) - for protected and unprotected memory.
2) Try to sneak through both the IDS and the O/S protection levels.
3) Try to attach to hardware and software interrupts.
4) Try to run the intruder at system startup before all layers of protection are provided.
5) Try to run as script, compiled, or assembled.
6) Try to overload a system resource such that it may compromise the stack - while you put a callback or return vector on the stack for your intruder.
Hope this provides some ideas....