I have seen the Shut Down dialog box(as you said Very very occasionally) when I use "SystemUtil.CloseProcessByName", but QTP never waits or hangs. The Shut Down dialog box is closed automatically and the test execution continues. I have seen this only when I execute my scripts using QTP AOM
I have also seen this behavior occasionally, but have not been able to identify any cause yet. I also use "SystemUtil.CloseProcessByName" but have not associated this behavior with use of this function.