1. Invoke the browser from QTP using systemutil.run("iexplore.exe") & then try to spy.
2. Are you working on CITRIX or any other tunnels .If so then you would need to ensure that both QTP & the AUT instances are opened up on same server.
3. Look at this link http://www.joecolantonio.com/2010/08...t-recognition/
Checkout below points...
1. Launch QuickTest Professional before launching Application under Test.
2. Keep the Browser zoom setting to 100%.
3. Ensure ‘Web Add-In’ is enabled in QuickTest Professional.
4. Ensure ‘Protected Mode’ is OFF in your browser(for IE9). For other lower versions, check for the options like security modes.