If you look in the Online documentation (Help > Books Online) access the BD.PDF book and go to Appendixes > The Java Environment. There are sections that may help you (JDK Versions, Java Plug-ins etc.)
I used a Java app recently that required recording using Multi protocol (Web/Winsock). Try that if you're still stuck. Also you may need to use url-based and web_custom_requests (recording options, url advanced)
Do you mean that it fails in the runtime viewer? If so, that does not necessarily mean the script itself is failing. Make sure you've enabled Java in the Runtime Viewer (in the first menu option in the RTV). If the problem persists, disable the RTV in the General Options. Use the execution log to determine if things are working.
Also, are you certain the applet communicates via HTTP? It may use RMI or tunneled HTTP with serialized objects, etc. It's best to ask a developer.