From a functional standpoint, testing a mainframe application is as easy as testing a web application. The only exception is that they tend to be clumsy with long time navigating through the hundreds of screens the old systems have (300+ is not unusual). You will really need to map processes out very clearly and I suggest you get a customer representative to help you with this.
For non-fucntional testing there are fancy tools out there that can record and playback network traffic and transactions when testing for perfomance, stress and volume.
Also I suggest you get a terminal emulator product that has scripting capabilities. see www.clientsoftware.com.au (no I don't work for them ....although they do buy me lunch ocassionaly). This can help navigating through the various screens.
Also you will need to understand the architecture of the application. What database do they use, what forms tool ar they using, what programming language.
The question is as vague as saying "How to I test NT?".
Are you testing the OS, Hardware, or application running on it? If this is testing hardware the ALU level, then I don't think you'll find good answers on this site since this is more of application level discussion.
If this is application testing, then what type of application is it? What type of server are you running on? What do you need to test?
Although if your simply performing GUI testing then it is little different from working with any other application other than the fact you these usually run inside a terminal emulator.
Often though with Mainframe testing you have to run batch jobs to perform 'batch' updates which are usually schedule to run nightly/weekly/monthly etc on prod systems. These are usually controlled by JCL (Job Control Language) which is broken down into a nbr of steps (much like shell script on UNIX) and submitted and monitored via TSS (cant remember what this stands for?).
This can be quite technically challenging e.g defining storage space for workfiles etc and often requires the aide of someone who has a lot of experience in this area.
More info required should you require any specific details.