Results 1 to 7 of 7
  1. #1

    High CPU Usage inspite of running for 2 months

    Hello All,

    I have a situation here. we had some module release 2 months ago. this module send emails to our customers according to a complex algorithm and series of DB checks. it was working fine for 2 months. but today we had a strange situation.

    Our application server CPU usage jumped to 100% and all the 12GB RAMs are used. Logs refer to this module that it was running.

    My question is. how shall a quality engineer plan a performance test to catch this type of issues. how can he think. a discussion about this will be very useful for me


  2. #2

    Re: High CPU Usage inspite of running for 2 months

    Your High CPU may be an artifact of the all 12GB of RAM being used and a secondary side effect of a memory leak in the algorthm in question. You need to determine the absolute cause of high CPU.

    High CPU can come as a result of System Activity or Application Activity. WIth all 12GB of RAM in use I would be examining carefully the split between system and application on the CPU. It is entirely possible that after your RAM was totally committed the system continued to allocate RAM which has resulted in high swap activity. As disk activity falls into the "you must handle this now" category for OS's, the high interrupt service from your disk/ram/swap activity may be pegging the CPU.

    This assumes you do not have an intelligent disk subsystem of SCSI, Fiber Channel or other intelligent variant which can offload the interrupt service for the disks to the intelligent host adapter.

    My working hypothesis would be that of a classical long term memory leak. Use standard long term test at a given load combined with monitoring on classical cross of system resources (CPU, DISK, NET, RAM) to watch for every increasing resource consumption associated with a "leak"
    James Pulley

    Replace ineffective offshore contracts, LoadRunnerByTheHour. Starting @ $19.95/hr USD.

    Put us to the test, skilled expertise is less expensive than you might imagine.

    Twitter: @LoadRunnerBTH @PerfBytes

  3. #3

    Re: High CPU Usage inspite of running for 2 months

    Runaway process perhaps?

    There is a lot of stuff this could be. Nail it down first.
    1. 9.5
    2. Web/HTTP (HTML mode)
    3. No
    5. None
    6. New
    7. Web 5800
    Network 1000
    DCOM 1000
    Oracle 1000
    Remote Access 1000
    8. Yes
    9. Windows Server 2003 SP1
    10. No

    What Everyone Should Read Before Asking For Help on the Internet

  4. #4

    Re: High CPU Usage inspite of running for 2 months

    We have a high level Raid systems controlled by a raid controller. also nothing else is running on the system except the application itself. and all indicators shows that the EJB layers is the only thing running and using the CPU.

  5. #5

    Re: High CPU Usage inspite of running for 2 months

    Hi ziohausam,

    I also feel that your CPU issue is related to high memory usage. Profiling tools like Rational Purify Plus will help you to find the memory leakage.


  6. #6

    Re: High CPU Usage inspite of running for 2 months

    If I were in this positionI would stop the app server to recover all the memory. I would then restart it and note the memory usage level.

    After this I would begin some soak tests and monitor the memory throughout. If it creeps up you have a leak.

    Run the soak tests for 12-24 hours initially and increase if you need to.

  7. #7

    Re: High CPU Usage inspite of running for 2 months

    When you say it 'jumped' do you mean you checked it two months ago and it was 25% and today is wat 105%? Or do you mean you have been monitoring or have a record of CPU usage over the last 2 months and it averages 40-50% but today it spiked to over 100?

    If it was a gradual memory leak, then it didn't 'spike'.

    As a tester, you should perform some sort of duration or stress test. Increase your volumes to a fixed level and allow it to run for an extended period of time. You should be monitoring system usage for the entire process. This is one way to identify a slow leak.

    Another (as mentioned above) is to have the developers run their code through something like Wily or JProbe to help catch potential memory handling issues.

    Is it possible your garbage collection processes failed? Is it customized or is it integrated into some platform package like BEA Weblogic?

    Do you have a quarterly process that runs or system maintenance that may have triggered at a bad time?

    How often to you bounce your system? If you reboot weekly, I would question the slow leak theory.

    Is your application making an external call? Are those systems available? How does your application handle timeouts?

    Are you cleaning up log files? Depending on how you handle archival of files, an aggregate of logs that are being archived but not deleted could attribute to the problem (unlikely, but I've seen a lot of odd architectures).



Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
BetaSoft Inc.
All times are GMT -8. The time now is 05:26 PM.

Copyright BetaSoft Inc.