SPONSORS:






User Tag List

Thanks Thanks:  0
Likes Likes:  0
Dislikes Dislikes:  0
Page 1 of 3 123 LastLast
Results 1 to 10 of 22
  1. #1
    Senior Member
    Join Date
    Apr 2003
    Location
    Wisconsin, USA
    Posts
    5,338
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Production build verification

    I have a situation that I will ask the experts here to comment on.

    We have two environments for our application. One is a development/staging/QA environment and the other is the production environment.

    Next week, we will be building our application to production. I was asked how I propose to test the application once it goes to production.

    Normally, if we had a separate staging environment, I would say to do a build to that environment and then do a round of functional testing in that environment to verify that the build process to a different environment worked. But we don't have that - only production. We don't want to do full boat testing in production, since that will push data, and there is an extensive database with historical data that goes with the application.

    About the best I can come up with is a form of smoke testing - make sure that we can navigate through the application and all the screens come up without error. Others have said we need to do more extensive testing - make sure that error messages fire where appropriate, since the theory is that data will not be pushed to the database if there is an error. Yet another opinion is that we do full boat testing in production, and then reload the data (again) after testing is done.

    What has been your experience in a situation like this?

  2. #2
    Senior Member
    Join Date
    Nov 2002
    Location
    London, UK
    Posts
    832
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Production build verification

    Darrel - are their time constraints in place for your testing activity on the day you release? In my current company we're lucky to have a Business Recovery site that we move the code to first and run a 3 hour high level test to ensure all the critical functionality is working and ensure all new changes are working, once that is completed we fail traffic over to the BR siet and move the code to production and follow the same test activity and once completed fall back to Production. In other companies similar to your situation we did not have a BR site, so we had a 12 hour prod down period that we communicated to clients and we used that time period to move the code run our entire automated suite and critical test cases manually. If we encountered issues they were either configuration issues where code was not moved properly, data issues or an actual bug which meant we tried to fix it quickly or eliminate the change from the releases.
    Life should NOT be a trip to the grave with the intention of arriving safely in an cool and well preserved body, but rather to skid in, chocolate in one hand, beer in the other, body wrecked, totally worn out and screaming WOO HOO what a ride!

  3. #3
    Moderator
    Join Date
    May 2001
    Location
    Michigan, USA
    Posts
    1,330
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Production build verification

    Here's what I've done, Darrel - (do I qualify to answer?)

    When possible - Full backups of each environment to start (duh). Wipe the staging/testing environment; Clone the production environment to the staging environment; Execute the install/roll-out process - including a quick sanity check of major components. If the wheels haven't fallen off, take another backup - then execute a more rigourous test cycle based on risk assessments you've already done. This should include checking conversion stuff (presuming data got manipulated) and at least one cycle of standard functions (depends on industry, no?) - We'll run an end-of-day cycle and an end-of-week cycle and check financials. Sometimes we'll tweak data to force an error to make sure that those routines are invoked correctly.

    Presuming there are no smoking hulks littering the lab, we declare it good to go.

    Twice I've had to do this in the real production environment. The process has been the same except at the end, after checking key processes, we restore to the check-point taken after the first swipe of sanity checking and before we tweaked data, ran 1-a-day/week processes and forced errors. We then spent the next several days watching intently to make sure we had not missed anything - and looking to isolate problems before users knew there were problems.

    The only time I've encountered a problem was when Development Manager types walked into the middle and "knew the right way to do things..." I now keep a candy jar at the door - they normally stop there and make it no further. [img]/images/graemlins/wink.gif[/img]

  4. #4
    Moderator
    Join Date
    Sep 2001
    Location
    Yankee Land
    Posts
    4,055
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Production build verification

    When I was doing live deploys, we had no staging, on multiple Web servers and App Servers we would take one of the Web servers out of the Load Balancer and use it for our tests. If we needed App or DB installs as well, we would schedule this for downtime, mostly weekend, and make sure we had a way to back out EVERYTHING. Then take a web head out, and do the deploys and test it, run a simple test that would cover the major functionality, then start the rest of the deploys. We tried to limit the data we entered, and rarely ever removed test data from the Acceptance Test we did; which we covered to all the major areas. After the deploy was done we would go back and do all the tests as time allowed, but our Acceptance Criteria was a subset of the Cases that allowed us to say we had minimal risk with the install.

    The only time this backfired, and when we took machines down for maintenance, was when we had to do massive data migrations. We would guesstimate how long the conversion would take, and this bit us once when the Cursor doing the conversion slowed the data conversion down because we never tested with the whole Prod database in QA; we were told we did not need to. After that we did, regardless of what anyone said after I spent 5 hours waiting for the data migration to finish.

    - M
    - M

    Nothing learns better than experience.

    "So as I struggle with this issue I am confronted with the reality that noting is perfect."
    - Unknown

    Now wasting blog space at QAForums Blogs - The Lookout

  5. #5
    Senior Member
    Join Date
    Apr 2003
    Location
    Wisconsin, USA
    Posts
    5,338
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Production build verification

    All good suggestions. However, they do not cover all the constraints I have to deal with. My apologies for not including them all in the first post.

    1. We have two environments only: a development/QA environment and a production environment.
    2. We cannot wipe the development/QA environment to use as a staging environment.
    3. The production build involves two pieces: building the actual web app to the production servers and moving a very large database to the production servers.
    4. The database will be moved only once and then data validation will be conducted in the same time window that we have to do build verification. The reason for a one-time only move is because this move involves gathering data from about 7 disparate sources and combining it into the master database. Approximate time: 6 hours.
    5. The build will take place on Saturday after Labor Day and we have the balance of Saturday after the build and Sunday to do our testing.
    6. We MUST be live on Monday. Backout or delay is not an option.
    7. We cannot do any testing that manipulates data, since this is production data and we have the constraint of #4 above.
    8. The problematical items that we really want to test all involve modifying data in the database.

    My answer so far has been that, given these constraints, about all I can do is navigate to each screen of the app and make sure that it doesn't give a code dump when we navigate to it. Someone else suggested that we go through the app and test everything that doesn't actually modify data (about 10-15% of the total functionality).

  6. #6
    Moderator
    Join Date
    Sep 2001
    Location
    Yankee Land
    Posts
    4,055
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Production build verification

    Without understanding what kind of data you have do you have the option of creating data that only QA can see and will be manipulated in the same way? If not is it possible to put up the data, load a copy into the database, do your tests, then reload the data since it is already moved to Production and wiping out your test changes?


    Your #7 and #8 seem to be contradictory, you cannot do tests that manipulate data but you want to test those. Its a big risk, if you can be confident of the set up in QA and sign off on the risk represented by your Production install constraints then you can only do what you can do. This is not a new problem, I've done similar deploys in the past, but we did have the option of setting up a QA Customer so we can manipulate its own data to do our verification. If you cannot do that, or do the copy option, then it looks as if you are in a Read Only mode and can do a scan through looking for code dumps or page errors. Considering the amount of functionality it leaves you to test its a big risk, which is why I suggest sign off so everyone knows and accepts it.

    - M
    - M

    Nothing learns better than experience.

    "So as I struggle with this issue I am confronted with the reality that noting is perfect."
    - Unknown

    Now wasting blog space at QAForums Blogs - The Lookout

  7. #7
    Moderator
    Join Date
    Mar 2004
    Location
    West Coast of the East Coast!
    Posts
    7,756
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Production build verification

    Darrel,
    We do the exact same thing, only we get the build on Friday and have the weekend to validate. Up until now we have been doing nothing in automation exept for a very large, very fast screen navigation test before the real testing begins. Now we have developed a test suite that runs an end to end test exercising as much functionality and data as possible to validate that the pieces of functionality fall into line properly. Other than that, there is not much you can do.
    Personal Comment

    Success is the ability to go from one failure to another with no loss of enthusiasm.
    ~ Winston Churchill ~


    ...Rich Wagner

  8. #8
    Senior Member
    Join Date
    Apr 2003
    Location
    Wisconsin, USA
    Posts
    5,338
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Production build verification

    No, we do not have the option of adding a QA user to the database. That was one suggestion I made which was shot down.

    And no, the database folks won't do a backup of the database and a restore. I suggested that too.

    Michael - you are absolutely correct - I have a conflicting set of requirments that I must work under. Makes it hard to find the "right" answer. I think Rich nailed it - there isn't much I can do to satisfy all the requirements.

  9. #9
    Moderator
    Join Date
    May 2001
    Location
    Michigan, USA
    Posts
    1,330
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Production build verification

    Ouch...

    OK - given those constraints - I think Rich has the right of it.

    Regarding point #6, what happens if the system implodes? Will the world grind to a halt? I don't mean to sound overly flippant, yet there are somethings that you may not be able to do anything about... Raid array crashes hard; squadron collapses; idiot operator hits the power switch on the wrong bloody machine and brings the entire server array down - hard. And my favorite... the idiot operator (actually their boss but guess who got blamed in a point-the-finger meeting Monday) hit the Halon discharge in the computer room... Been there and done that - seen too much to not laugh when I get told with a stern look that something "must* happen...

    I've worked on one project with similar constraints - and I made them promise me three successive weekends of dry-runs - back everything up, run thru the install process, check it, restore it, run production updates that have been sitting waiting to go. We needed every single one of them because there are some things that you can not fight forward through.

    Best of luck, old boy.

  10. #10
    Moderator
    Join Date
    Sep 2001
    Location
    Yankee Land
    Posts
    4,055
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Production build verification

    Well good luck with it, I'd work on the DB folks more to try and get a back up and restore so you can test properly but unless people are willing to change and provide less risk there is not much you can do.

    - M
    - M

    Nothing learns better than experience.

    "So as I struggle with this issue I am confronted with the reality that noting is perfect."
    - Unknown

    Now wasting blog space at QAForums Blogs - The Lookout

 

 
Page 1 of 3 123 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Search Engine Optimisation provided by DragonByte SEO v2.0.36 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Resources saved on this page: MySQL 9.38%
vBulletin Optimisation provided by vB Optimise v2.6.4 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
User Alert System provided by Advanced User Tagging v3.2.8 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
vBNominate (Lite) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Feedback Buttons provided by Advanced Post Thanks / Like (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Username Changing provided by Username Change (Free) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
BetaSoft Inc.
Digital Point modules: Sphinx-based search
All times are GMT -8. The time now is 04:19 PM.

Copyright BetaSoft Inc.