SPONSORS:






User Tag List

Results 1 to 3 of 3
  1. #1
    Junior Member
    Join Date
    Mar 2001
    Posts
    7
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    DB_FetchNext () returns incorrect precision

    I have a setup something link this.
    [ ] REAL rTotalOnOrder
    [ ] HDATABASE hdbc
    [ ] HSQL hstmnt
    [ ] hdbc = DB_Connect (Connection)
    [ ] hstmnt = DB_ExecuteSql (hdbc, "SELECT sum(amount*exchange_rate) FROM po_line_item_budget_amount WHERE budget = '{sBudget}'")
    [ ] DB_FetchNext (hstmnt, rTotalOnOrder)

    OK, if I run this using the ODBC driver supplied with SQL 2000 by Microsoft, I get a value like 1116.56200
    However, running the same query in Query analyzer (by Microsoft) I get a value of 1116.52640

    it looks like SilkTest is only allowing a precision of 3 decimal places to come through. even though the database is set to a precision of .0001.

    Has anyone seen this before? and I am currious if others have determined if this is a SilkTest problem, or a limitation of the ODBC driver.




    ------------------
    Jason B. Archibald
    Jason B. Archibald

  2. #2
    Senior Member
    Join Date
    Jun 2000
    Location
    Chicago, IL USA
    Posts
    383
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: DB_FetchNext () returns incorrect precision

    Most likely this is the ODBC driver. With our Oracle database we have a choice of the Oracle and Microsoft ODBC drivers and we get significantly different data coversions with each.

    ODBC is a really nice idea, but unfortunately, it one of those poorly defined Microsoft "standards" that frequently seems to be a "gotcha" for any software that competes with Microsoft :-)

    Not a very solid footing for my testware :-(


    ------------------
    Rob Kapteyn
    Rob Kapteyn, CSTE
    RobKapteyn@gmail.com

  3. #3
    Senior Member
    Join Date
    Aug 1999
    Location
    Cambridge, UK
    Posts
    470
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: DB_FetchNext () returns incorrect precision

    If you're using a version of silktest before 5.5 SP1, then it could well be the problem with float precision (imprecision from the fifth or sixth significant figure onwards). I describe this problem more thoroughly (including the recommended workaround) at http://www.qaforums.com/Forum1/HTML/001739.html but the best way of solving the problem is to upgrade to 5.5 SP2.

    ------------------

 

 

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

vBulletin Optimisation provided by vB Optimise v2.6.0 Beta 4 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
User Alert System provided by Advanced User Tagging v3.0.9 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Questions / Answers Form provided by vBAnswers (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
vBNominatevBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Feedback Buttons provided by Advanced Post Thanks / Like (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Username Changing provided by Username Change (Free) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
BetaSoft Inc.
Digital Point modules: Sphinx-based search
All times are GMT -8. The time now is 12:27 AM.

Copyright BetaSoft Inc.