Thanks:  0
Likes:  0
Dislikes:  0

# Thread: calculating very large numbers

1. ## calculating very large numbers

In WR,
Calculating "2754269 / 2.715"
= 1014463.7200736649000000000000000

However Oracle calculates it as
= 1014463.7200736648250460405156538

the difference being the decimal places and the seemingly truncated numbers being "0" at the end.

This causes a mismatch.

I have used the following code:

</font><blockquote><font size="1" face="Verdana, Arial, Helvetica">code:</font><hr /><pre style="font-size:x-small; font-family: monospace;">

x = 2754269;
y = 2.715;
Oracle = sprintf(&quot;%.25f&quot;,1014463.7200736648250460 405156538); # Oracle shows in the watch list as 1014463.7200736648000000000000000

calc_x_y = sprintf(&quot;%.25f&quot;,x/y); #calculated_x_y returns (to 25 decimal places)1014463.7200736649000000000000000

if(calc_x_y == Oracle)
#1014463.7200736648 (oracle val)
#1014463.7200736649 (calculated val)

printf(&quot;success&quot;
else (&quot;fail&quot;</pre><hr /></blockquote><font size="2" face="Verdana, Arial, Helvetica">Why is 'sprintf' not getting more than 10 decimal places when the "%.25f" format is specified? And what can be done to get the true value from 'Oracle'?

Help!

TM

2. ## Re: calculating very large numbers

WinRunner is not the best tool for crunching numbers - especially to a precision of 25 decimal places. If you really need this kind of precision, I would reccomend writing a dll in C which will perform the calculation for you. Otherwise you can do something like this:

</font><blockquote><font size="1" face="Verdana, Arial, Helvetica">code:</font><hr /><pre style="font-size:x-small; font-family: monospace;">public function compare_floats(in float1, in float2,in precision)
{
#If precision is not specified, the lesser precision of the two floats will be used
auto prec1, prec2;
if (nargs() &lt; 3)
{
if ((prec1 = (length(substr(float1,index(float1,&quot;.&quot)) - 1)) &lt; (prec2 = (length(substr(float2,index(float2,&quot;.&quot)) - 1)))
precision= prec1;
else
precision = prec2;
}
return((sprintf(&quot;%.&quot; &amp; precision &amp; &quot;f&quot;,float1)) == (sprintf(&quot;%.&quot; &amp; precision &amp; &quot;f&quot;,float2)));
}

x = 2754269;
y = 2.715;
Oracle = 1014463.7200736648250460405156538;
if(compare_floats(x/y,Oracle))
pause(&quot;success&quot;
else
pause(&quot;fail&quot;</pre><hr /></blockquote><font size="2" face="Verdana, Arial, Helvetica">

3. ## Re: calculating very large numbers

Thanks for your response:

However - it doesn't work as desired - I wanted WR to do the x/y calculation and come up with the same as what oracle (& MS calculator) does but it's only using 2 dp in your code above and I need accuracy of at least 25.

Any ideas how to write a DLL in C?

Regards,
TM

4. ## Re: calculating very large numbers

ok have managed to solve this by utilising ODBC on oracle to send a simple select statement to calculate the x/y to infinitum:

</font><blockquote><font size="1" face="Verdana, Arial, Helvetica">code:</font><hr /><pre style="font-size:x-small; font-family: monospace;">
db_connect(&quot;Session&quot;,&quot;DRIVER={Micro soft ODBC for Oracle};UID=User;PWD=Password;SERVER=Server.WORLD; &quot;,30);

query = (&quot; select &quot;
&quot; to_char(&quot;x&quot;/&quot;y&quot calc&quot;
&quot; from any_schema.any_table&quot;

db_execute_query(&quot;Session&quot;,query,num_rec ords); # will return many records to best query on a table with only 1 record.
value_i_needed = db_get_field_value (&quot;Session&quot;,&quot;#0&quot;,&quot;#0&quo t # gives the value without rounding or any missing characters - is quite fast too !</pre><hr /></blockquote><font size="2" face="Verdana, Arial, Helvetica">Cheers Dudes,
TM

Nice!

6. ## Re: calculating very large numbers

Sure beats writting the formula to XL and reading the result back.

#### Posting Permissions

• You may not post new threads
• You may not post replies
• You may not post attachments
• You may not edit your posts
•
Search Engine Optimisation provided by DragonByte SEO v2.0.36 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Resources saved on this page: MySQL 10.34%
vBulletin Optimisation provided by vB Optimise v2.6.4 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
User Alert System provided by Advanced User Tagging v3.2.8 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
vBNominate (Lite) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Feedback Buttons provided by Advanced Post Thanks / Like (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Username Changing provided by Username Change (Free) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
BetaSoft Inc.
Digital Point modules: Sphinx-based search
All times are GMT -8. The time now is 05:46 PM.