Thanks:  0
Likes:  0
Dislikes:  0

1. ## Re: trailing garbage

User Kerr, James (Kerr@lsec.lete.dnd.ca.nospam) posted:

as the song goes..."you are so beautiful to meeeeee, can't you
seeeeeee"...thanks bud

2. ## Re: trailing garbage

User Rick Wellinghoff (RWELLINGHOFF@remember.com.nospam) posted:

Just out of curiosity, where did the garbage come from in the first place?
I could understand if var3 equaled 0.100000 but how does it arrive at
0.100004? Formatting the solution makes it look correct, but the derived
solution of the equation is not correct.

3. ## Re: trailing garbage

User Steve Gray (sgray@BridgeMedical.com.nospam) posted:

Welcome to floating point math...

4. ## Re: trailing garbage

User Tim Gerrells (tim@intellocity.com.nospam) posted:

And it can be different depending on the type of CPU you have, the libraries
installed on the machine, the database engine you might access/store data
in.

Indeed: welcome to floating point math!

As long as we all agree to be as 'precise' as we want, together...

5. ## trailing garbage

User Kerr, James (Kerr@lsec.lete.dnd.ca.nospam) posted:

i need some help with the following Robot/Access issue:

a function in my Robot library, among other things, does the following:
- variables var1, var2, and var3 are declared as Singles
- variables str1 & str2 are declared as Strings
- our Access database is queried
- str1 is assigned "13.8" from a field in the db
- str2 is assigned "13.7" from a field in the db
- then the following code is performed in Robot:

var1 = CSng(Trim\$(str1))
var2 = CSng(Trim\$(str2))

var3 = (var1 - var2)

- you'd think that var3 would equal 0.1 but instead it equals 0.100004...i
just want 0.1 and not the trailing garbage...how do i make this

---

6. ## Re: trailing garbage

User Tom Ba (tombasay@hotmail.com.nospam) posted:

Don't want no stinkin' garbage?

var3 = format((var1 - var2),"0.0")
----Original Message Follows----
From: "Kerr, James" <Kerr@lsec.lete.dnd.ca>
Reply-To: "SQA Suite Team Test Users" <sqa-suite-users@lyris.dundee.net>
To: "SQA Suite Team Test Users" <sqa-suite-users@lyris.dundee.net>
Subject: trailing garbage
Date: Mon, 8 Jul 2002 15:35:12 -0400

i need some help with the following Robot/Access issue:

__________________________________________________ _______________
Chat with friends online, try MSN Messenger: http://messenger.msn.com

---

7. ## Re: trailing garbage

User Tom Ba (tombasay@hotmail.com.nospam) posted:

Tomba say
Floating point math
much better than submerged math;
personality suffer much
if spend too long in deep end.

John Nash Fan Club
----Original Message Follows----
From: Tim Gerrells <tim@intellocity.com>
Reply-To: "SQA Suite Team Test Users" <sqa-suite-users@lyris.dundee.net>
To: "SQA Suite Team Test Users" <sqa-suite-users@lyris.dundee.net>
Subject: RE: trailing garbage
Date: Mon, 8 Jul 2002 17:40:26 -0600

And it can be different depending on the type of CPU you have, the libraries
installed on the machine, the database engine you might access/store data
in.

Indeed: welcome to floating point math!

As long as we all agree to be as 'precise' as we want, together...

8. ## Re: trailing garbage

User Kevin Petry (72627.2200@compuserve.com.nospam) posted:

Tomba take out HP 4x6 pocket calculator and divide 1/3 then multiply by 3.
What happened. Floating bits lost to the ozone. Then discovered how to
spell shelloil using the LCD.

Math precision.
Here's whats happens. In the computer cpu we are trying to represent an
infinite set of real number representations (fractions or decimals) in a
finite natural number set (integer or counting locations of computer
memory). Since the first infinitite set is confined by the second finite
set overflow must occur and deviation occurs. The problem can be
illustrated in Cantor's diagonal.

Create an infinite array of floating point numbers. Evaluate a new
infinite floating value using cantor's diaganol. The real set is now
larger then the natural set.

Most compilers build in the default math precision. Java default is 6
places before expontial denotation.

---

#### Posting Permissions

• You may not post new threads
• You may not post replies
• You may not post attachments
• You may not edit your posts
•
Search Engine Optimisation provided by DragonByte SEO v2.0.36 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.