Geeks With Blogs
John Workman The Workdog's weblog.... a brain dump, of sorts.

I've told this story in the past, but since I'm new to this blog, it's worth repeating.

There is an inherent flaw with IEEE's representation of decimals. This problem effects many languages, not just Visual Basic.  Most people are astonished when they learn of this.  First, I will show you the problem, then I will show you how to work around this problem, followed by an explanation for why this happens.

In VB, open the Immediate window. Type the following:

  ? 2.07 * 100
  207
  ? int (2.07 * 100)
  206
                        

Now type the following:

  ? int (2.07*100+.00001) 
  207 
                        

Often times I convert decimal numbers to "implied decimal" integers.  Now that I've learned of this issue, I always add .00001 before converting to an integer.

I know what your next question is. HOW and WHY does this happen???

The problem stems from computers using base 2 and number using base 10. Any number that is not a base 2 will have some rounding when you convert types. Your choices are to except some rounding errors or using 16 bit representation (which waste 6 bits).

Posted on Friday, May 13, 2005 12:28 AM | Back to top


Comments on this post: inherent flaw with IEEE's representation of decimals

No comments posted yet.
Your comment:
 (will show your gravatar)


Copyright © John Workman | Powered by: GeeksWithBlogs.net