lee@TIS.COM (Theodore Lee) (01/24/89)
Is anyone aware of a bug in the handling of negative integers,
either in TML Pascal or perhaps in the toolbox SANE routines?
To make a long story short, it appears that when negative
integers are converted to floating point representation, only
16 bits of precision are carried along. That's OK for positive
numbers, but if you set the lower 16 bits of a negative
number to zero you make an error of about 1/2**15. I can't tell easily
how the error arises, but it's easy to demonstrate.
Here's a fragment of a Pascal program ...
var one,i:longint; a:extended.
.
.
.
one := $40000000; {in fractional representation}
read(a);
i := trunc(a*one);
.
.
.
if the value -1 (or -1.0, or -1.00, etc.) is entered for a, the
value assigned (in binary) to i is
10111111111111110000000000000000
whereas if -1.0000000001 is entered you get
11000000000000000000000000000000
and if you enter -.9999999999 you get
11000000000000000000000000000001
an easy way to demonstrate the bug is the following fragment:
read(a);
writeln(trunc(a*one)/one:20:15)
almost any number except negative integer comes out right, e.g.,
.5, or -.5 are fine; -1 comes out -1.00006 or so (I forgot to write
down how many zeros, but its about that.)
Again, anyone know where the problem lies? I don't have a SANE manual.
(In case someone asks why, I was trying to speed up the inner loop in
a calculation by using the fractional representation math functions
in the Intmath toolset rather than the normal floating point ones; in
the course of it I wanted to see if the fractional representation is
actually sign-magnitude as some of the manuals say (it isn't) or the
usual 2's complement (it is). It was while experimenting with supplying
different values to the fracmul routine and finding anomalous results
that I traced it down to the above.)
Please reply directly since the site I normally read info-apple
on seesm to be temporarily cut off from the net.
lee@tis.com