bates@waubesa.stat.wisc.edu (Douglas Bates) (02/08/90)
I have discovered that my previous assumptions about the "precision"
part of the g format on NeXT's printf are wrong. On most systems the
precision specification in g format is taken to mean the desired
number of significant digits (with trailing zeros after the decimal
point removed). Apparently NeXT's printf sometimes interprets it as
the number of digits after the decimal point. Observe that
#include <stdio.h>
main()
{
double x = 1524.68;
printf("with %%.16g x = %.16g\n", x);
printf("with %%.15g x = %.15g\n", x);
printf("with %%.14g x = %.14g\n", x);
printf("with %%.13g x = %.13g\n", x);
printf("with %%.12g x = %.12g\n", x);
}
produces
with %.16g x = 1524.6800000000001
with %.15g x = 1524.6800000000001
with %.14g x = 1524.6800000000001
with %.13g x = 1524.6800000000001
with %.12g x = 1524.68
on a NeXT. I can come up with a possible explanation for the %.16g
result (althought it is a bit contorted since it involves 16 digits
after the decimal point plus 1 digit before the decimal point in e
format which is then converted to an f format) and I can explain the
%.13g and %.12g results (13 digits after the decimal point and 12
digits after the decimal point with trailing zeros removed) but I'll
be damned if I can explain the %.15g and %.14g results.
Can anyone give a definitive answer on what the precision
specification means to NeXT's printf?
--Douglas Bates, Statistics Dept., U. of Wisconsin-Madison