[Glass] 4.85 asStringUsingFormat: #(1 1 false) = 4.8

[Martin McClure]

Yes, this kind of thing is inherent in using binary floating points.

Few decimal fractions can be exactly represented by a binary float. Ones 
ending in ".85" can never be exactly represented. Strings such as "4.85" 
and "20.85" are rounded to the nearest representable binary float. 
Depending on the integer part, the closest float can be either greater 
than or less than the ideal value.
So when further decimal rounding is done, sometimes down is the correct 
direction, sometimes up is. Converting the float to a fraction can show 
you what the exact value of a float is, and how it compares to the 
decimal string.

4.85 asStringUsingFormat: #(1 1 false) = 4.8
4.85 asFraction < (485/100) --> true

20.85 asStringUsingFormat: #(1 1 false) = 20.9
20.85 asFraction > (2085 / 100) --> true

1.85 => 1.9
1.85 asFraction > (185 / 100) --> true

5.85 => 5.8
5.85 asFraction < (585 / 100) --> true

Regards,
-Martin


On 3/4/21 12:32 PM, Marten Feldtmann via Glass wrote:
> 4.85 asStringUsingFormat: #(1 1 false) = 4.8
> 20.85 asStringUsingFormat: #(1 1 false) = 20.9
>
> 1.85 => 1.9
> 5.85 => 5.8
>
> So, actually this is the typical floating point problem ... :-(
>
> _______________________________________________
> Glass mailing list
> Glass at lists.gemtalksystems.com
> https://lists.gemtalksystems.com/mailman/listinfo/glass

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.gemtalksystems.com/mailman/private/glass/attachments/20210304/a20ce824/attachment.htm>