Debug FormatNumber(5.55, 1)
Debug StrD(5.55, 1)
returns 5.5
should 5.6
I understand, that StrD returns the wrong value, because the inaccuracy of floats. But I don't understand that for FormatNumber(). A formated float should be rounded correctly...
FormatNumber() rounds wrong
Re: FormatNumber() rounds wrong
Rounding is for the nearest integer part; not for the nearest floating point. And the docs don't say that FormatNumber() supports rounding at all.
Re: FormatNumber() rounds wrong
If there is a special function for rounding and displaying float numbers then I expect a correct output according to the rounding rules...
Re: FormatNumber() rounds wrong
5.55 do not exist as floating point number.
This number is 5.5499999999999998 and therefore it is round to 5.5.
Why FormatNumber() should behave different? It operates also with double precision.
This number is 5.5499999999999998 and therefore it is round to 5.5.
Why FormatNumber() should behave different? It operates also with double precision.
PB 6.01 ― Win 10, 21H2 ― Ryzen 9 3900X, 32 GB ― NVIDIA GeForce RTX 3080 ― Vivaldi 6.0 ― www.unionbytes.de
Lizard - Script language for symbolic calculations and more ― Typeface - Sprite-based font include/module
Lizard - Script language for symbolic calculations and more ― Typeface - Sprite-based font include/module
Re: FormatNumber() rounds wrong
You need a special round function. But you must not continue calculating with the value. So only for output.
Code: Select all
Procedure.d RoundEx(Value.d, NBDecimals)
Protected deci.d = Pow(10, NBDecimals)
ProcedureReturn Round(Value * deci, #PB_Round_Nearest) / deci
EndProcedure
d.d = 5.55
Debug FormatNumber(RoundEx(d, 1), 1)
Debug StrD(RoundEx(d, 1), 1)
My Projects ThreadToGUI / OOP-BaseClass / EventDesigner V3
PB v3.30 / v5.75 - OS Mac Mini OSX 10.xx - VM Window Pro / Linux Ubuntu
Downloads on my Webspace / OneDrive
PB v3.30 / v5.75 - OS Mac Mini OSX 10.xx - VM Window Pro / Linux Ubuntu
Downloads on my Webspace / OneDrive