FormatNumber() rounds wrong

Just starting out? Need help? Post your questions and find answers here.
Lebostein
Addict
Addict
Posts: 807
Joined: Fri Jun 11, 2004 7:07 am

FormatNumber() rounds wrong

Post by Lebostein »

Debug FormatNumber(5.55, 1)
Debug StrD(5.55, 1)

returns 5.5
should 5.6

I understand, that StrD returns the wrong value, because the inaccuracy of floats. But I don't understand that for FormatNumber(). A formated float should be rounded correctly...
BarryG
Addict
Addict
Posts: 3292
Joined: Thu Apr 18, 2019 8:17 am

Re: FormatNumber() rounds wrong

Post by BarryG »

Rounding is for the nearest integer part; not for the nearest floating point. And the docs don't say that FormatNumber() supports rounding at all.
Lebostein
Addict
Addict
Posts: 807
Joined: Fri Jun 11, 2004 7:07 am

Re: FormatNumber() rounds wrong

Post by Lebostein »

If there is a special function for rounding and displaying float numbers then I expect a correct output according to the rounding rules...
User avatar
STARGÅTE
Addict
Addict
Posts: 2067
Joined: Thu Jan 10, 2008 1:30 pm
Location: Germany, Glienicke
Contact:

Re: FormatNumber() rounds wrong

Post by STARGÅTE »

5.55 do not exist as floating point number.
This number is 5.5499999999999998 and therefore it is round to 5.5.
Why FormatNumber() should behave different? It operates also with double precision.
PB 6.01 ― Win 10, 21H2 ― Ryzen 9 3900X, 32 GB ― NVIDIA GeForce RTX 3080 ― Vivaldi 6.0 ― www.unionbytes.de
Lizard - Script language for symbolic calculations and moreTypeface - Sprite-based font include/module
User avatar
mk-soft
Always Here
Always Here
Posts: 5333
Joined: Fri May 12, 2006 6:51 pm
Location: Germany

Re: FormatNumber() rounds wrong

Post by mk-soft »

You need a special round function. But you must not continue calculating with the value. So only for output.

Code: Select all


Procedure.d RoundEx(Value.d, NBDecimals)
  Protected deci.d = Pow(10, NBDecimals)
  ProcedureReturn Round(Value * deci, #PB_Round_Nearest) / deci
EndProcedure

d.d = 5.55
Debug FormatNumber(RoundEx(d, 1), 1)
Debug StrD(RoundEx(d, 1), 1)
My Projects ThreadToGUI / OOP-BaseClass / EventDesigner V3
PB v3.30 / v5.75 - OS Mac Mini OSX 10.xx - VM Window Pro / Linux Ubuntu
Downloads on my Webspace / OneDrive
Post Reply