Warning about the (in)accuracy of ValF()

Just starting out? Need help? Post your questions and find answers here.
ozzie
Enthusiast
Enthusiast
Posts: 429
Joined: Sun Apr 06, 2008 12:54 pm
Location: Brisbane, Qld, Australia
Contact:

Warning about the (in)accuracy of ValF()

Post by ozzie »

The PB documentation for ValF() states that ValD() is more accurate than ValF(). I just found that out the hard way with a string that contained "84002390". See the following example:

Code: Select all

Debug Val("84002390")
Debug ValF("84002390")
Debug ValF("84002390.0")
Debug ValD("84002390")
Debug ValD("84002390.0")
This returns 84002392.0 for the two ValF() calls, but the correct value for the Val() and ValD() calls.

I'm now scanning my code for other uses of ValF(), to consider changing them to ValD().
User avatar
NicTheQuick
Addict
Addict
Posts: 1224
Joined: Sun Jun 22, 2003 7:43 pm
Location: Germany, Saarbrücken
Contact:

Re: Warning about the (in)accuracy of ValF()

Post by NicTheQuick »

Your number has a bit length of 27 and a normal float value has only a precision of 23 bits, see wikipedia.

Code: Select all

Debug "Bit length: " + Str(Len(Bin(84002390)))
The english grammar is freeware, you can use it freely - But it's not Open Source, i.e. you can not change it or publish it in altered way.
ozzie
Enthusiast
Enthusiast
Posts: 429
Joined: Sun Apr 06, 2008 12:54 pm
Location: Brisbane, Qld, Australia
Contact:

Re: Warning about the (in)accuracy of ValF()

Post by ozzie »

Thanks for the explanation. So ValF() is 'safe' for string values up to "8388607".
BarryG
Addict
Addict
Posts: 3292
Joined: Thu Apr 18, 2019 8:17 am

Re: Warning about the (in)accuracy of ValF()

Post by BarryG »

ozzie wrote: Fri May 20, 2022 8:55 amI'm now scanning my code for other uses of ValF(), to consider changing them to ValD().
I gave up on ValF() years ago and only use ValD() for accuracy reasons.
User avatar
STARGÅTE
Addict
Addict
Posts: 2067
Joined: Thu Jan 10, 2008 1:30 pm
Location: Germany, Glienicke
Contact:

Re: Warning about the (in)accuracy of ValF()

Post by STARGÅTE »

ozzie wrote: Fri May 20, 2022 9:52 am Thanks for the explanation. So ValF() is 'safe' for string values up to "8388607".
It has nothing to do with the magnitude of the number. It is the limit of precise digits (in the base-2 representation).
Anyway, you will have always the trouble when converting base-10 numbers (like your string) into base-2 format, even with doubles, like in this simple example:

Code: Select all

Debug ValD("4.6")
If numbers have to be exact, integers must be used.
PB 6.01 ― Win 10, 21H2 ― Ryzen 9 3900X, 32 GB ― NVIDIA GeForce RTX 3080 ― Vivaldi 6.0 ― www.unionbytes.de
Lizard - Script language for symbolic calculations and moreTypeface - Sprite-based font include/module
Post Reply