Checking for whole numbers

Renoise Tool Development
1

I need to do some checking for a whole number in Lua, and knowing how floating points can be a pain to deal with I came up with the following:

function is_whole_number(n)  
 return (("%.8f"):format(n-math.floor(n)) == "0.00000000")   
end

The string.format() method does a wee bit of floating point correction of it’s own, which is why this seems like a good/simple/fast approach.
I’m basically assuming that it behaves similarly across any/all platforms - my only issue is that I haven’t seen this approach being used anywhere else :lol:/>

2

I normally use this:

  
  
function is_whole_number(n)  
 return (n%1 == 0)   
end

What is the advantage of your approach?

3

Well, your approach would work well with numbers that are specified as a whole number to begin with.
But this might not always be the case! Consider the following code:

local float = 64/11 -- define an 'unclean' number  
  
local num_is_whole = function(n)  
 --return (("%.8f"):format(n-math.floor(n)) == "0.00000000")   
 return (n%1 == 0)   
end  
  
local tmp = float  
while not num_is_whole(float) do  
 float = float + tmp  
 print("raise to whole number",float)  
end

If you run this in the lua console, you would expect to get something like this:

raise to whole number 11.636363636364  
raise to whole number 17.454545454545  
raise to whole number 23.272727272727  
raise to whole number 29.090909090909  
raise to whole number 34.909090909091  
raise to whole number 40.727272727273  
raise to whole number 46.545454545455  
raise to whole number 52.363636363636  
raise to whole number 58.181818181818  
raise to whole number 64

But, instead it continues all this way up to 1408 before it finds a ‘clean’ whole number!!
This is obviously due to the internal representation of the floating point…tricky stuff.

Try running the code, replacing your modulo with my string.format() and see the difference?

4

Ah I see, the way I do it is only accurate to 6 decimal places so you would have to round down, in which case your method is a neat solution!

And I thought computers were super accurate!

5

Well, at least computers are predictable - the same system would return the same minuscule differences each time.
It basically boils down to the fact that a floating point number is theoretically of infinite precision, something computers simply can’t express.

I think this solution is working fine, I did test it on a couple of different systems.