You are using /dump incorrectly based on your intent. In your second example, you're not calling the function and dumping the result -- you're just dumping the function. If you want to call it without any arguments, you still need to write the parentheses:
/dump IsSpellInRange()
If you were actually trying to just check if the function exists, the fact that your first test didn't generate an "attempt to call a nil value" error already told you it exists.
As for the API itself, according to
the documentation for IsSpellInRange you must pass a unit token, eg.
/dump IsSpellInRange(8092, "target")
... and "nil" is the expected result if the arguments are invalid (eg. you didn't pass a unit) or don't apply (eg. if you tried to check if a friendly target was in range for a hostile spell)