Welcome! Please see the About page for a little more info on how this works.

+1 vote
in Sequences by

While randomly playing around with big fibonacci numbers, I stumbled across this apparent inconsistency between def and let.

user=> (def fibs (fn ([] (fibs 1N 1N)) ([a b] (lazy-seq (cons a (fibs b (+ a b))))))) 'user/fibs user=> (take 20 (fibs))
(1N 1N 2N 3N 5N 8N 13N 21N 34N 55N 89N 144N 233N 377N 610N 987N 1597N 2584N 4181N 6765N)
user=> (def f1m (nth (fibs) 1000000))
OutOfMemoryError Java heap space java.math.BigInteger.add (BigInteger.java:1315)
user=> (def f1m (let [x (nth (fibs) 1000000)] x)) 'user/f1m user=> (take 20 (str f1m))
(\3 \1 \6 \0 \4 \7 \6 \8 \7 \3 \8 \6 \6 \8 \9 \8 \7 \3 \4 \4)
user=> (count (str f1m))

Why does the def blow memory, but the let does not? Does def hold onto the head of the lazy list in a way that let does not?

1 Answer

+2 votes

this is actually a difference in how eval handles "simple" expressions at the repl vs. "complex" expressions.

For expressions that appear to be simple enough (so like a def where the value is invoking some other function) the eval will actually act like an interpreter, and not emit jvm byte code and run it. The interpreter code path is what is holding on to the head here, it is a lot more naive then the generated byte code.

let expressions, fns, basically anything other than a literal value or function calls with simple arguments, are complex enough that they get compiled to bytecode which has more careful handling of arguments.

Is the behavior for "simple expressions" an optimization or is it necessary?
Hard to say, it is often characterized as an optimization. Clojure initially target java 5, which had significant costs related to creating many small classes (if you want to generate byte code and execute it, it must live in a class). Java these days has improved on that front to some degree. So it might be fine to just always generate bytecode.

The interpreter code actually looks like it is trying to avoid holding the head, but somehow it still is, there might be someway to fix that.
ah, the issue is nth is being inline expanded to a static method call instead of a function call, and the reflective static method call the interpreter does doesn't attempt to avoid holding the head.