Does def hold onto the head?

Question

While randomly playing around with big fibonacci numbers, I stumbled across this apparent inconsistency between def and let.

user=> (def fibs (fn ([] (fibs 1N 1N)) ([a b] (lazy-seq (cons a (fibs b (+ a b)))))))
'user/fibs
user=> (take 20 (fibs))

(1N 1N 2N 3N 5N 8N 13N 21N 34N 55N 89N 144N 233N 377N 610N 987N 1597N 2584N 4181N 6765N)

user=> (def f1m (nth (fibs) 1000000))

OutOfMemoryError Java heap space  java.math.BigInteger.add (BigInteger.java:1315)

user=> (def f1m (let [x (nth (fibs) 1000000)] x))
'user/f1m
user=> (take 20 (str f1m))

(\3 \1 \6 \0 \4 \7 \6 \8 \7 \3 \8 \6 \6 \8 \9 \8 \7 \3 \4 \4)

user=> (count (str f1m))

208988

Why does the def blow memory, but the let does not? Does def hold onto the head of the lazy list in a way that let does not?

Answer 1

this is actually a difference in how eval handles "simple" expressions at the repl vs. "complex" expressions.

For expressions that appear to be simple enough (so like a def where the value is invoking some other function) the eval will actually act like an interpreter, and not emit jvm byte code and run it. The interpreter code path is what is holding on to the head here, it is a lot more naive then the generated byte code.

let expressions, fns, basically anything other than a literal value or function calls with simple arguments, are complex enough that they get compiled to bytecode which has more careful handling of arguments.

Comments

Is the behavior for "simple expressions" an optimization or is it necessary?

---

Hard to say, it is often characterized as an optimization. Clojure initially target java 5, which had significant costs related to creating many small classes (if you want to generate byte code and execute it, it must live in a class). Java these days has improved on that front to some degree. So it might be fine to just always generate bytecode.

The interpreter code actually looks like it is trying to avoid holding the head, but somehow it still is, there might be someway to fix that.

---

ah, the issue is nth is being inline expanded to a static method call instead of a function call, and the reflective static method call the interpreter does doesn't attempt to avoid holding the head.