inconsistent behavior of lazy-seq w/ macro & closure on excptions

Question

lazy-seq seems to evaluate inconsistently when body includes a macro and throws and exception. 1st evalutation throws the exceptions, subsequent ones return empty sequence.

demo code:

`
(defn gen-lazy []
(let [coll [1 2 3]]

(lazy-seq
  (when-let [s (seq coll)]
    (throw (Exception.))))))

(def lazy (gen-lazy))

(try
(println "lazy:" lazy)
(catch Exception ex

(println ex)))

(try
(println "lazy, again:" lazy)
(catch Exception ex

(println ex)))

`

It should throw an exception both times but only does on 1st. Generally speaking an expression shouldn't evaluate to different things depending on whether it's been evaluated before.

When removing the closure ...

`
(defn gen-lazy []
(lazy-seq

(when-let [s (seq [1 2 3])]
  (throw (Exception.)))))

`

... or removing the when-let macro ...

`
(defn gen-lazy []
(let [coll [1 2 3]]

(lazy-seq
  (seq coll)
  (throw (Exception.)))))

`

It works i.e. consistently throws the exception, so seems to be some interaction between the closure and the macro at work here. This particular combination is used in the 'map' function.

See also: https://groups.google.com/forum/?fromgroups=#!topic/clojure/Z3EiBUQ7Inc

Answer 1

Comment made by: hank

N.B. The primary use case I have for this, in case it matters, is interrupting the evaluation of a 'map' expression in which the mapped fn is slow to evaluate (throwing an InterruptedException), because I am not interested in its result any more. Then later I re-evaluate it because I am interested in the result after all, however with above bug the lazy sequence terminates instead of recommencing where it left off.

(UPDATE: This use case is similar to the kind of ersatz continuations that Jetty does (RetryRequest runtime exception) or even Clojure itself when barging STM transactions with the RetryEx exception.)

Answer 2

Comment made by: hank

Related to CLJ-457 according to Christophe. His patch fixes this,too.

Answer 3

Comment made by: hank

Sorry Christophe's patch doesn't work for me here. It avoids evaluating the LazySeq a second time by prematurely throwing an exception. However a LazySeq might evaluate properly the second time around b/c the situation causing the exception was transient. As per comment above an evaluation might get interrupted, throwing InterruptedException the first time around but not the second time.

Also the observation with the closure and macro need explanation IMHO.

Answer 4

Comment made by: hank

further insight: 'delay' exhibits the same behavior and is a more simple case to examine. the macro suspicion is a red herring: as demoed below it is actually the closed over variable magically turns to nil, the when-let macro simply turned that into a nil for the whole expression.

`
(def delayed
(let [a true]

(delay
  (print "a=" a)
  (throw (Exception.)))))

(try
(print "delayed 1:")
(force delayed)
(catch Exception ex (println ex)))

(try
(print "delayed 2:")
(force delayed)
(catch Exception ex (println ex)))
`

prints:

delayed 1:a= true#
delayed 2:a= nil#

Answer 5

_Comment made by: hank_

The above leads to dodgy outcomes such as: The following expression leads to an Exception on 1st evaluation and to "w00t!" on subsequent ones.


(def delayed
  (let [a true]
    (delay
      (if a
        (throw (Exception.))
        "w00t!"))))


Try it like this:


(try
  (print "delayed 1:" )
  (println (force delayed))
  (catch Exception ex (println ex)))

(try
  (print "delayed 2:")
  (println (force delayed))
  (catch Exception ex (println ex)))


Results in:
delayed 1:#<Exception java.lang.Exception>
delayed 2:w00t!

This code shows that the problem is tied to the :once flag as suspected.


(def test-once
  (let [a true]
    (^{:once true} fn* foo []
        (println "a=" a)
        (throw (Exception.)))))


Invoking the fn twice will show 'a' turning from 'true' to 'nil', try it like this:


(try
  (print "test-once 1:")
  (test-once)
  (catch Exception ex (println ex)))

(try
  (print "test-once 2:")
  (test-once)
  (catch Exception ex (println ex)))


Results in:
test-once 1:a= true
#<Exception java.lang.Exception>
test-once 2:a= nil
#<Exception java.lang.Exception>

That doesn't happen when the ^{:once true} is removed. Now one could argue that above fn is invoked twice, which is exactly what one is not supposed to do when decorated with the :once flag, but I argue that an unsuccessful call doesn't count as invocation towards the :once flag. The delay and lazy-seq macros agree with me there as the resulting objects are not considered realized (as per realized? function) if the evaluation of the body throws an exception, and realization/evaluation of the body is therefore repeated on re-evaluation of the delay/lazy-seq.

Try this using (realized? delayed) after the first evaluation in the code above. In the implementation this can be seen e.g. [here for clojure.lang.Delay|https://github.com/clojure/clojure/blob/d0c380d9809fd242bec688c7134e900f0bbedcac/src/jvm/clojure/lang/Delay.java#L33] (similarly for LazySeq), the body-fn is set to null (meaning realized) after the invocation returns *successfully* only.

The :once flag affects [this part of the compiler only|https://github.com/clojure/clojure/blob/d0c380d9809fd242bec688c7134e900f0bbedcac/src/jvm/clojure/lang/Compiler.java#L4701]. Some field is set to nil there in the course of a function invocation, for the good reason of letting the garbage compiler collect objects, however this should to be done after the function successfully completes only. Can this be changed?

Answer 6

Comment made by: hank

A workaround for the case of the 'map' function as described in the 1st comment, works as this: The original map function, if you take out the cases for several colls, the performance enhancements for chunked seqs and forcing the coll argument to a seq, looks like this:

`
(defn map [f s]
(lazy-seq

(cons (f (first s)) (map f (rest s)))))

`

In my workaround I evaluate f twice:

`
(defn map [f s]
(lazy-seq

(f (first s))
(cons (f (first s)) (map f (rest s)))))

`

Because the downstream functions that are slow to evaluate are all of the deref kind that cache their result (more lazy-seqs, delays, futures, promises), the InterruptedException can only happen during the 1st evaluation, while the tail call optimization that sets closed-over variables to nil (it pays to read this here: http://clojure.org/lazy) only happens on the second call. The first still creates an fn that captures the head of the sequence 's', however this is not being held onto as it is not returned.

I use this special version of map (and other, similarly rewritten functions based on lazy-seq such as iterate) when I want interruptible, restartable seq evaluations.

Answer 7

Comment made by: hank

Instead of above hack, implementing map and all other combinators using a lazy Y-combinator, and removing the :once meta-data tag in the lazy-seq definition fixes things properly. Since the compiler sort-of-hack that clears closed-over variables that's triggered by the :once meta-data tag is basically only for cases of recursion that can be implemented using the Y-combinator, that won't be needed anymore either.

Answer 8

Comment made by: alexmiller

Re the Delay reference above, this is covered (and addressed) in CLJ-1175 which is waiting for a final screen.

Answer 9

Reference: https://clojure.atlassian.net/browse/CLJ-1119 (reported by alex+import)