Why does doall return a lazy seq instead of its fully materialized list?

Question

Here is a commented sample function printing unexpected results.
The problems are:
Why does doall not provide me the actual data so that I can produce a string from them?
How can I get at the data hiding below the lazy seq if doall does not work?
Is this a bug in version 1.11.1 or am I totally misunderstanding doall?
Here's the demo code:

(defn lazytest []
(let [a [1 2 3 4] b [1 0 3 9] res (map #(if (= %1 %2) %1) a b)]

(println (str res))                

;; the .toString of this lazy seq is sth like clojure.lang.LazySeq@e8c3d

(println (str (doall res))) 

;; doall should materialize it but ALSO returns just the lzy seq!!

(println "seq is a " (class res))  

;; so this is a clojure.lang.LazySeq

(println "doall results in a " (class (doall res))) 

;; and this is ALSO a clojure.lang.LazySeq!!! WTF?

(println res)      ;; this prints (1 nil 3 nil) which I would 
;; expect also from the second println above!
(println "are res and (doall res) identical? " (identical? res (doall res)))

;; this prints true, which seems wrong/buggy...
) )

Answer 1

doall does work, by walking down the lazy seq and forcing each element to be materialized, then returning you the head. So, the class is not different, but side effects have occurred inside it to realize values. You now have realized data instead of a hanging unrealized pointer at the head.

The thing with str of a LazySeq is unfortunate, and there is another question here/jira with that issue. I suspect it did not print the LazySeq contents originally because LazySeqs can be infinite and printing those takes forever.

Comments

Thanks for the quick response - that was helpful. To summarize it:
.toString will (as expected) not trigger materialization and it is more intended for debugging purposes and not for text formatting (at least with lazy seqs).

My mistake was to expect doall to return a plain collection (since everything is materialized) but it just returns its argument. And, for consistency, .toString does not check that everything is materialized now but still exhibits the same behaviour of not looking inside.

If I apply str to the elements of the lazy seq, it will work:
(defn str-unwrap [items]
  (apply str (interpose " " items))
)
I added interpose to get a separator and ignore that str represents nil as empty string. In most cases this will not be a problem for the purpose at hand (in my application I am filtering away all nil elements anyway).

I could also produce a non-lazy collection manually via e.g.:
(defn unwrap [items]
  (reduce #(conj %1 %2) [] items)
)
I could use this in further processing and - as easiest way - call (str (unwrap items))

This works almost like (print items) except for the square brackets of the vector instead of parentheses shown by (println items) - not really a problem since all this is anyway subject to further processing... beyond the simple use of just str.

My mistake was to expect doall to do an "unwrap" of the lazy wrapper (since the whole wrapping stuff will - at that point in time - have no more use and could just be garbage collected when we want to keep just the collection.

Well, I can always roll my own unwrap-function - or maybe one exists already for that very reason? (would be nice)

Anyway, I think that newbies should be warned about the behaviour of str/.toString with lazy wrappers and about the fact, that doall does NOT unwrap anything. Note that the documentation says:
"...Walks through the successive nexts of the seq, retains the head and returns it..."
which sounds like we might get the head of a linked list with no "laziness" attached. Novices like me might take this literally and end up pretty confused and nervous...

---

That’s more or less true, but there really is no difference between a realized lazy sequence and an ordinary non-lazy list. Once it is realized, that is exactly what you get: the head of a linked list with no laziness left in it, which is the whole purpose of doall. But you get back the head of the same list you started with, for efficiency. It is just no longer lazy.

There would be no benefit to copying everything into new list nodes in doall, and a cost in terms of time, newly allocated structures, and garbage. If you really want to incur those costs, you can (for example) create a vector from the head of the lazy sequence. The unwrap example you give above can be replaced with just this: (into [] items) or even (vec items).

There is really no way around the fact that lazy evaluation confuses and confounds people who are not used to it. There are going to be far more weird surprises in your future until it becomes second nature.

Another thing to consider is that generally the only reason you use doall is if there are side effects that occur during the computation of the lazy elements of the sequence, and you need those side effects to happen now. Otherwise you don’t care when (if ever) the laziness is resolved. But there are many bugs that result from people putting, for example, network operations inside a (for …) and never doing anything with the lazy result that comes back, and then wondering why the network operations never took place. Eventually you learn that the right construct for code like that is (doseq …), not (for …).