Use transducers on clojure.core lazy sequence transformations

Question

Hi,

I heard that if transducers were included before in the language, they would have been used as the building blocks of all sequence lazy operations. Since they were added later, you need to adapt your code a bit to use transducers. And I was wondering why using eduction is not an option? I'm trying to understand the scenarios in which it's not a performance advantage to define the sequence lazy operations like this (of course assuming the same behaviour):

(defn map
  ([f] ;; the standard transducer definition
   ,,,)
  ([f coll]
   (eduction (map f) coll))

I'm trying to understand given the assumption (which might be an erroneous assumption) that

(->> (range 5000000)
     (eduction (map inc))
     (eduction (filter odd?))
     (eduction (map dec))
     (eduction (filter even?))
     (eduction (map (fn [n] (+ 3 n))))
     (eduction (filter odd?))
     (eduction (map inc))
     (eduction (filter odd?))
     (eduction (map dec))
     (eduction (filter even?))
     (eduction (map (fn [n] (+ 3 n))))
     (eduction (filter odd?))
     (into []))

is the same as

(->> (range 5000000)
     (map inc)
     (filter odd?)
     (map dec)
     (filter even?)
     (map (fn [n] (+ 3 n)))
     (filter odd?)
     (map inc)
     (filter odd?)
     (map dec)
     (filter even?)
     (map (fn [n] (+ 3 n)))
     (filter odd?)
     (into []))

Answer 1

This example assumes using all of the result. eductions are not sequences, defer but don't cache when computed (from the docstring: "Note that these applications will be performed every time reduce/iterator is called"), and are not lazy, so that has a lot of ramifications (for concurrency and for how/when work is done and memory consumed).

If you only want to do as much of the work as needed, eductions or reduce are not great options as they do all the work. The sequence function (with transducers) can give you some aspects of incremental computation but is not as lazy in the face of expanding transducers (like mapcat) as sequences.

It's hard to come up with the set of design consequences that may have resulted if transducers had existed first. Perhaps Clojure would have been more eager and reduce-heavy with escapes out to lazy alternatives instead of the reverse.

Comments

> If you only want to do as much of the work as needed, eductions or reduce are not great options as they do all the work

I was under the impression that eductions only do as much work as needed (or rather, "requested" by the consumer of the eduction):

(let [ed (->> (range 5000000)
              (eduction (map #(doto % println)))
              (eduction (map #(doto % (-> (+ 100) println)))))]
  (println "taking")

  (into [] (take 3) ed)

  (println "taken"))
taking
0
100
1
101
2
102
taken
=> nil

Is there something obvious I'm missing here, Alex?

---

Clarification from Alex on Slack:

> I was really referring to reduce in that sentence
> it [eduction] is delayed
> it is often used in eager scenarios
> it's iterator based, but often used with reduce, which is eager

More details in the thread at https://clojurians.slack.com/archives/C03S1KBA2/p1681394119498109?thread_ts=1681383034.191669&cid=C03S1KBA2