Clojure Q&A - Recent questions in Transducers

The behavior of `filter` on infinite sequences in Transducers

Sun, 24 May 2026 05:38:39 +0000

Could you tell me where I can find information understanding the difference in behavior below?

user=> (clojure-version)
"1.12.5"
user=> (def iter1 (eduction (filter even?) (range)))
#'user/iter1
user=> (def iter2 (eduction (filter zero?) (range)))
#'user/iter2
user=> (take 1 iter1)
(0)
user=> (take 1 iter2)
;;; Pressed Ctrl-C due to no response

This behavior occurs when the input source is infinite, but I'm confused because I think filter is stateless. Of course, it works fine when Transducers aren't used.

user=> (take 1 (filter even? (range)))
(0)
user=> (take 1 (filter zero? (range)))
(0)

Optimize `eduction`

Fri, 03 Apr 2026 14:25:17 +0000

Problem statement

Eduction is useful part of the transducer ecosystem, which has a benefit/aim of improved performance. However, the function eduction only has a single varargs arity, which adds multiple function calls and and seq traversal allocations.

As seen in this PR to clj-kondo, manually passing in an xform directly can improve both overall performance and lower allocations.

Proposal

Adding arities to eduction can sidestep those allocations and seq traversals, improving baseline performance of all eduction invocations.

(defn eduction2
  {:arglists '([xform* coll])
   :added "1.7"}
  ([coll] (->Eduction identity coll))
  ([f1 coll] (->Eduction f1 coll))
  ([f1 f2 coll] (->Eduction (comp f1 f2) coll))
  ([f1 f2 f3 coll] (->Eduction (comp f1 f2 f3) coll))
  ([f1 f2 f3 f4 & args]
   (->Eduction (apply comp f1 f2 f3 f4 (butlast args)) (last args))))

There is currently no option for `partition` to use the advantage of transducers.

Tue, 22 Aug 2023 18:23:14 +0000

As said in the title. There are several options to solve this problem:

Introduce the 1-arity only for (partition n coll), which keeps the current standard pattern. It doesn't however allow the other arities and attempting to call (partition n step) or (partition n step coll) would result to unexpected behavior. That could be solved by checks of (seqable? step-or-coll) or similar things.
The 1-arity would be passed as a vector. That could solve the arity collision problem but itwould break the pattern.
Introduce a function like partition-xf that would do the work. The only issues could be that this function only returns a transducer, which is not common and that some people could expect the partition function to return a transducer if called like (partition n step), but I think it's just about getting used to it like everything else.

Could `run!` gain another arity to cover the common case of feeding into `eduction`?

Wed, 09 Aug 2023 14:03:15 +0000

run! is convenient to ensure that no intermediate sequence is created when applying functions over large collections. A common need is to couple it with eduction to apply
transformations on the input.

I was wondering if adding an arity to run! was something that could be considered to make this common case even more handy.

This would make (run! xf proc coll) a shorthand for (run! proc (eduction xf coll)).
As always if there's actual interest in this, I'm happy to supply a patch for it.

Transducer arity for vec

Fri, 28 Jul 2023 04:26:18 +0000

In light of Metosin's recent comments on the costs of laziness, what do you think about adding a transducer arity to vec? In essence it becomes an eager version sequence. Semantically I think (vec (map inc) coll) integrates fairly well with the idea of vec. That tiny bit of terseness over (into [] (map inc) coll) I think is worth having personally.

https://clojure-goes-fast.com/blog/clojures-deadly-sin/

calling take-nth with 0 causes an OutOfMemoryError, and kills the repl.

Wed, 08 Mar 2023 07:59:54 +0000

calling take-nth with 0 causes an OutOfMemoryError, and kills the repl. Perhaps a validation > 0 required here?

to reproduce,

user=> (take-nth 0 [1 2])
OutOfMemoryError Java heap space
        reply.eval-modes.nrepl/session-responses (nrepl.clj:51)
        reply.eval-modes.nrepl/session-responses (nrepl.clj:51)
        reply.eval-modes.nrepl/session-responses/fn--1340 (nrepl.clj:56)
        clojure.lang.LazySeq.sval (LazySeq.java:42)
        clojure.lang.LazySeq.seq (LazySeq.java:51)
        clojure.lang.RT.seq (RT.java:535)
        clojure.core/seq--5467 (core.clj:139)
        clojure.core/filter/fn--5962 (core.clj:2826)
        clojure.lang.LazySeq.sval (LazySeq.java:42)
        clojure.lang.LazySeq.seq (LazySeq.java:58)
        clojure.lang.RT.seq (RT.java:535)
        clojure.core/seq--5467 (core.clj:139)
Bye for now!
Error printing return value (OutOfMemoryError) at clojure.core/take-nth (core.clj:4289).

Why doesn't transduce call initializers?

Sat, 14 Jan 2023 16:32:46 +0000

I've seen this bug report before: https://clojure.atlassian.net/browse/CLJ-1569.
Now, while trying to implement Clojure-like transducers in another language for educational purposes, I've realized how many things are affected by this. So some thoughts first...
Performing initialization lazily with the current implementation in general case requires extra checks in the step function and in the completion function as well in case of empty input. So it's not like there are no workarounds, but this is very unintuitive, especially with all standard transducers still having nullary versions that are never called (so in terms of implementation it's skipping a step in the process, but in terms of logic it's adding an implicit step of dropping a transformed initial value and replacing it with an unrelated value).
Before any further examples, I should note that alt-transduce in the linked issue doesn't check if the initial value is already reduced after initialization process, which I believe it should do because reduce itself doesn't do that. So my updated alt-reduce is

(defn alt-transduce
  ([xform f coll]
     (let [rf (xform f)
           result (rf)]
       (rf (if (reduced? result)
             (unreduced result)
             (reduce rf (rf) coll)))))
  ([xform f init coll]
     (let [rf (xform
               (fn
                 ([] init)
                 ([result] (f result))
                 ([result input] (f result input))))
           result (rf)]
       (rf (if (reduced? result)
             (unreduced result)
             (reduce rf (rf) coll))))))

Speaking of affected laziness and extra checks, examples are already there in the standard library. take is defined as

(defn take
...
  ([n]
     (fn [rf]
       (let [nv (volatile! n)]
         (fn
           ([] (rf))
           ([result] (rf result))
           ([result input]
              (let [n @nv
                    nn (vswap! nv dec)
                    result (if (pos? n)
                             (rf result input)
                             result)]
                (if (not (pos? nn))
                  (ensure-reduced result)
                  result)))))))
...

First, take 0 still has to consume one item because it's the first time it has control. Second, even though I don't know why it's implemented in this somewhat complicated way, there's only one value to keep track of and yet two comparisons, apparently, just because it has to handle n = 0 when it receives its extra input. With a functioning initialization stage, it can be implemented like this:

    (defn alt-take
  ([n]
     (fn [rf]
       (let [nv (volatile! n)]
         (fn
           ([]
              (let [result (rf)]
                (if (not (pos? n))
                  (ensure-reduced result)
                  result)))
           ([result] (rf result))
           ([result input]
              (let [nn (vswap! nv dec)
                    result (rf result input)]
                (if (not (pos? nn))
                  (ensure-reduced result)
                  result))))))))

(Close to original, can be reordered if necessary, the point is that the two comparisons are in the places where they naturally should be.)
If this issue isn't fixed yet, there must be some strong reasons for that? It looks like this questioned never was answered, I guess my question is the same at this point: https://clojure.atlassian.net/browse/CLJ-1569?focusedCommentId=18296.

I’d appreciate any further insight you can offer on why this design choice has been taken.

group-by reducing function

Tue, 02 Aug 2022 11:55:38 +0000

I often find myself writing a reducing function that does what group-by does. It would be convenient to have group-by return a reducing function (without needing to import Christophe Grand's xforms lib). So instead of writing:

  (->> (range 10)
       (transduce (map inc) (completing (fn [r x] (update r (even? x) (fnil conj []) x))) {}))

I could write:

  (->> (range 10)
       (transduce (map inc) (group-by even?)))

Maybe something like this?

  (defn group-by
    ([f]
     (fn
       ([] (transient {}))
       ([r] (persistent! r))
       ([r x]
        (let [k (f x)]
          (assoc! r k (conj (get r k []) x))))))
    ([f coll]  
     (persistent!
      (reduce
       (fn [ret x]
         (let [k (f x)]
           (assoc! ret k (conj (get ret k []) x))))
       (transient {}) coll))))

Does not eduction work with reduce anymore?

Thu, 07 Oct 2021 03:03:37 +0000

I found that the eduction example in clojuredocs does not work with Clojure 1.10.3

user=> (reduce + (eduction (map inc) (range 3)))
Execution error (ClassCastException) at user/eval149 (REPL:1).
clojure.core.Eduction cannot be cast to clojure.lang.IReduce

eduction does not work with reduce function anymore? or is it kind of bug?

With a default value, it works again.

user=> (reduce + 0 (eduction (map inc) (range 3)))
6

What would transducers-first Clojure look like?

Sat, 02 Oct 2021 19:32:19 +0000

Rich says in the History of Clojure:

I think transducers are a fundamental primitive that decouples critical logic from list/sequence processing and construction, and if I had Clojure to do all over I would put them at the bottom.

This fascinates me, and I am really curious what would transducers-first Clojure look like? I don't really expect Rich to answer here (I should have asked this at his HOPL talk...) but perhaps someone who is closer to core might have some insight.

I would love to see a Clojure where transducers are the default and easiest way to solve things, where all the tutorials and gists and example code everywhere around would be using them and a context (vector, lazy, first etc.) was only chosen at the edges. Currently it is easier (but perhaps not simpler) to write lazy code than transducing code.

I am wondering if transducers-first Clojure would address the above or would it mostly be about the plumbing behind a clojure.core similar to the current one.

For my personal case, I actively have to fight laziness - I don't think I encountered a single place in our codebase where it would be beneficial but as it's still the easiest way of writing clojure I have to put in extra effort to not accidentally venture over to lazy land and to spot these patterns in PRs of others.

I would love a clojure.core2 where transducers are the default, it's easy to write transducing code, potentially with less typing/ceremonies, the likes of comp and into [] etc. And stepping over to lazy land is always an opt-in thing, like the impure parts of clojure - stm, transients etc. I'm not entirely sure how a core library like that would look but it's something I'd love to try.

Moreover, there are some transducer-specific optimisations like https://github.com/cgrand/xforms#on-key-value-pairs that could be considered. And perhaps there could also be a way to use the multiple-coll arities of map as part of a transducer somehow.

There could also be more baked-in support for an xform arg in functions that currently take a coll to preprocess them, like run! and str/join - if you actively use transducers you start to notice things like 'hey, I don't even need a collection there' and start to look for different transducing contexts - so instead of (count (into [] xform input)) you'd go (x/count xform input)

halt-when transducer throws class cast exception when used with into

Thu, 19 Aug 2021 12:22:08 +0000

The transducer halt-when throws an exception when used with into, but not with sequence.

(sequence (halt-when #{4}) (range)) ;; => (0 1 2 3)
(into [] (halt-when #{4}) (range))
;; class java.lang.Long cannot be cast to class clojure.lang.ITransientCollection (java.lang.Long is in module java.base of loader 'bootstrap'; clojure.lang.ITransientCollection is in unnamed module of loader 'app')

I think that is caused by a bug where it's returning input instead of result, and I think this implementation should fix that specific issue:

(defn halt-when1
  ([pred] (halt-when1 pred nil))
  ([pred retf]
   (fn [rf]
     (fn
       ([] (rf))
       ([result]
        (if (and (map? result) (contains? result ::halt))
          (::halt result)
          (rf result)))
       ([result input]
        (if (pred input)
          (reduced {::halt (if retf (retf (rf result) input) (rf result))})
          (rf result input)))))))

However, I think that the retf function is now coupled to the transducing context. So if you want to transform the result to, for example include the input that triggered the halt, then you need to know when you create the transducer that it will be used in a specific context and that you need to call conj, conj! or whatever. So I think that this signature of the function makes more sense:

(defn halt-when2
  ([pred] (halt-when2 pred nil))
  ([pred retf]
   (fn [rf]
     (fn
       ([] (rf))
       ([result]
        (if (and (map? result) (contains? result ::halt))
          (::halt result)
          (rf result)))
       ([result input]
        (if (pred input)
          (let [r (if retf
                    (retf rf result input)
                    result)]
            (reduced {::halt (rf r)}))
          (rf result input)))))))

which changes the signature of the retf function to accept the reducing function as well as the result and the input, and now you get to actually decide what to do:

(into [] (halt-when2 #{4} (fn [rf r i] (rf r i))) (range)) ;; => [0 1 2 3 4]

Could maybe add a third arity to halt-when with some options map to tell it if we'd passed in a 2 arity retf or a 3 arity one or something? Does that make sense? Or maybe I'm asking for a new thing called stop-at or something?

--- edit ---

Just found this https://groups.google.com/g/clojure/c/6HvmJIUsXKk/m/gLqUsfcnAwAJ which I think explains more the reasoning behind returning the input, but I still think it's useful to provide a strategy for transforming the result without needing to know what the actual reducing function is.

clojure.walk/walk can use transducers and protocols

Fri, 13 Nov 2020 19:00:00 +0000

Looking at clojure.walk/walk's implementation it looks like there's a good opportunity to improve its performance for vectors and maps by using a transducer for the coll? case:

   (coll? form) (outer (into (empty form) (map inner form))) ; old
   (coll? form) (outer (into (empty form) (map inner) form)) ; new

Another opportunity is replacing the cond dispatch with a protocol.

Also see this Jira: faster, more flexible dispatch for clojure.walk

See full implementation and benchmarks below:

Walk with transducer implementation

(defn walk*
  [inner outer form]
  (cond
    (list? form) (outer (apply list (map inner form)))
    (instance? clojure.lang.IMapEntry form)
    (outer (clojure.lang.MapEntry/create (inner (key form)) (inner (val form))))
    (seq? form) (outer (doall (map inner form)))
    (instance? clojure.lang.IRecord form)
    (outer (reduce (fn [r x] (conj r (inner x))) form form))
    (coll? form) (outer (into (empty form) (map inner) form))
    :else (outer form)))

(defn postwalk*
  [f form]
  (walk* (fn [form'] (postwalk* f form')) f form ))

Protocol implementation (with transducer):

(defprotocol IWalk
  (-walk [form inner outer]))

(extend-protocol IWalk

  clojure.lang.PersistentList
  (-walk [form inner outer]
    (outer (apply list (map inner form))))

  clojure.lang.PersistentQueue
  (-walk [form inner outer]
    (outer (apply list (map inner form))))

  clojure.lang.MapEntry
  (-walk [form inner outer]
    (outer (clojure.lang.MapEntry/create (inner (key form)) (inner (val form)))))

  clojure.lang.LazySeq
  (-walk [form inner outer]
    (outer (doall (map inner form))))

  clojure.lang.PersistentVector
  (-walk [form inner outer]
    (outer (into (empty form) (map inner) form)))

  clojure.lang.PersistentArrayMap
  (-walk [form inner outer]
    (outer (into (empty form) (map inner) form)))

  clojure.lang.PersistentHashMap
  (-walk [form inner outer]
    (outer (into (empty form) (map inner) form)))

  clojure.lang.PersistentHashSet
  (-walk [form inner outer]
    (outer (into (empty form) (map inner) form)))

  Object
  (-walk [form inner outer]
    (if (instance? clojure.lang.IRecord form)
      (outer (reduce (fn [r x] (conj r (inner x))) form form))
      (outer form)))

  nil
  (-walk [form inner outer]
    (outer form)))

(defn postwalk
  [f form]
  (-walk form (fn [form'] (postwalk f form')) f))

Benchmark results (with criterium)

(require '[criterium.core :as cc])

(def form
  '(1 2 [3 4 5] {:a 6 7 8} [9 [10]] #{:b 7}))


(do
  (cc/bench (postwalk identity form))
  (cc/bench (postwalk* identity form))
  (cc/bench (walk/postwalk identity form)))

;;; Evaluation count : 8413020 in 60 samples of 140217 calls.
;;;              Execution time mean : 7.287719 µs
;;;     Execution time std-deviation : 128.290658 ns
;;;    Execution time lower quantile : 7.119399 µs ( 2.5%)
;;;    Execution time upper quantile : 7.509465 µs (97.5%)
;;;                    Overhead used : 9.033571 ns
;;;
;;; Found 1 outliers in 60 samples (1.6667 %)
;;; 	low-severe	 1 (1.6667 %)
;;;  Variance from outliers : 6.2932 % Variance is slightly inflated by outliers
;;; Evaluation count : 7252680 in 60 samples of 120878 calls.
;;;              Execution time mean : 8.393008 µs
;;;     Execution time std-deviation : 140.292941 ns
;;;    Execution time lower quantile : 8.222419 µs ( 2.5%)
;;;    Execution time upper quantile : 8.724502 µs (97.5%)
;;;                    Overhead used : 9.033571 ns
;;;
;;; Found 3 outliers in 60 samples (5.0000 %)
;;; 	low-severe	 2 (3.3333 %)
;;; 	low-mild	 1 (1.6667 %)
;;;  Variance from outliers : 6.2524 % Variance is slightly inflated by outliers
;;; Evaluation count : 5888880 in 60 samples of 98148 calls.
;;;              Execution time mean : 10.259563 µs
;;;     Execution time std-deviation : 344.368716 ns
;;;    Execution time lower quantile : 10.017438 µs ( 2.5%)
;;;    Execution time upper quantile : 10.594850 µs (97.5%)
;;;                    Overhead used : 9.033571 ns
;;;
;;; Found 2 outliers in 60 samples (3.3333 %)
;;; 	low-severe	 1 (1.6667 %)
;;; 	low-mild	 1 (1.6667 %)
;;;  Variance from outliers : 20.5816 % Variance is moderately inflated by outliers

(def form
  '(defn walk*
     [inner outer form]
     (cond
       (list? form) (outer (apply list (map inner form)))
       (instance? clojure.lang.IMapEntry form)
       (outer (clojure.lang.MapEntry/create (inner (key form)) (inner (val form))))
       (seq? form) (outer (doall (map inner form)))
       (instance? clojure.lang.IRecord form)
       (outer (reduce (fn [r x] (conj r (inner x))) form form))
       (coll? form) (outer (into (empty form) (map inner) form))
       :else (outer form))))


(do
  (cc/bench (postwalk identity form))
  (cc/bench (postwalk* identity form))
  (cc/bench (walk/postwalk identity form)))

;;; Evaluation count : 1812840 in 60 samples of 30214 calls.
;;;              Execution time mean : 33.196956 µs
;;;     Execution time std-deviation : 961.919396 ns
;;;    Execution time lower quantile : 32.063979 µs ( 2.5%)
;;;    Execution time upper quantile : 34.546564 µs (97.5%)
;;;                    Overhead used : 9.033571 ns
;;;
;;; Found 6 outliers in 60 samples (10.0000 %)
;;; 	low-severe	 2 (3.3333 %)
;;; 	low-mild	 1 (1.6667 %)
;;; 	high-mild	 3 (5.0000 %)
;;;  Variance from outliers : 15.8051 % Variance is moderately inflated by outliers
;;; Evaluation count : 1653840 in 60 samples of 27564 calls.
;;;              Execution time mean : 36.626230 µs
;;;     Execution time std-deviation : 441.227719 ns
;;;    Execution time lower quantile : 35.798588 µs ( 2.5%)
;;;    Execution time upper quantile : 37.373995 µs (97.5%)
;;;                    Overhead used : 9.033571 ns
;;; Evaluation count : 1728600 in 60 samples of 28810 calls.
;;;              Execution time mean : 35.173883 µs
;;;     Execution time std-deviation : 400.776590 ns
;;;    Execution time lower quantile : 34.697017 µs ( 2.5%)
;;;    Execution time upper quantile : 35.825413 µs (97.5%)
;;;                    Overhead used : 9.033571 ns
;;;
;;; Found 1 outliers in 60 samples (1.6667 %)
;;; 	low-severe	 1 (1.6667 %)
;;;  Variance from outliers : 1.6389 % Variance is slightly inflated by outliers

(def form
  {1 {2 {3 {4 {5 {6 {7 {8 {9 {10 11}}}}}}}}}})


(do
  (cc/bench (postwalk identity form))
  (cc/bench (postwalk* identity form))
  (cc/bench (walk/postwalk identity form)))

;;; Evaluation count : 6947100 in 60 samples of 115785 calls.
;;;              Execution time mean : 8.809319 µs
;;;     Execution time std-deviation : 163.576702 ns
;;;    Execution time lower quantile : 8.627843 µs ( 2.5%)
;;;    Execution time upper quantile : 9.126265 µs (97.5%)
;;;                    Overhead used : 9.033571 ns
;;;
;;; Found 2 outliers in 60 samples (3.3333 %)
;;; 	low-severe	 2 (3.3333 %)
;;;  Variance from outliers : 7.8088 % Variance is slightly inflated by outliers
;;; Evaluation count : 6457380 in 60 samples of 107623 calls.
;;;              Execution time mean : 9.628546 µs
;;;     Execution time std-deviation : 171.963701 ns
;;;    Execution time lower quantile : 9.316393 µs ( 2.5%)
;;;    Execution time upper quantile : 9.976758 µs (97.5%)
;;;                    Overhead used : 9.033571 ns
;;;
;;; Found 5 outliers in 60 samples (8.3333 %)
;;; 	low-severe	 2 (3.3333 %)
;;; 	low-mild	 2 (3.3333 %)
;;; 	high-mild	 1 (1.6667 %)
;;;  Variance from outliers : 7.7664 % Variance is slightly inflated by outliers
;;; Evaluation count : 5483100 in 60 samples of 91385 calls.
;;;              Execution time mean : 11.064318 µs
;;;     Execution time std-deviation : 167.430489 ns
;;;    Execution time lower quantile : 10.854539 µs ( 2.5%)
;;;    Execution time upper quantile : 11.447064 µs (97.5%)
;;;                    Overhead used : 9.033571 ns
;;;
;;; Found 2 outliers in 60 samples (3.3333 %)
;;; 	low-severe	 1 (1.6667 %)
;;; 	low-mild	 1 (1.6667 %)
;;;  Variance from outliers : 1.6389 % Variance is slightly inflated by outliers

(def form
  [1 [2 [3 [4 [5 [6 [7 [8 [9 [10 11]]]]]]]]]])

(do
  (cc/bench (postwalk identity form))
  (cc/bench (postwalk* identity form))
  (cc/bench (walk/postwalk identity form)))

;;; Evaluation count : 7627620 in 60 samples of 127127 calls.
;;;              Execution time mean : 7.770194 µs
;;;     Execution time std-deviation : 81.222440 ns
;;;    Execution time lower quantile : 7.610275 µs ( 2.5%)
;;;    Execution time upper quantile : 7.913045 µs (97.5%)
;;;                    Overhead used : 9.033571 ns
;;; Evaluation count : 6941880 in 60 samples of 115698 calls.
;;;              Execution time mean : 8.726047 µs
;;;     Execution time std-deviation : 133.165422 ns
;;;    Execution time lower quantile : 8.557593 µs ( 2.5%)
;;;    Execution time upper quantile : 8.961663 µs (97.5%)
;;;                    Overhead used : 9.033571 ns
;;;
;;; Found 1 outliers in 60 samples (1.6667 %)
;;; 	low-severe	 1 (1.6667 %)
;;;  Variance from outliers : 1.6389 % Variance is slightly inflated by outliers
;;; Evaluation count : 5045520 in 60 samples of 84092 calls.
;;;              Execution time mean : 12.051122 µs
;;;     Execution time std-deviation : 223.757365 ns
;;;    Execution time lower quantile : 11.799274 µs ( 2.5%)
;;;    Execution time upper quantile : 12.768594 µs (97.5%)
;;;                    Overhead used : 9.033571 ns
;;;
;;; Found 3 outliers in 60 samples (5.0000 %)
;;; 	low-severe	 1 (1.6667 %)
;;; 	low-mild	 2 (3.3333 %)
;;;  Variance from outliers : 7.8088

Reducing to a collection

Fri, 25 Sep 2020 21:37:22 +0000

Hi guys. New to clojure. First post here. I'll try to keep it short. I can't figure out why this function doesn't work.

(def countToFive
  (reduce
    (fn [acc x] (into acc [(+ x (last acc))]))
    [1]
    [1 1 1 1]
    )
  )

As the name suggests its supposed to count to five.

Thanks in advance to anyone who can help explain.

Infinite loop with stateful transducer on core.async channel

Sat, 08 Aug 2020 13:03:55 +0000

I am trying to implement a simple stateful transducer that counts the number of items (with ClojureScript):

(defn stateful-counter []
  (fn [xf]
    (let [counter (atom 0)]

      (fn
        ([] (xf))

        ([result]
         (xf (xf result @counter)))

        ([result _]
          (swap! counter inc)
          result)))))

When running this on a sequence I get the output:

(into [] (stateful-counter) (range 5))

[5]

This is exactly what I expect.

When running this on a core.async channel I get an infinite sequence of 5s:

(go (println
     (<! (let [c (async/chan 1 (stateful-counter))]

           (async/onto-chan! c (range 5))

           (async/into []
                       (async/take 10 c))))))

[5 5 5 5 5 5 5 5 5 5]

If I don't use (async/take 10 _) there seems to be an infinite loop. The expected outcome would be [5].

I have also tried using net.cgrand.xforms/count from the xforms library and get the same unexpected result:

(go (println
     (<! (let [c (async/chan 1 xforms/count)]

           (async/onto-chan! c (range 5))

           (async/into []
                       (async/take 10 c))))))

I was assuming that I made an implementation error in stateful-counter that caused the loop. But am confused that xforms/count produces the same (unexpected) result.

This is a ClojureScript snippet, but I have been able to produce the same unexpected result with Clojure.

Can somebody help me understand why applying the stateful transducer on a core.async channel results in an infinite sequence?

EDIT: Same unexpected behavior seen with Clojure.

Stateful transducers, thread safety and performance

Mon, 17 Feb 2020 09:46:33 +0000

A quote from https://clojure.org/reference/transducers#_creating_transducible_processes:

A transducing process must encapsulate references to the function
returned by invoking a transducer - these may be stateful and unsafe
for use across threads.

Does it mean that it is OK to create unsafe transducers for better
performance?
Can core's stateful transducers be written unsafe
for having performance close to loop/recur?

Benchmarks for unsafe implementation of clojure.core/drop:

(definterface IMutable
  (get [])
  (set [new-val]))

(deftype UnsynchronizedMutable [^:unsynchronized-mutable n]
  IMutable
  (get [_] n)
  (set [_, nv] (set! n nv)))

(defn drop!
  ([n]
   (fn [rf]
     (let [nv (UnsynchronizedMutable. n)]
       (fn
         ([] (rf))
         ([result] (rf result))
         ([result input]
          (let [^long n (.get nv)]
            (.set nv (dec n))
            (if (pos? n)
              result
              (rf result input)))))))))

(comment
  (criterium.core/quick-bench
    (transduce (comp
                 (map inc)
                 (drop 1))
      + (range 1000)))
  #_"Execution time mean : 39,455570 µs"
  
  (criterium.core/quick-bench
    (transduce (comp
                 (map inc)
                 (drop! 1))
      + (range 1000)))
  #_"Execution time mean : 24,979885 µs")

Why does eduction take xforms rather than a single xform?

Sun, 13 Oct 2019 16:56:03 +0000

Eduction is unusual compared to transduce, sequence and a/chan in that it takes a list of xforms rather than a user-composed list. Is there a motivation for this, or an accident of history?

Transducers and Maps

Tue, 24 Sep 2019 20:14:24 +0000

Hi,

I'm learning more about transducers, but coming up with some deadends.

What I'm looking to understand, is whether they are appropriate to use (in my use-case, but also as a learning exercise) and whether what I am doing is right/efficient/maybe-there-is-a-better-way :-)

Let' say I have a data stucture thus:

 (def trip {:tripData 
            {:segments [{:dataPoints 
                         [{:location {:lat 1 :lng 2}} 
                          {:location {:lat 3 :lng 4}}]}]}})

There could be hundreds/thousands of dataPoints with each dataPoint having a single location.

I want to efficiently extract out only the lat and lng into a single collection and turn it into a string. I came up with this:

 (def xf
   (comp
    (mapcat :dataPoints)
    (map :location)
    (map (fn [{lat :lat lng :lng}] (str lat " " lng)))))

then evaluated via:

(def lat-lng (into [] xf (->> trip :tripData :segments)))

And I get back something like this:

["1 2" "3 4"]

Which I can then do (for the purposes of my exercise), this:

 (clojure.string/join ", " lat-lng)

to obtain, finally, this:

"1 2, 3 4"

Which is all fine and dandy :-)

However, given my inexperience with transducers, I'm left wondering if there is a different/better way. For example, a way, within the comp xf, to turn the data into the string and joined at the end instead of using clojure.string/join.

I also discovered, I can do this too, without the use of transducers:

 (def lat-lng-2 (->> trip
                     :tripData
                     :segments
                     (mapcat :dataPoints)
                     (map :location)
                     (map (fn [{lat :lat lng :lng}] (str lat " " lng)))))

Which, given the clojure.string/join, ends up with the same result.

However, it's my understanding that you can't use a map keyword (i.e., :tripData, :segments) as part of a comp as keywords are not transducers.

I'm at a loss on how to make this efficent/better whilst learning how I can use transducers.

I would appreciate some help/guidance/feedback!

Thank you.

What are other possible transducer contexts?

Tue, 30 Jul 2019 14:27:22 +0000

In Rich Hickey's talks Transducers (2014) he speaks about transducer contexts that exist "today" — collections and core.async, and what might be "tomorrow" (observables).

Now that 5 years has gone, I wonder, what are other transducer contexts that are created and used in the wild?

Avoid using keywords as sentinel values in transducers

Fri, 12 Jan 2018 12:21:15 +0000

The use of keywords as sentinels in transducers could in rare circumstances expose some applications to bugs and potential security risks.

(sequence (partition-by keyword) ["1" "none" "2" "clojure.core/none" "3" "4"]) ;(["1"] ["none"] ["2"] ["clojure.core/none" "3"] ["4"])

Ideally a private or local value that cannot be injected into the functions domain should be used instead, e.g. {{(Object.)}}.

partition-by and partition-all transducers should ensure visibility of state changes

Mon, 10 Apr 2017 03:28:43 +0000

The partition-by and partition-all transducers use state stored in an ArrayList. This state should be protected (for example, by volatile) to ensure visibility if used in a transducing process that moves computations across threads.

Provide a transducer for reductions

Thu, 17 Mar 2016 18:09:30 +0000

Reductions does not currently provide a transducer when called with a 1-arity.

Proposed:

A reductions transducer with explicit initialization values: reductions-with
Do to arity conflicts, this is a separate function, not combined with reductions

A second patch proposes a variant which allows explicit initialization values: reductions-with

(assert (= (sequence (reductions-with + 0) [1 2 3 4 5]) [1 3 6 10 15])))

Patch: 0003-add-reductions-with.patch

Prescreened by: Alex Miller

Support transducers in vec and set fns

Wed, 24 Feb 2016 21:42:51 +0000

Rather than
`(into [] (map inc) [1 2 3])`
{{vec}} (and {{set}}) could support the transducer directly:

(vec (map inc) [1 2 3]) (set (map inc) #{1 2 3})

Depending how far we wanted to take this, the implementation could be somewhat clever for vec in building the initial set of results in an array and then creating the vector with it directly as is already done in some other cases.

Transducer for partition-all with step

Sat, 28 Nov 2015 19:28:18 +0000

The docs for partition-all(link: 1) mention that when a coll is not provided, it returns a transducer. This is true for the form (partition-all n), but not true for (partition-all n step). There's no clear way that I can see to combine transducers from core to produce this sort of "sliding window" transducer, i.e.

user=> (into (link: ) (partition-all 2 1) (link: 1 2 3 4 5))
((1 2) (2 3) (3 4) (4 5) (5))

Of course, there's an arity collision between the above hypothesized (partition-all n step) and the concrete, non-transducer-producing (partition-all n coll), which could be resolved by switching on the type of the second argument, or less hackily by providing the functionality in a separate function, e.g. (sliding window-size step-size), or perhaps by some other means.

I implemented this function with reference to the existing (partition-all n) transducer here: https://gist.github.com/nornagon/03b85fbc22b3613087f6

Would it make sense to work on getting something like this into core?

(link: 1): http://clojuredocs.org/clojure.core/partition-all

group-by as reducer / reduction fn

Sun, 30 Aug 2015 02:05:41 +0000

Whilst working on a query engine heavily based on transducers, I noticed that it'd be great to have {{group-by}} able to be used as reduction fn. The attached patch adds a single arity version to {{group-by}} which enables this use case and also includes a few tests.

take transducer optimization

Tue, 25 Aug 2015 16:32:18 +0000

A basic refactoring to remove the let form and only requires a single counter check for each iteration, yields an 25% performance increase. With the patch, 2 checks are only required for the last iteration (in case counter arg was <= 0)...

;; master
(quick-bench (into [] (take 1000) (range 2000)))
WARNING: Final GC required 34.82584189073624 % of runtime
Evaluation count : 13050 in 6 samples of 2175 calls.
             Execution time mean : 46.921254 µs
    Execution time std-deviation : 1.904733 µs
   Execution time lower quantile : 45.124921 µs ( 2.5%)
   Execution time upper quantile : 49.427201 µs (97.5%)
                   Overhead used : 2.367243 ns

;; w/ patch
(quick-bench (into [] (take 1000) (range 2000)))
WARNING: Final GC required 34.74448252054369 % of runtime
Evaluation count : 18102 in 6 samples of 3017 calls.
             Execution time mean : 34.301193 µs
    Execution time std-deviation : 1.714105 µs
   Execution time lower quantile : 32.341349 µs ( 2.5%)
   Execution time upper quantile : 37.046851 µs (97.5%)
                   Overhead used : 2.367243 ns

Parallel transduce

Tue, 07 Oct 2014 18:58:07 +0000

Consider how to create a parallel path for transducers, similar to reducers fold.

Consider kv support for transducers (similar to reducers fold)

Tue, 07 Oct 2014 18:56:59 +0000

In reducers, fold over a map has special support for kv. Consider whether/how to add this for transducers.

Consider transducer support for primitives

Tue, 07 Oct 2014 18:55:19 +0000

Need to consider how we can support primitives for transducers. In particular it may be that IFn needs overloading for L/D in addition to O.

Add take-until

Fri, 20 Jun 2014 16:03:24 +0000

Discussion: https://groups.google.com/d/topic/clojure-dev/NaAuBz6SpkY/discussion

It comes up when I would otherwise use (take-while pred coll), but I need to include the first item for which (pred item) is false.

(take-while pos? [1 2 0 3]) => (1 2) (take-until zero? [1 2 0 3]) => (1 2 0)

Patch: clj-1451.patch

Includes transducer arity of take-until
Includes inclusion in transducer generative tests