Clojure Q&A - Recent questions tagged transducers

Dedupe doesn't mention statefullness in docstring

Mon, 22 Jan 2024 12:22:44 +0000

Unlike all other core functions which can return a stateful transducer, dedupe doesn't mention this in its docstring. E.g. distinct, drop-while etc. all say:

Returns a stateful transducer when no collection is provided.

But dedupe just says:

Returns a transducer when no collection is provided.

Missing stateful string accumulator function (for transduce target)

Fri, 15 Sep 2023 19:04:56 +0000

If you have a transducer chain that makes a stream of strings, you might want to accumulate them at the end into a string. Existing functions like str will generate a lot of overhead as there is no stateful accumulator you can maintain.

What you really want is a StringBuilder that accumulates, and emits the string at finalization.

String joining could then be something like:

(transduce (interpose ",") str! coll)

extensible with more transducers as needed.

cgrand's xforms lib has something like this: https://github.com/cgrand/xforms/blob/2079b74271b858b6a91dcb87bc58f3b93ea0b19c/src/net/cgrand/xforms/rfs.cljc#L145-L147

Use transducer arity of `into` in `clojure.walk` namespace

Wed, 13 Sep 2023 22:49:05 +0000

It's simple. Just like that:

> clojure.walk/walk:
(outer (reduce (fn [r x] (conj r (inner x))) form form))
=> (outer (into form (map inner) form))
(outer (into (empty form) (map inner form)))
=> (outer (into (empty form) (map inner) form))

> clojure.walk/keywordize-keys:
(postwalk (fn [x] (if (map? x) (into {} (map f x)) x)) m)
=> (postwalk (fn [x] (if (map? x) (into {} (map f) x) x)) m)

> clojure.walk/strigify-keys:
(postwalk (fn [x] (if (map? x) (into {} (map f x)) x)) m)
=> (postwalk (fn [x] (if (map? x) (into {} (map f) x) x)) m)

Efficient merge

Sat, 26 Aug 2023 18:28:54 +0000

I was wondering why the merge function doesn't use transients and if there could be a better version of that.

Instead of this:
(defn merge [& maps] (when (some identity maps) (reduce #(conj (or %1 {}) %2) maps)))

we can use transducers to keep the code still easy to read and have better performance:
(defn merge [& maps] (when (some identity maps) (into {} cat maps)))

There is currently no option for `partition` to use the advantage of transducers.

Tue, 22 Aug 2023 18:23:14 +0000

As said in the title. There are several options to solve this problem:

Introduce the 1-arity only for (partition n coll), which keeps the current standard pattern. It doesn't however allow the other arities and attempting to call (partition n step) or (partition n step coll) would result to unexpected behavior. That could be solved by checks of (seqable? step-or-coll) or similar things.
The 1-arity would be passed as a vector. That could solve the arity collision problem but itwould break the pattern.
Introduce a function like partition-xf that would do the work. The only issues could be that this function only returns a transducer, which is not common and that some people could expect the partition function to return a transducer if called like (partition n step), but I think it's just about getting used to it like everything else.

Use transducers on clojure.core lazy sequence transformations

Tue, 11 Apr 2023 06:55:00 +0000

Hi,

I heard that if transducers were included before in the language, they would have been used as the building blocks of all sequence lazy operations. Since they were added later, you need to adapt your code a bit to use transducers. And I was wondering why using eduction is not an option? I'm trying to understand the scenarios in which it's not a performance advantage to define the sequence lazy operations like this (of course assuming the same behaviour):

(defn map
  ([f] ;; the standard transducer definition
   ,,,)
  ([f coll]
   (eduction (map f) coll))

I'm trying to understand given the assumption (which might be an erroneous assumption) that

(->> (range 5000000)
     (eduction (map inc))
     (eduction (filter odd?))
     (eduction (map dec))
     (eduction (filter even?))
     (eduction (map (fn [n] (+ 3 n))))
     (eduction (filter odd?))
     (eduction (map inc))
     (eduction (filter odd?))
     (eduction (map dec))
     (eduction (filter even?))
     (eduction (map (fn [n] (+ 3 n))))
     (eduction (filter odd?))
     (into []))

is the same as

(->> (range 5000000)
     (map inc)
     (filter odd?)
     (map dec)
     (filter even?)
     (map (fn [n] (+ 3 n)))
     (filter odd?)
     (map inc)
     (filter odd?)
     (map dec)
     (filter even?)
     (map (fn [n] (+ 3 n)))
     (filter odd?)
     (into []))

group-by reducing function

Tue, 02 Aug 2022 11:55:38 +0000

I often find myself writing a reducing function that does what group-by does. It would be convenient to have group-by return a reducing function (without needing to import Christophe Grand's xforms lib). So instead of writing:

  (->> (range 10)
       (transduce (map inc) (completing (fn [r x] (update r (even? x) (fnil conj []) x))) {}))

I could write:

  (->> (range 10)
       (transduce (map inc) (group-by even?)))

Maybe something like this?

  (defn group-by
    ([f]
     (fn
       ([] (transient {}))
       ([r] (persistent! r))
       ([r x]
        (let [k (f x)]
          (assoc! r k (conj (get r k []) x))))))
    ([f coll]  
     (persistent!
      (reduce
       (fn [ret x]
         (let [k (f x)]
           (assoc! ret k (conj (get ret k []) x))))
       (transient {}) coll))))

Cast to Map$Entry Exception When Trying to (Into {}) After Sequence Operations

Sun, 08 May 2022 09:17:12 +0000

Is it intended behavior to get this exception :

class clojure.lang.Keyword cannot be cast to class java.util.Map$Entry

on this piece of code :

(def a [["Active" :Transit-RLC "Active"]])

(->> a
     (transduce
       (comp 
             (map drop-last)
             (map reverse))
       conj [])
     (into {}))

changing the transducer to (map (-> [(second %) (first %)]) solves the issue.
Thanks

dedupe and partition-by depending on magic value

Thu, 28 Apr 2022 14:57:11 +0000

I noticed dedupe and the transducer-returning arity of partition-by use the magic value :clojure.core/none to indicate the start of processing. There are some cases where this results in an incorrect result:

;;;; dedupe

(dedupe
 [:clojure.core/none 1 2 3])
;;=> (1 2 3)
;; expected: (:clojure.core/none 1 2 3)

(dedupe
 [:clojure.core/none :clojure.core/none 1 2 3])
;;=> (1 2 3)
;; expected: (:clojure.core/none 1 2 3)

;; transducing arity is also affected
(sequence
 (dedupe)
 [:clojure.core/none 1 2 3])
;;=> (1 2 3)
;; expected: (:clojure.core/none 1 2 3)

;;;; partition-by

(sequence (partition-by
           {0 0
            1 :clojure.core/none
            2 2})
          (range 3))
;;=> ([0] [1 2])
;; expected: ([0] [1] [2])

;; non-trasducing arity works OK:
(partition-by
 {0 0
  1 :clojure.core/none
  2 2}
 (range 3))
;;=> ((0) (1) (2))

Is this a conscious tradeoff for performance or is it something that might use a bit of additional thinking about better handling?

What would transducers-first Clojure look like?

Sat, 02 Oct 2021 19:32:19 +0000

Rich says in the History of Clojure:

I think transducers are a fundamental primitive that decouples critical logic from list/sequence processing and construction, and if I had Clojure to do all over I would put them at the bottom.

This fascinates me, and I am really curious what would transducers-first Clojure look like? I don't really expect Rich to answer here (I should have asked this at his HOPL talk...) but perhaps someone who is closer to core might have some insight.

I would love to see a Clojure where transducers are the default and easiest way to solve things, where all the tutorials and gists and example code everywhere around would be using them and a context (vector, lazy, first etc.) was only chosen at the edges. Currently it is easier (but perhaps not simpler) to write lazy code than transducing code.

I am wondering if transducers-first Clojure would address the above or would it mostly be about the plumbing behind a clojure.core similar to the current one.

For my personal case, I actively have to fight laziness - I don't think I encountered a single place in our codebase where it would be beneficial but as it's still the easiest way of writing clojure I have to put in extra effort to not accidentally venture over to lazy land and to spot these patterns in PRs of others.

I would love a clojure.core2 where transducers are the default, it's easy to write transducing code, potentially with less typing/ceremonies, the likes of comp and into [] etc. And stepping over to lazy land is always an opt-in thing, like the impure parts of clojure - stm, transients etc. I'm not entirely sure how a core library like that would look but it's something I'd love to try.

Moreover, there are some transducer-specific optimisations like https://github.com/cgrand/xforms#on-key-value-pairs that could be considered. And perhaps there could also be a way to use the multiple-coll arities of map as part of a transducer somehow.

There could also be more baked-in support for an xform arg in functions that currently take a coll to preprocess them, like run! and str/join - if you actively use transducers you start to notice things like 'hey, I don't even need a collection there' and start to look for different transducing contexts - so instead of (count (into [] xform input)) you'd go (x/count xform input)

halt-when transducer throws class cast exception when used with into

Thu, 19 Aug 2021 12:22:08 +0000

The transducer halt-when throws an exception when used with into, but not with sequence.

(sequence (halt-when #{4}) (range)) ;; => (0 1 2 3)
(into [] (halt-when #{4}) (range))
;; class java.lang.Long cannot be cast to class clojure.lang.ITransientCollection (java.lang.Long is in module java.base of loader 'bootstrap'; clojure.lang.ITransientCollection is in unnamed module of loader 'app')

I think that is caused by a bug where it's returning input instead of result, and I think this implementation should fix that specific issue:

(defn halt-when1
  ([pred] (halt-when1 pred nil))
  ([pred retf]
   (fn [rf]
     (fn
       ([] (rf))
       ([result]
        (if (and (map? result) (contains? result ::halt))
          (::halt result)
          (rf result)))
       ([result input]
        (if (pred input)
          (reduced {::halt (if retf (retf (rf result) input) (rf result))})
          (rf result input)))))))

However, I think that the retf function is now coupled to the transducing context. So if you want to transform the result to, for example include the input that triggered the halt, then you need to know when you create the transducer that it will be used in a specific context and that you need to call conj, conj! or whatever. So I think that this signature of the function makes more sense:

(defn halt-when2
  ([pred] (halt-when2 pred nil))
  ([pred retf]
   (fn [rf]
     (fn
       ([] (rf))
       ([result]
        (if (and (map? result) (contains? result ::halt))
          (::halt result)
          (rf result)))
       ([result input]
        (if (pred input)
          (let [r (if retf
                    (retf rf result input)
                    result)]
            (reduced {::halt (rf r)}))
          (rf result input)))))))

which changes the signature of the retf function to accept the reducing function as well as the result and the input, and now you get to actually decide what to do:

(into [] (halt-when2 #{4} (fn [rf r i] (rf r i))) (range)) ;; => [0 1 2 3 4]

Could maybe add a third arity to halt-when with some options map to tell it if we'd passed in a 2 arity retf or a 3 arity one or something? Does that make sense? Or maybe I'm asking for a new thing called stop-at or something?

--- edit ---

Just found this https://groups.google.com/g/clojure/c/6HvmJIUsXKk/m/gLqUsfcnAwAJ which I think explains more the reasoning behind returning the input, but I still think it's useful to provide a strategy for transforming the result without needing to know what the actual reducing function is.

"concat" two xforms into one?

Tue, 26 Jan 2021 21:57:14 +0000

(let [coll [{:a 1 :b 2}
            {:a 3 :b 4}
            {:a 7 :b 5}
            {:a 1 :b 4}]
      xf1 (comp (map :a) (filter #{1 3}))
      xf2 (comp (map :b) (filter #{2 4}))]
  (concat
    (into [] xf1 coll)
    (into [] xf2 coll)))

is there a way to combine the two xforms above (xf1 and xf2) into a single xform somehow so the collection coll will be traversed only once?

Why partition-by and partition-all have a transducer arity but partition does not?

Thu, 19 Nov 2020 19:40:31 +0000

I couldn't find anything related in the Jira.

Infinite loop with stateful transducer on core.async channel

Sat, 08 Aug 2020 13:03:55 +0000

I am trying to implement a simple stateful transducer that counts the number of items (with ClojureScript):

(defn stateful-counter []
  (fn [xf]
    (let [counter (atom 0)]

      (fn
        ([] (xf))

        ([result]
         (xf (xf result @counter)))

        ([result _]
          (swap! counter inc)
          result)))))

When running this on a sequence I get the output:

(into [] (stateful-counter) (range 5))

[5]

This is exactly what I expect.

When running this on a core.async channel I get an infinite sequence of 5s:

(go (println
     (<! (let [c (async/chan 1 (stateful-counter))]

           (async/onto-chan! c (range 5))

           (async/into []
                       (async/take 10 c))))))

[5 5 5 5 5 5 5 5 5 5]

If I don't use (async/take 10 _) there seems to be an infinite loop. The expected outcome would be [5].

I have also tried using net.cgrand.xforms/count from the xforms library and get the same unexpected result:

(go (println
     (<! (let [c (async/chan 1 xforms/count)]

           (async/onto-chan! c (range 5))

           (async/into []
                       (async/take 10 c))))))

I was assuming that I made an implementation error in stateful-counter that caused the loop. But am confused that xforms/count produces the same (unexpected) result.

This is a ClojureScript snippet, but I have been able to produce the same unexpected result with Clojure.

Can somebody help me understand why applying the stateful transducer on a core.async channel results in an infinite sequence?

EDIT: Same unexpected behavior seen with Clojure.

How do sequences cache?

Tue, 24 Mar 2020 17:55:31 +0000

I have read that Clojure sequences "cache" their values as they are realized (here and here for example). This often seems to come up when discussing the relative merits of using transducers for transforming values rather than chaining sequence functions, as the latter approach creates "intermediate cached sequences".

However, I have also come across the advice "don’t hold on to your head" when dealing with sequences, because holding a reference to the beginning of the sequence prevents GC from freeing up memory used to hold previously seen values in the sequence.

At my current level of understanding these ideas appear to conflict. If sequences cache their results, then why does it matter whether you hold the head? Does caching not imply that all the realised elements are held onto anyway? Or looked at the other way, why would sequences cache their realised values if the GC is free to clean them up?

I know I must be missing something obvious here, but I would really appreciate an explanation.