Clojure Q&A - Recent activity in Sequences

Edited: Destructuring {:as opts} unexpected behaviour with seq

Sun, 05 Apr 2026 09:19:47 +0000

Hi,

I found out something weird in some old code, where I wanted to pass the first map in a sequence and by mistake the code was working while actually passing the sequence itself!

After digging to make sense of this, I found out it has to do with the {:as opts} destructuring.

Here's some code:

;; even if something is assigned sequential vs associative
;; the `:as` is transparent
(let [[:as opts] {1 2 3 4}]
  opts)
#_=> {1 2, 3 4}
(let [{:as opts} [1 2 3 4]]
  opts)
#_=> [1 2 3 4]
;; however with a seq it's returned as a map
(let [{:as opts} (seq [1 2 3 4])]
  opts)
#_=> {1 2, 3 4}

;; and it only gets weirder!
(let [{:as opts} (seq [{:k :v}])]
  opts)
#_=> {:k :v}
(let [{:as opts} (seq [{:k1 :v1} {:k2 :v2}])]
  opts)
#_=> {{:k1 :v1} {:k2 :v2}}
(let [{:as opts} (seq [{:k1 :v1} {:k2 :v2} {:k3 :v3}])]
  opts)
#_=> {{:k1 :v1} {:k2 :v2}, :k3 :v3}
(let [{:as opts} (seq [{:k1 :v1} {:k2 :v2} :not-a-map])]
  opts)
;; java.lang.IllegalArgumentException: Don't know how to create ISeq from: clojure.lang.Keyword
(let [{:as opts} (seq [1 2 3])]
  opts)
;; java.lang.IllegalArgumentException: Don't know how to create ISeq from: java.lang.Long

In my case I didn't realize my code was wrong because {:as opts} {:k :v} is the same as {:as opts} (seq [{:k :v}]).

To me this is behaviour I would expect closer to & {:as opts} rather than just plain map destructuring {:as opts}. Maybe both behaviours should be separated?

I think this should be tackled either by
1. make :as transparent regardless of what comes in to always bind
2. throw an error earlier when it's not of the expected destructuring
3. Document this very unexpected edge case in https://clojure.org/reference/special_forms#associative-destructuring
4. Separate the keyword arguments use case from plain map destructuring? (Assuming both are now mixed together)

Answered: Optimized str function

Sun, 29 Mar 2026 08:47:19 +0000

I've done some independent exploration of the options space for optimizing str which does not involve anything ground shattering but still gives good performance improvements

variadic arities (up to 20, yuck) will probably give the best performance. Can settle for common cases like up to 5 (apply str will still suck but whatever)
using reduce instead of a loop (would probably require redefining str or providing a str1 with the old impl for bootstrap purposes)
using inline meta, orthogonal to all other options

Additionally, having a function like str which only takes a sequence and reduces over it could provide a better code path than (apply str xs)

Similar work has been done in spork and stringer

Answer selected: seque forces n+2 items ahead of consumer instead of n

Wed, 09 Oct 2024 15:42:55 +0000

Logged as https://clojure.atlassian.net/browse/CLJ-2885

Commented: `sequence` partially consumes input

Mon, 07 Oct 2024 15:24:51 +0000

This is extremely important to me as I want to be able to use `sequence` in order to handle consuming over potentially stateful sequence generators.

In particular, I had to invent my own equivalent to `sequence` which did not eagerly consume the sequence in order to allow me to write a binary parsing library where multiple parsing functions consume different parts of the sequence, where any chunking or eager consumption would result in bad offsets and as a result an invalid parse of binary data.

Commented: Suggested improvements to clojure.core/distinct

Tue, 01 Oct 2024 17:02:43 +0000

For anyone who may be following this, I expanded the benchmarking, separating out the laziness change from the transducer variations. I also added a new variation for completeness.

To be honest, the best performing functions surprised me. I did not expect the persistent set to perform as well as it did, and I also did not expect the transient variations to perform worse for large sequences. However, these both made sense when I considered the shape of the data (large sequences with a small percentage of unique values).

I was also surprised that the transient set approach that skips the call to `contains?` performed as well as it did. While it avoids some of the code path of an earlier function, I am surprised that it didn't slow down similarly to the `distinct-transient` function.

Closed: `(nthrest nil n)` returns `nil` instead of `()`

Thu, 23 May 2024 15:15:43 +0000

According to the documentation nthrest should be equivalent to calling rest repeatedly, but its behaviour differs when coll is nil.

I also noticed neither of the nthrest and nthnext functions are included in the test-nil-punning.

Answered: How to map over a sequence of vectors?

Sun, 10 Dec 2023 10:22:26 +0000

You can turn that list of pairs (vectors) into a map with (into {} ...) (which also lets you use transducers, check them out). And then you'll be able to use update-vals.

(map #('((first %) (second %))) input) throws

The reader expands that #(...) into (fn* [p1__9#] ((quote ((first p1__9#) (second p1__9#))))) - you can check with (macroexpand-1 '#(...)).

The result of (quote ...) there is a list, and then you're calling that list, which is where the exception comes from.

(map #([(first %) (second %)]) input) makes the tooling complain

A similar thing - #(...) expands into something that immediately tries to call the result of [...] as a function with no arguments.

(map #(vector (first %) (second %)) input) throws Execution error (ClassCastException) a couple of lines later

Treating a map as a sequence gives you a sequence of kv-pairs with a type that implements java.util.Map$Entry. The keys function is supposed to be called on a map, not on a random sequence, and it just happens to work with a sequence of kv-pairs, but it expects for each pair to be a proper instance of that Map$Entry interface.
By using that (vector ...) you're turning each map entry into a vector that doesn't implement that interface.

Some generic advice:

For kv-pairs, use key and val instead of first and second - they also expect a map entry and will fail much earlier if something's wrong, making figuring stuff out easier. They're also faster and more indicative what the code's intent
If you have problems with #(...), try writing it as (fn [...] ...) with properly named arguments (so e.g. not x but kv-pair - doesn't sound like much but helps with figuring stuff out if you're stuck)
If you have ClassCastException, carefully read the docs of the functions that you're calling
When you see an exception, study its stacktrace and try to figure out what exactly went wrong. Might be hard at first but will become a breeze later, at least in most cases

Commented: clojure.lang.LazySeq synchronized methods pin virtual thread

Wed, 27 Sep 2023 12:05:34 +0000

Thanks for your attention, Alex!
I guess my banal scenarioses will just bring noise to already clarified "Request to find and eliminate virtual thread pinning" issue:
- blocking IO call from function called from clojure.core/map
- blocking IO call from clojure.core/deref

Thanks!

Commented: distinct should support sets

Mon, 28 Aug 2023 17:42:06 +0000

This should be fixed since the transducer version works fine and so behaves differently.

```
(into [] (distinct) #{:a :b})
=> [:b :a]
```

The regular distinct implementation chooses to use destructuring which needs nth.

And yes, we also have some util functions that take `coll` and are expected to work across all types of coll.

Answer selected: Infinite lazy sequence expands until JVM runs out of memory

Sun, 30 Jul 2023 18:20:58 +0000

I suspect your problem is that you are "holding onto the head" -- your function binds lazy-primes to the sequence and then you walk down the sequence so you still have a reference to the entire sequence, and if you ask for a large enough num, you are going to exceed the heap since your function needs the entire sequence to work.

Retagged: Add interleave-all (like partition-all is to partition)

Tue, 18 Jul 2023 13:33:11 +0000

At work we have some situations where we need to interleave lazy sequences in such a way that if one sequence is shorter than the other we want elements from the longer sequence to be concatenated after the interleaving runs out of the shorter sequence(s).

The following code could be used -- a slight variant of the existing interleave function:

(defn interleave-all
  "Like interleave, but stops when the longest seq is done, instead of
   the shortest."
  {:copyright "Rich Hickey, since this is a modified version of interleave"}
  ([] ())
  ([c1] (lazy-seq c1))
  ([c1 c2]
   (lazy-seq
    (let [s1 (seq c1) s2 (seq c2)]
      (cond
       (and s1 s2) ; there are elements left in both
       (cons (first s1) (cons (first s2)
                              (interleave-all (rest s1) (rest s2))))
       s1 ; s2 is done
       s1
       s2 ; s1 is done
       s2))))
  ([c1 c2 & colls]
   (lazy-seq
    (let [ss (filter identity (map seq (conj colls c2 c1)))]
      (concat (map first ss) (apply interleave-all (map rest ss)))))))

Answered: Is there a version of `range` that includes the end argument?

Tue, 20 Jun 2023 21:33:58 +0000

I often need something like this and so wrote the thru function:

`
tupelo.core/thru

(thru end)
(thru start end)
(thru start end step)

Returns a sequence of numbers. Like clojure.core/rng, but is inclusive of the right boundary value. Not lazy.
`

Commented: Performance improvements to creation of small vectors with TransientVector

Mon, 19 Jun 2023 20:22:28 +0000

Indeed, a raw patch is not the most human-friendly thing to look at in the browser. When viewed locally, most editors will at least colorize the changes, but you still can't navigate it as well as a git commit within the repo.

The thing is, sending patches is the only accepted workflow for contributing code to Clojure. That is why you'll see those Jira tickets with patches attached and no PRs on Github.

Anyway, here is my commit on Github: https://github.com/alexander-yakushev/clojure/commit/9d12dec06e4a9e0ec21e1b43741dc7d6adafc45e

Edited: Run a task parallely to delete tags

Mon, 05 Jun 2023 05:36:51 +0000

(defn- find-story-ids-by-tag [tag-id]

(map :id (db-content-tag/read-contents-by-tag-id (config/db-spec) tag-id)))

(defn- update-story [txn publisher-id story-id tag-id]
(let [{:keys [published-json]} (db-content/find-by-id txn story-id)

    updated-tags (filter (fn [tag] (not= (:id tag) tag-id)) (:tags published-json))
    _ (db-story/update-published-json-without-timestamps txn publisher-id story-id (assoc published-json :tags updated-tags))]
(log/info {:message "[TAG-DELETION] Updated Story Tags in Published JSON"
           :publisher-id publisher-id
           :tag-id tag-id
           :story-id story-id
           :updated-tags-json updated-tags})))

(defn- delete-tag [publisher-id tag-id]
(let [associated-content-ids (find-story-ids-by-tag tag-id)]

(transaction/with-transaction
  [txn (config/db-spec)]
  (do
    (when (seq associated-content-ids)
      (do
        (doseq [story-id associated-content-ids]
          (update-story txn publisher-id story-id tag-id))
        (db-content-tag/delete-batch-by-tag txn tag-id associated-content-ids)
        (log/info {:message "[TAG-DELETION] Deleted from Content Tag"
                   :publisher-id publisher-id
                   :tag-id tag-id
                   :story-ids associated-content-ids})))
    (db-tag/delete txn publisher-id tag-id)
    (log/info {:message "[TAG-DELETION] Deleted Tag"
               :publisher-id publisher-id
               :tag-id tag-id})))))

(defn run [publisher-id tag-ids]
(comment run 123 [4 5 6])
(try

(do
  (log/info {:message "[TAG-DELETION] started"
             :publisher-id publisher-id
             :tag-ids tag-ids})
  (doseq [tag-id tag-ids] (if (db-tag/find-by-id (config/db-spec) publisher-id tag-id)
                            (delete-tag publisher-id tag-id)
                            (log/info {:message "[TAG-DELETION] Tag Not Found"
                                       :publisher-id publisher-id
                                       :tag-id tag-id})))
  (log/info {:message "[TAG-DELETION] completed"
             :publisher-id publisher-id
             :tag-ids tag-ids}))
(catch Exception e
  (log/exception e {:message "[TAG-DELETION] errored"
                    :publisher-id publisher-id
                    :tag-ids tag-ids}))))

If I have less than 100 tags this works, but say I have 100 000 tags it is very time consuming. How to modify this code which can run in parallel and takes less time? Currently the task uses seq for both deleting and updating the story

Edited: Why do r/foldcat return types give inconsistent results when passed directly to `vec`?

Wed, 31 May 2023 03:28:13 +0000

When using reducers, I will occasionally use foldcat to "realize" the transformed collection through the reducer chain. For downstream code, its preferable to have a fully realized standard clojure collection like vec. This is often the case when updating core collection code to use reducers for performance reasons:

Note the following works just fine:

(->> (repeat 2 3) (vec) (r/map inc) (r/foldcat) (vec))
=> [4 4]

If the input gets large enough however, this triggers the parallel behavior which changes the return type from Array to Cat and causes the last vec to choke:

(->> (repeat 513 3) (vec) (r/map inc) (r/foldcat) (vec))
Execution error at user/eval19976 (form-init6687476487684153971.clj:1).
Unable to convert: class clojure.core.reducers.Cat to Object[]

I surmise that the "correct" way to do this is to use (into []) instead of vec in the last position which works just fine.

That said, this came as fairly surprising behavior given that seq and into work just fine on both Array and Cat. Its also quite dangerous because such code can function quietly in production until the input data grows passed a certain threshold. Is there pragmatic reason that seq works for Cat but vec doesn't?

Answered: Partitioning a list based on comparing the elements within the list

Mon, 08 May 2023 08:25:25 +0000

The library dev.weavejester/medley ("A small collection of useful, mostly pure functions that might not look out of place in the clojure.core namespace") contains partition-between since 1.7.0. That function can be used to partition a sorted list:

(require '[medley.core :refer [partition-between]])

(partition-between (fn [x y] (> y (inc x))) [1 2 3 7 8 14 15 16 17 20 21 22])
;; => ((1 2 3) (7 8) (14 15 16 17) (20 21 22))

Commented: Use transducers on clojure.core lazy sequence transformations

Thu, 13 Apr 2023 16:59:10 +0000

Clarification from Alex on Slack:

> I was really referring to reduce in that sentence
> it [eduction] is delayed
> it is often used in eager scenarios
> it's iterator based, but often used with reduce, which is eager

More details in the thread at https://clojurians.slack.com/archives/C03S1KBA2/p1681394119498109?thread_ts=1681383034.191669&cid=C03S1KBA2

Commented: Tree recursion in Clojure

Wed, 05 Apr 2023 20:01:23 +0000

You should be able `loop/recur` if you keep a stack in your loop onto which you push things that you need to visit later? That's basically what you do with recursion?

Answer selected: find on seqs

Thu, 28 Apr 2022 09:28:54 +0000

It has been discussed and rejected before https://clojure.atlassian.net/browse/CLJ-2056

Answered: Calling str on lazy-seq and eductions has confusing semantics

Thu, 29 Jul 2021 00:30:14 +0000

Logged as https://clojure.atlassian.net/browse/CLJ-2647

Answer edited: Why does partition-by iterates over elements more than once?

Sat, 08 May 2021 09:39:51 +0000

([f coll]
 (lazy-seq
  (when-let [s (seq coll)]
    (let [fst (first s)
          fv (f fst)
          run (cons fst (take-while #(= fv (f %)) (next s)))]
      (cons run (partition-by f (lazy-seq (drop (count run) s))))))))

Looks like f gets called 1x when binding fv. Since the take-while portion doesn't yield any similar keys, the value of run is '(1). However, f "was" invoked on the first value of the rest of s, so 2 gets printed 1 time. When partition-by pseudo recurses during the lazy-seq construction, the first arg of the new partition is now 2. fv is determined by invoking f on 2, which prints two, etc. The process repeats.

The transducer variant never passes the head of the next partition recursively, so it only ever calls f once per element of the original input.

Note: we can derive a 1-time partition function variant without state like so:

user> (defn partition-by-once [f coll]
        (->> coll
             (map (fn [x] [(f x) x]))
             (partition-by first)
             (map #(map second %))))
#'user/partition-by-once
user> (partition-by-once #(do (println %)
                               (rand)) [1 2 3])
1
2
3
((1) (2) (3))

We can do the volatile trick that the transducer uses to get an implementation too:

    (defn partition-by-once-statefully
      [f coll]
      (let [pv   (volatile! ::none)
            test (fn [v] (vreset! pv (f v)))
            aux  (fn aux
                   ([coll]
                    (lazy-seq
                     (when-let [s (seq coll)]
                       (let [fst (first coll)
                             fv  @pv
                             run (cons fst (take-while #(= fv (test %)) (next s)))]
                         (cons run (aux (lazy-seq (drop (count run) s))))))))
                  ([fst remaining]
                   (if-let [s (seq remaining)]
                     (let [fv  (test fst)
                           run (cons fst (take-while #(= fv (test %)) s))]
                       (cons run (aux (lazy-seq (drop (dec (count run)) s)))))
                     `((~fst)))))]
        (when-let [s (seq coll)]
          (aux  (first coll) (next coll)))))

    user> (doall (partition-by-once-vol (fn [v] (println v) (odd? v)) [1 2 3 5 7 8 10 11]))
    1
    2
    3
    5
    7
    8
    10
    11
    ((1) (2) (3 5 7) (8 10) (11))

Commented: Does def hold onto the head?

Wed, 24 Feb 2021 00:11:01 +0000

ah, the issue is nth is being inline expanded to a static method call instead of a function call, and the reflective static method call the interpreter does doesn't attempt to avoid holding the head.

Commented: Suspected bug in `clojure.core/partition`

Tue, 29 Dec 2020 22:05:22 +0000

The `step` parameter isn't relevant to the described behavior: (partition n coll) just means (partition n n coll), i.e., partition with a step of n, so it _always_ has a step value and that just determines how many elements are skipped in coll before starting the next partition. What Alex has described as the behavior is not affected by the step value.

The pad collection only matters if the last partition that would be generated doesn't have n items in it:

dev=> (partition 3 3 [:a :b :c] (range 5))

((0 1 2) (3 4 :a))

dev=> (partition 3 3 (range 5))

((0 1 2))

dev=> (partition 3 2 [:a :b :c] (range 5))

((0 1 2) (2 3 4) (4 :a :b))

dev=> (partition 3 2 (range 5))

((0 1 2) (2 3 4))

dev=> (partition 3 1 [:a :b :c] (range 5))

((0 1 2) (1 2 3) (2 3 4) (3 4 :a))

dev=> (partition 3 1 (range 5))

((0 1 2) (1 2 3) (2 3 4))

Commented: Why is r/fold not parallelized across all available cpus?

Mon, 20 Jul 2020 21:07:20 +0000

Got it. That makes sense, and is a cool example of the streaming possibilities of `clojure.data.csv`. This machine has 32GB of ram, so I was able to amend my code to write out 100M row csv and nippy datasets. The csv on disk is ~658MB and the nippy is ~537MB. The csv dataset loads in ~47s and the nippy dataset loads in ~2.8s. (:

Answered: On the realization of lazy-seqs which throw exceptions

Mon, 25 May 2020 15:13:15 +0000

Variants of this have been discussed in the past. The general advice is: we don't make any guarantees about what happens if lazy seqs throw on realization. If you're doing side effect-y things in your lazy seq, consider using run!, loop/recur, etc.

See https://clojure.atlassian.net/browse/CLJ-2069 for a similar ticket.

Answer edited: Duplicated `next` before `recur`?

Tue, 11 Feb 2020 11:32:30 +0000

Could it be the order in which functions are defined in core?

Skimming through core.clj the if-let macro is actually defined at line 1841:

https://github.com/clojure/clojure/blob/653b8465845a78ef7543e0a250078eea2d56b659/src/clj/clojure/core.clj#L1841

while the "next more recur next more" fragments (in math stuff) are used before let or if-let exists (line 1055):

https://github.com/clojure/clojure/blob/653b8465845a78ef7543e0a250078eea2d56b659/src/clj/clojure/core.clj#L1055

Code after defmacro if-let is actually using the pattern you expect (line 3690):

https://github.com/clojure/clojure/blob/653b8465845a78ef7543e0a250078eea2d56b659/src/clj/clojure/core.clj#L3690

I think there is the regular single pass compilation with core.clj as with regular clojure code. Functions have to be defined before you can use them, both macros and defns.

I am not sure why the order have to be like this, maybe something with bootstrapping.

Comment edited: changing some?

Thu, 26 Sep 2019 18:45:52 +0000

I agree with Andy that having a real-time discussion on Slack may be a lot more productive since right now it sounds like you can't even articulate any specific questions to help remove any confusion?

Answered: pmap - hard wired level of parallelism

Fri, 16 Aug 2019 04:41:06 +0000

As Sean said pmap is a quick-and-dirty mechanism which you should generally avoid.
People often recommend using java Executors framework instead (via interop).
There's also an alternative implementation of pmap (and other concurrency constructs in https://github.com/TheClimateCorporation/claypoole where you can specify your own thread pool.