Clojure Q&A - Recent questions in Collections

select-keys on nil map

Fri, 01 Aug 2025 14:42:56 +0000

I know that https://ask.clojure.org/index.php/1913/use-transients-with-select-keys-if-possible exists to optimize select-keys for many keys, but I noticed that it does a lot of work even if the original map is nil. Would there be interest in a patch that returns an empty map when given nil? Something like:

(defn select-keys*
  "Returns a map containing only those entries in map whose key is in keys"
  {:added "1.0"
   :static true}
  [map keyseq]
  (if map
    (loop [ret {} keys (seq keyseq)]
      (if keys
        (let [entry (. clojure.lang.RT (find map (first keys)))]
          (recur
           (if entry
             (conj ret entry)
             ret)
           (next keys)))
        (with-meta ret (meta map))))
    {}))

`merge-with` doesn´t treat pair (vectors of 2) the same way `merge` does

Thu, 01 May 2025 11:25:20 +0000

;; merge
(merge {:a :b} [:a :c]) ;=> {:a :c}

;; merge-with
(merge-with (fn [_ x] x) {} {:a :b} [:a :c])
;; => class clojure.lang.Keyword cannot be cast to class java.util.Map$Entry

I think this happens because merge-with eventually calls seq on the "Map", expecting to get a "Map Entry" from which then it can extracts key and value, and fails when receiving a Vector, because seq will just yield one element at a time:

(key (first (seq {:a :c}))) ;; => :a
(key (first (seq [:a :c])))
;; => class clojure.lang.Keyword cannot be cast to class java.util.Map$Entry

I think it is justified to expect that (partial merge-with (fn [_ x] x)) should behave exactly as merge and my proposal is to make it happen, if what I described makes sense.

Thank you!

partition-all with step yields surprising result

Tue, 26 Nov 2024 21:01:19 +0000

I was expecting (partition-all n step coll) to be equal to (partition n step nil coll) but that's not always the case.

(partition-all 3 1 (range 5))
;=> ((0 1 2) (1 2 3) (2 3 4) (3 4) (4))
;; Notice the second to last partition is (3 4).  Why not stop there?

;; My work-around is to use a nil pad.  This gives my expected result
(partition 3 1 nil  (range 5))
;=> ((0 1 2) (1 2 3) (2 3 4) (3 4))

Gather all keys and values of nested map that satisfies some condition

Thu, 07 Nov 2024 21:38:20 +0000

Hi,
From a nested map, I'd like to build a map that saves all key - value pairs whose keys satisfy some conditions.
I'm using babashka.
Tried some different solutions, but I'm not able to retrieve all pairs.

The first one, using (I think) a more functional approach j-h5.

    #!/usr/bin/env bb
    
    ; expected output: {"k23" {"k231" "v231"} "k24" "v24"}
 (def j {"k1" "v1" "k2" {"k21" "v21" "k22" "v22" "k23" {"k231" "v231"} "k24" "v24"} "k3" "v3"})
(println j)

(println "Doing j-h5")
(defn j-h5 [s mymap]
  (let [first-k (first (first s))
        first-v (second (first s)) ]
    (cond 
      (empty? s) nil
      (or (= "k23" first-k) (= "k24" first-k))
      (do 
        (assoc mymap first-k first-v)
        (when (and ( map? first-v) (seq first-v))
            (println  (str "doing recursion1 when k:" first-k "  and v:" first-v))
            (recur first-v mymap))
        )
      (map? first-v)
            (do 
              (println  (str "doing recursion2 when k:" first-k "  and v:" first-v))
              (recur first-v mymap))
      :else (recur (rest s) mymap) 
      )))

(j-h5 j {})

; mymap defined outside of function   
(println "Doing j-h1")
(def  mymap {})
(defn j-h1 [s]
  (let [first-k (first (first s))
        first-v (second (first s)) ]
    (cond 
      (empty? s) mymap
      (or (= "k23" first-k) (= "k24" first-k))
      (do 
        (assoc mymap first-k first-v)
        (when (and ( map? first-v) (seq first-v))
            (println  (str "doing recursion1 when k:" first-k "  and v:" first-v))
            (recur first-v))
        )
      (map? first-v)
            (do 
              (println  (str "doing recursion2 when k:" first-k "  and v:" first-v))
              (recur first-v))
      :else (recur (rest s)) 
      )))

(j-h1 j)

How can this be fixed?

Apply two functions to a collection and to another that is a transformation of the other one

Tue, 22 Oct 2024 22:13:46 +0000

Hi,

The challenge is to get the server name part of a collection of strings of fully qualified domain names.
From: ["server1.oo.com" "s2.ee.com"]
To: ["server1" "s2"]

I can do this to a single element as:

(subs "server1.oo.com" 0 (str/index-of "server1.oo.com" \.))

(subs "server1.oo.com" 0 (#(str/index-of % \.) "server1.oo.com"))

But nested #() functions seem not to be possible.

(#(subs % 0 (#(str/index-of % \.) "server1.oo.com")) "server1.oo.com")

returns an error.

When using the collection, something like this works:

(let 
  [s-col ["server1.oo.com" "s2.ee.com"]]   
  (map #(subs % 0 5) s-col))

But, how can I use the (str/index-of % \.) instead of that hard-coded 5?
All my attempts were unsuccessful.

How to persist Clojure data structures to disk?

Tue, 22 Oct 2024 08:45:52 +0000

Hello,
I'm building a web crawler in Clojure that records its results in a map of maps. The data structure basically looks like this:

{"https://example.org" {:base-url "https://example.org"
                        :crawled {"/blog" {:relevant true :status 200}
                                  "/contact" {:relevant false :status 200}}
 "https://foo.com" {:base-url "https://foo.com"
                    :crawled {"/home" {:relevant true :status 200}}}

The web crawler is feed with a list of websites. However, for the sake of efficiency I want to avoid crawling already crawled websites. Therefore, I was looking for a way to persist the above data structure so that it can be consulted in subsequent runs to check if a given website was already crawled. (Of course, I have other reasons to store the results too, e.g. to do various analysis on the captured data)

First, I thought EDN would be a perfect solution. To my surprise, clojure.edn does not have write functions. A lot of advice online says to use pr-str. However, using pr-str has caveats as pointed out in https://nitor.com/en/articles/pitfalls-and-bumps-clojures-extensible-data-notation-edn (e.g. possbile truncations when print-length is set). And it's totally possible to generate invalid edn as pointed out in this comment by @alexmiller: https://ask.clojure.org/index.php/10278/should-pr-str-on-a-function-produce-valid-edn-syntax?show=10282#c10282.

Despite https://clojuredocs.org/clojure.edn claiming that it is "designed to be used in a similar way to JSON or XML" I have the impression it is more geared towards writing edn files by hand (i.e. configuration files).

Looking for alternatives, I stumbled upon Transit. However, its mainly intended "as a wire protocol for transferring data between applications". And the README further says

If storing Transit data durably, readers and writers are expected to use the same version of Transit and you are responsible for migrating/transforming/re-storing that data when and if the transit format changes.

Because in Clojure, data is code and code is data and because of the existence of EDN, I really thought persisting simple data structures like the above would be a breeze in Clojure.

Instead I see these sub-optimal options with my current knowledge:

Hope that pr-str produces valid EDN without loss in my particular case
Build a custom XML serialization
Use Transit
Use JSON?! (what about keywords etc.?!)

Is there a more idiomatic way to store and read back in simple Clojure data structures?

Optimize `select-keys`

Tue, 11 Jun 2024 07:18:28 +0000

Some preliminary testing show that select-keys can be almost twice as fast if it were rewritten to use a transient map rather than a proper map as it does today.

In order to do so, select-keys would need to move to after where transient and its friends are defined.

If there is any interest in pursuing this, I'd be happy to provide a problem statement, more data on the performance, a couple of possible solutions, and perhaps ultimately a patch.

Cheshire parse-string coerce keys to keywords

Wed, 06 Mar 2024 18:06:15 +0000

Just learning Clojure. I have the following json string:

(def json "{\"a\": 1, \"b\": \"val\", \"c\": {\"D/E\": [1, 2, 3]}}}")

If the key-fn param is true in the Cheshire parse-string call I got the following map:

(def parsed-json (cheshire/parse-string json true))
;; ==> {:a 1, :b "val", :c #:D{:E [1 2 3]}}
(type parsed-json)
;; => clojure.lang.PersistentArrayMap
(type (:c parsed-json))
;; => clojure.lang.PersistentArrayMap

How can I access the [1 2 3] array? I don't understand what is #:D here, and how I can access :E.

CLJ-2713 (IDrop) causes exception in previously working code

Fri, 09 Feb 2024 17:51:12 +0000

In Clojure 1.11.1:

user=> (split-at Long/MAX_VALUE [1 2 3 4 5 6])
[(1 2 3 4 5 6) ()]

In Clojure 1.12 alpha7:

user=> (split-at Long/MAX_VALUE [1 2 3 4 5 6])
Execution error (ArithmeticException) at java.lang.Math/toIntExact (Math.java:1371).

Possibly IDrop should take a long parameter (instead of int) for compatibility?

Why doesn't clojure.set provide a "symmetric difference" function?

Mon, 25 Sep 2023 20:44:24 +0000

Is there a particular reason why clojure.set does not provide a symmetric-difference function?

It seems a relatively frequent thing to want.

Use transducer arity of `into` in `clojure.walk` namespace

Wed, 13 Sep 2023 22:49:05 +0000

It's simple. Just like that:

> clojure.walk/walk:
(outer (reduce (fn [r x] (conj r (inner x))) form form))
=> (outer (into form (map inner) form))
(outer (into (empty form) (map inner form)))
=> (outer (into (empty form) (map inner) form))

> clojure.walk/keywordize-keys:
(postwalk (fn [x] (if (map? x) (into {} (map f x)) x)) m)
=> (postwalk (fn [x] (if (map? x) (into {} (map f) x) x)) m)

> clojure.walk/strigify-keys:
(postwalk (fn [x] (if (map? x) (into {} (map f x)) x)) m)
=> (postwalk (fn [x] (if (map? x) (into {} (map f) x) x)) m)

Immutable wrapper around Java arrays?

Mon, 28 Aug 2023 11:20:27 +0000

Are there any immutable wrapper around Java arrays in Clojure, similar to Scala's ArraySeq?

What are your thoughts on introducing such a wrapper to Clojure's core?

Efficient merge

Sat, 26 Aug 2023 18:28:54 +0000

I was wondering why the merge function doesn't use transients and if there could be a better version of that.

Instead of this:
(defn merge [& maps] (when (some identity maps) (reduce #(conj (or %1 {}) %2) maps)))

we can use transducers to keep the code still easy to read and have better performance:
(defn merge [& maps] (when (some identity maps) (into {} cat maps)))

Is there any performance difference between creating vector via (into [] xs) or (vec xs)?

Sat, 03 Jun 2023 11:35:12 +0000

I can create set/map/vector by two ways:

(into #{} xs) ;; creates PersistentHashSet
(set xs)      ;; creates PersistentHashSet

(into {} xs)  ;; creates PersistentArrayMap or PersistentHashMap (depends on xs size) 
(apply hash-map xs) ;; creates PersistentHashMap

(into [] xs)  ;; creates PersistentVector
(vec xs)      ;; creates PersistentVector

(into r xs):
- defines generic algorithm by conj!-ing input sequence 1 by 1 to passed result collection (uses transient/persistent trick)

(ctor xs):
- defines precisely type of result data structure
- passes xs sequence as an argument to constructor (which is static java function) so every ctor may define its own logic of creation (maybe by not conj-ing input sequence 1 by 1)

Are there any performance differences between these two ways?
Maybe there is an established idiom what to use when?

Thanks!

merging pairs of pairs?

Fri, 14 Apr 2023 20:29:10 +0000

i want to create a list of pairs, and can build the list best as a series of pairs of pairs:

(for [ni (range n)
          di (range d)]
     (let [j1 (mod (inc di) n)
              j2 (mod (- n (inc di)) n) ]
          (conj [] [ni j1] [ni j2]) ))

this gets me close:

([[0 1] [0 9]] [[0 2] [0 8]] [[1 1] [1 9]]...

but is adding an extra level of brackets. what i want would look like this:

([0 1] [0 9] [0 2] [0 8] [1 1] [1 9] ...

I am confident there is a simple solution for this but I can't figure out what it!?

A faster merge?

Fri, 17 Mar 2023 11:35:44 +0000

merge is implemented as

(defn merge
 ;; stuff removed
  [& maps]
  (when (some identity maps)
    (reduce1 #(conj (or %1 {}) %2) maps)))

This is somewhat suboptimal from a couple of reasons. reduce1 is slower than reduce (might not matter when the number of maps is small, but perhaps more importantly, in the case where you just want to merge two maps, it does a lot of unnecessary work, since merging two maps is basically

(when (or m1 m2) (conj (or m1 {}) m2))

Would it be reasonable to have a multi arity version of merge which optimized for this case?
A quick benchmark shows a potential for a 4 times speed up in this case.

I've looked through Jira, but surprisingly, I wasn't able to find a ticket for this?

submap? - recursively check if one map is contained in another

Thu, 09 Mar 2023 19:24:33 +0000

(submap? m1 m2) recursively checks if m1 is contained in m2

Usage:

(deftest submap
  (is (submap? {} {}))
  (is (not (submap? {:k 1} {})))
  (is (submap? {:k 1} {:k 1 :j 2}))
  (is (submap? {:k {:kk 1}} {:k {:kk 1 :j 2}}))
  (is (not (submap? {:k {:kk 1}} {:j 2}))))

This need often arises in tests, where you want to check if the map returned by function-under-test has a certain shape and while assuming an open world - i.e. you want to ignore any additional keys so that changing function-under-test doesn't break the tests:

;; BAD: doesn't assume open world
(is (= {:k :v} (function-under-test)))

;; BAD: not expressive
(is (= :v (:k (function-under-test))))

;; BAD: not expressive, need to write :k twice
(is (= {:k :v} (select-keys (function-under-test) [:k])))

;; GOOD: expressive
(is (submap? {:k :v} (function-under-test)))

Implementations already exist in many codebases (400+ hits on Github code search). Examples:

https://github.com/clojure/tools.deps/blob/ecc80420c1b734b384f7a42df91684cdbc37ddc6/src/test/clojure/clojure/tools/deps/util.clj#LL12-L25C18

https://github.com/ptaoussanis/encore/blob/96547700c03a5e81d784bcc9253980d7355f2f2b/src/taoensso/encore.cljc#L2071

Function to create a concrete list

Wed, 15 Feb 2023 19:10:11 +0000

The core collection data structures (lists, vectors, maps, and sets) have variadic functions to create them: list, vector, hash-map, hash-set. 2 of them have 1-arity functions that take in a collection and convert them to the desired concrete collection type: vec and set. Maps and lists don't have such a function. I think I understand why maps don't: Hard/impossible to convert between it and other collection types when it requires pairs without a middle step to perform the pairing.

But it seems that there's a missing function that takes in a single object of any collection type and converts it to a concrete PersistentList. Something as simple as (defn to-list [coll] (clojure.lang.PersistentList/create coll)). Is there a specific reason for its exclusion?

My reasons for asking/desiring such a function are because I am building and then traversing nested data structures, and it can be challenging to correctly handle branches where the output of the various Clojure core functions that operate on "lists" end up returning non-PersistentLists (such as list* and cons and syntax quote). I wrote my own to-list and have been peppering it where necessary to make my decision trees easier after realizing that list* didn't fit the bill.

I know that the sequence abstraction is a big part of the mindset of Clojure, but list? not returning true on things I wrote that I know for sure have a bounded size has led to some frustration.

I'm primarily interested in hearing about any reasoning for the deliberate exclusion of such a function but alongside that request, if the core team is interested, I can provide a patch for its addition.

Addition/Multiplication operations on loop

Sat, 10 Dec 2022 05:55:32 +0000

Hi!
It's me again. I followed up on the previous answers that have been posted on my questions to do another operation but I don't get exactly what I want.
Here's the situation:

My first table (products) looks like : (product_name price)
([1 (candies 6.5)] [2 (sweets 1.75)]
[3 (jam 2.99)] [4 (gum 1.25)])

My first table (products) looks like : (customer_name product_name quantity)
([1 (Sara candies 3)] [2 (Joe jam 3)]
[3 (Sara gum 1)])

What I'm trying to do is if i type for example Sara, I'll get the sum of the sales made by Sara, which means: (36.5 + 11.25) = $20.75 (in this case)

I'm fine for the input part (I get the name of the customer as an input from the terminal)
However, my code:

(defn sales_prices_with_cond [name_cus] (map
 (fn [y z]
   (if (= (str name_cus) (str (nth (first z) 1)))
   list (* (Double. (nth (second y) 1)) (Integer/parseInt (nth (second z) 2))))
   )
products 
sales))
(println (reduce + sales_prices_with_cond "Sara"))

Gives me the sum of ALL the sales*quantities. It's like the condition is skipped or maybe not well written ...
I also tried with (some) and got the same result...

Please Help :') .

Help with looping/filtering elements

Fri, 09 Dec 2022 20:19:28 +0000

Hi!
I'm a newbie in Clojure and I'm struggling with looping through data in data structures.

I have a list that looks like this : (list1)
([1 (item1 item2 item3)]
[2 (item21 item22 item33)])

And another one that looks like this: (list2)
([1 (rec1 rec2 rec3)]
[2 (rec21 rec22 rec33)])

And the last one is: (list3)
([1 (1 1 3)]
[2 (2 2 3)])

Now, I have to loop into list3 and create a new collection that replaces the 1 (in my list3) by "item1" and replaces the second 1 (in my list3 too) by "rec1" ... and so on for all the vectors in list3...

The three lists have the exact same number of records.

My third list should look like :
([1 (item1 rec1 3)]
[2 (item21 rec22 3])

Any idea how to proceed?

Can group-by be generalized?

Tue, 18 Oct 2022 15:37:22 +0000

It would be nice if group-by let users control the aggregated collection type and manipulate values before aggregation. It's a common scenarios when grouping a collection of maps according to one key, and maybe aggregating, even numerically according to another.

group-by could be generalized like so:

(defn group-by
  "Returns a map of the elements of coll keyed by the result of
  f on each element. The value at each key will be a vector of the
  corresponding elements, in the order they appeared in coll."
  {:added "1.2"
   :static true}
  ([kf coll]
   (group-by kf [] coll))
  ([kf init coll]
   (group-by kf identity init coll))
  ([kf vf init coll]
   (group-by kf vf conj init coll))
  ([kf vf rf init coll]
   (persistent!
    (reduce
     (fn [ret x]
       (let [k (kf x)]
         (assoc! ret k (rf (get ret k init) (vf x)))))
     (transient {}) coll))))

Why does `(str (map identity [1 2]))` return `clojure.lang.LazySeq@`? Is this a bug?

Wed, 12 Oct 2022 19:21:59 +0000

Why does (str (map identity [1 2])) return clojure.lang.LazySeq@3e2?
Is this a bug?
For example, (str (seq (map identity [1 2]))) returns (1 2) while (str (doall (map identity [1 2]))) still returns clojure.lang.LazySeq@3e2.

Does it make sense to define filter-keys/filter-vals functions for maps in clojure.core?

Sat, 08 Oct 2022 13:40:25 +0000

Hi!

Does it make sense to define filter-keys/filter-vals functions for map which takes a map and a predicate and returns a map satisfied predicate?

For example, there is an utility lib called medley and it defines such functions for maps as filter-keys, filter-vals and some others.

It seems that there is a reason for not having it in the core lib.
Using core lib i can use select-keys to query a map with a vector of keys and get a map with requested keys if map has it but there is no more advanced query functions for map except powerful reduce-kv which mixes condition + new map building.

Clojure set lib defines a set/select function allowing to apply a predicate over a set and gets a set as a result.

Any ideas why there is no similar functions for maps?

How read list of files, read all data from and compare with another list of keys?

Mon, 03 Oct 2022 12:26:25 +0000

How to read a list of files, compare the text in the files with a set of keys, and return the key with the value true or false if that key is found in the file?

For example I have list of my files:

["src/cljs/project/cofx.cljs"
 "src/cljs/project/datetime.cljs"
 "src/cljs/project/core.cljs"
 "src/cljs/project/app_db.cljs"]

and set of keys:

#{:tooltip/terminate
  :member-services/description
  :event-errors/link-duplicated
  :firm-admin/pre-approved-applicants}

I want read all files (I used slurp), compare text from all files with first key and return this key and true and name of file if key exist in this file, or false if key not exist in all files.

for example:

[:tooltip/termiante true "src/cljs/project/cofx.cljs"
 :member-services/description false]

Do you have any tips? Because this task very difficult for me.

`update-keys` and `update-vals` missing the `& args` from `update`

Tue, 09 Aug 2022 15:03:31 +0000

The & args in update, swap! and etc are very ergonomic and compose really well, so I was surprised to learn that update-keys and update-vals don't have that, they're only [m f].

What is the reason for this?

I think adding & args to these functions would be more consistent with the existing core functions and help avoid anon functions where not needed.

SortedMap not IEditableCollection

Mon, 01 Aug 2022 06:46:53 +0000

It seems to me that the reason https://ask.clojure.org/index.php/12073/update-vals-not-preserving-sorted-maps is a question at all, is that PersistentTreeMap doesn't implement IEditableCollection.

So to satisfy my own curiosity, why is this?

`update-vals` not preserving sorted maps?

Wed, 27 Jul 2022 08:35:05 +0000

Hello! We recently found that when applying update-vals to a sorted map, it's converted back into an unsorted array/hash map due to the collection not being transientable. Example given below. As a note, this differs from medley/map-vals which preserves sorting and which we were using for this functionality previously to 1.11.

Is this intended behavior? If so, I'd be curious about the intent behind it :) Thanks!

user> (def m (sorted-map :a 0 :b 0))
;; => #'user/m
user> (type m)
;; => clojure.lang.PersistentTreeMap
user> (type (update-vals m inc))
;; => clojure.lang.PersistentArrayMap
user> (type (medley.core/map-vals inc m))
;; => clojure.lang.PersistentTreeMap

Cast to Map$Entry Exception When Trying to (Into {}) After Sequence Operations

Sun, 08 May 2022 09:17:12 +0000

Is it intended behavior to get this exception :

class clojure.lang.Keyword cannot be cast to class java.util.Map$Entry

on this piece of code :

(def a [["Active" :Transit-RLC "Active"]])

(->> a
     (transduce
       (comp 
             (map drop-last)
             (map reverse))
       conj [])
     (into {}))

changing the transducer to (map (-> [(second %) (first %)]) solves the issue.
Thanks

Should clojure.lang.Range/LongRange be made to implement clojure.lang.Reversible?

Sun, 01 May 2022 00:13:06 +0000

Sometimes I want to iterate through a range like (range 100), but in reverse order. For this, there are two existing choices:

1) The programmer thinks about the would be the right start/end/step to construct the reversed range, and constructs the range accordingly, with a negative step:

(range 99 -1 -1)

2) Use reverse as in (reverse (range 100))

Solution 1 is not great since well it is harder to think about because you have to change the start/end because of the reversed bounds inclusive/exclusive bounds on the start and end. It would also require more mental math than I like to do, in the case where step is not 1. For instance, quickly solve how to reverse (range 23 96 7)! The answer is (range 93 22 -7), but it's not ideal to have to do this math in your head and to commit it to code when really what you meant was (reverse (range 23 96 7)). For instance, if you tomorrow want to change the lower bound from 23 inclusive to 24 inclusive, you have to actually recompute the upper bound and change the code to (range 94 23 -7).

This would get even more difficult/impossible for the programmer if they're using non-integer ranges:

(range 1.5 5.4 1.23)

Solution 2 is not great since it takes O(n) space and time:

(defn reverse
  "Returns a seq of the items in coll in reverse order. Not lazy."
  {:added "1.0"
   :static true}
  [coll]
    (reduce1 conj () coll))

So then what?

Reversible

Maybe the clojure.lang.Range/clojure.lang.LongRange objects could implement clojure.lang.Reversible, since they can be reversed in O(1) time and space, with something like:

public ISeq rseq() {
    final Number difference = Numbers.minus(end, start);
    final Number remainder = Numbers.remainder(difference, step);
    final Number last = Numbers.isZero(remainder) ?
            Numbers.minus(end, step) : Numbers.minus(end, remainder);
    final Number reverseStep = Numbers.minus(step);
    BoundsCheck bc;
    Object newEnd;
    if (Numbers.isPos(reverseStep)) {
        newEnd = Numbers.inc(start);
        bc = positiveStep(newEnd);
    } else {
        newEnd = Numbers.dec(start);
        bc = negativeStep(newEnd);
    }
    return new Range(last, newEnd, reverseStep, bc);
}

And then the reversing could be done in clojure using the rseq function

(rseq (range 10))
=> (9 8 7 6 5 4 3 2 1 0)

Problems with this approach and/or challenges:

Still doesn't work on "empty ranges" because when you do (range 0 0) you actually get an empty list (not a Range object actually). So since lists aren't Reversible, you'd get an exception when you try to rseq it. This could be fixed by allowing there to be an empty Range object, or perhaps by making empty lists Reversible? Anyways probably too aggressive of a change. I actually don't think there should be a problem with allowing an empty Range, it's not clear to me if it's important that empty ranges are specifically lists, but for instance it isn't the case that the empty Vector is a list, or that the empty Set/Map are the empty list, so it doesn't seem to be bizarre that there would be an empty Range.
some more thought would have to go into the algorithm to compute the reverse range in the case of floating point ranges, since the above algorithm doesn't produce the exact same result in reverse. For instance:

(range 1.5 5.4 1.23)
=> (1.5 2.73 3.96 5.1899999999999995)
(rseq (range 1.5 5.4 1.23))
=> (5.1899999999999995 3.9599999999999995 2.7299999999999995 1.4999999999999996)

If not that, then maybe `:reverse true` keyword arg

Maybe the reversibility belongs more in the range function:

(range 10 :reverse true)

or similar.

Anyways I am not saying Clojure's wrong or anything, just submitting some food for thought. Thanks for reading!

Is lazy sequence chunking deterministic?

Mon, 28 Mar 2022 16:11:27 +0000

I know that lazy sequences can be processed in a chunked manner. That is, it may be that if my program needs n elements from a lazy sequence, more than n might be realized.

Can I assume that if I run the same code a second time in the same version of Clojure, that exactly the same elements will get realized? I would think so, but I don't want to assume that without knowing.

(I'm including additional info here in case anyone is interested in why this matters, but I don't think it's necessary to know my use case in order to answer the question. So feel free to skip this paragraph. I'm running simulations that make use of a pseudorandom number generator. These involve random walks that are truncated if the walk encounters a "target". It's convenient to generate a walk in advance as a lazy sequence, and then stop realizing its elements when a target is found. Normally, I throw away intermediate results such as the original random walk. They exist in a data structure at one time, but then I don't use them in the end. However, I want to be able to go back and rerun a simulation, sometimes, in order to examine the intermediate results. It's easy enough to store the seed that I used for a simulation run, and use it to set the random number generator to start in the same state. However, chunking can cause the random number generator to be called more times than are strictly necessary. That's OK, but if chunking can vary depending on something other than my code and the Clojure version, that could mean that rerunning a simulation with the same seed would not be guaranteed to produce the same results. This is not at present a big deal. I can make the intermediate data non-lazy without much of a performance hit. However, I'm still wondering whether that's necessary.)

clojure.set different behaviour depending on the position of nil

Sun, 06 Mar 2022 08:50:03 +0000

If a nil reaches, e.g., clojure.set/union or clojure.set/intersection, the output depends on the position. I think those functions should give back always a hash-set, irrespective of nils or the position of nil.

$ clj
Clojure 1.10.3
user=> (require '[clojure.set :refer [intersection union]])
nil
user=> (union #{} nil)
#{}
user=> (union nil #{})
nil
user=> (intersection #{} nil)
#{}
user=> (intersection nil #{})
nil
user=> (union #{} nil nil)
nil
user=> (union nil #{} nil)
nil
user=> (union nil nil #{})
#{}
user=> (intersection #{} nil nil)
nil
user=> (intersection nil #{} nil)
nil
user=> (intersection nil nil #{})
#{}

I think the order of the arguments shouldn't matter and should always be a set, in my opinion, just assuming nil as an empty set. Plus if one intends to call later (set-of-things thing) an exception might arise (this is what bit me).

And maybe this is not limited to union and intersection only and more functions have this "issue"

Common patterns for optional fields interact poorly with spec

Tue, 15 Feb 2022 21:48:34 +0000

I do lots of work with spec, and a frequent problem that I run into is in constructing a spec'ed object that is a map with optional fields.

A common pattern looks like this:

(assoc {::required-field :value}
       ::optional-field (when some-condition
                          :optional-value))

There is a significant issue with this pattern however: if ::optional-field is on a spec/keys :opt field that is non-nilable, then every object that excludes the optional field will be invalid according to that spec because it will have a nil value instead of omitting the key.

The alternative to this approach is to use something like cond-> as seen below:

(cond-> {::required-field :value}
  some-condition (assoc ::optional-field :optional-value))

This pattern is far from the only usage of cond->, but it appears fairly frequently. I see this pattern as detrimental because it requires the condition to be surfaced all the way at the top here, and there cannot be a way for some function used to produce :optional-value to abort via nil punning while still producing a valid value without requiring additional flow control and local bindings.

A more flexible solution to this problem uses if-some, but this solution is verbose and becomes unwieldy when multiple optional fields need to be included at once, as can be seen below.

(let [m {::required-field :value}
      m (if-some [v (produces-optional-value)]
          (assoc m ::optional-field v)
          m)
      m (if-some [v (produces-other-value)]
          (assoc m ::other-field v)
          m)]
  m)

One solution to this problem is the function assoc-some, which is like assoc, but when given a nil value it elides the key. This function is provided in medley, and sees some use.

The above unwieldy example would, with assoc-some, look like this:

(assoc-some {::required-field :value}
            ::optional-field (produces-optional-value)
            ::other-field (produces-other-value))

This is likely not the only solution to this problem, but I think it is a problem worth considering.

An example of another instance where the named solution wouldn't work, but that appears to follow more or less the same pattern is this code in farolero, which would be solved by a parallel function (that I have not yet found an implementation of in the wild besides my own) update-some which will update a key with a new value, and if that value is nil will remove the key.

subvec doesn't preserve metadata

Thu, 10 Feb 2022 15:36:55 +0000

The vector returned by subvec has not metadata even when the original vector has metadata.

Example:

(meta (subvec 
       (with-meta [0 1 2 3 4 5] {:foo 42})
       0))
;; nil

Expected value:

 {:foo 42}

Live demo here.

Somewhat confusing sorted-set behavior

Thu, 23 Dec 2021 12:22:35 +0000

Just ran into some interesting behavior with sorted-set. Namely, the contract to the user for a normal set is nil when the element doesn't exist. However, with sorted-set, the call will throw if the comparator doesn't gracefully handle the type. The reason makes sense (the comparator requires the ability to sanely compare elements), but the interface is somewhat confusing in the "no membership" case since we know that the element doesn't exist in the set if its type can't be compared by default.

In other words, I guess I'm proposing that a sane default behavior for a sorted-set using the default comparator should be nil rather than throw when the type cannot be compared, in order to allow nil-punning.

As an example, this case was encountered when the form was passed to a keyword function (:foobar form), which allows nil-punning on a lot of other collections. Here's a small snippet:

`
user> (def normal #{"0000"})

;; => #'user/normal

user> (def sorted-version (sorted-set "0000"))

;; => #'user/sorted-version

user> normal

;; => #{"0000"}

user> sorted-version

;; => #{"0000"}

user> (:foobar normal)

;; => nil

user> (:foobar sorted-version)

Execution error (ClassCastException) at user/eval246355 (form-init59552761865518221.clj:8123).
class java.lang.String cannot be cast to class clojure.lang.Keyword (java.lang.String is in module java.base of loader 'bootstrap'; clojure.lang.Keyword is in unnamed module of loader 'app')
`

I think my options for handling this are either try/catch or write a custom comparator, and both seem a little wonky :)

I'm probably missing something, but thought it was interesting anyway :)

Rest for map destructuring

Mon, 25 Oct 2021 10:04:49 +0000

I suggest adding :rest key for map destructuring syntax.
How do you think the following feature would be useful?

(let [m {:a 1 :b 2}
      {:keys [a] :rest r} m]
  (assert (= {:b 2} r)))

Or even like the rest of an array [a b & r]:

(let [m {:a 1 :b 2}
      {:keys [a]
       :& r} m]
  (assert (= {:b 2} r)))

Vector equiv() causes lazy seqs to be fully realized

Fri, 15 Oct 2021 10:12:53 +0000

The implementation of equiv() for PersistentVector involves calling .size() on the object to be compared, even when it is a lazy seq.

https://github.com/clojure/clojure/blob/b8132f92f3c3862aa6cdd8a72e4e74802a63f673/src/jvm/clojure/lang/APersistentVector.java#L98

As a result, a predicate such as (= [1] (range)) does not terminate.
More practically, comparing vectors against large partially-realised seqs may cause them to be fully evaluated, where one might reasonably expect = to short circuit to false on the first non-matching element.

Is this an intentional tradeoff for the more common use case where calling .size() on the second arg does not come with a performance cost?

Interestingly, this size comparison is not implemented for Clojure's persistent lists even though they are counted collections, i.e. (= '(1) (range)) terminates.

Thanks to @opqdonut on Clojurians Slack for helping to pinpoint the cause in the Java source.

Persistent collections can implement equiv() more efficiently

Thu, 30 Sep 2021 16:57:25 +0000

I found that structural equality between persistent collections makes very few assumptions which lead to inefficient implementations, especially for vectors and maps.

The thrust of the implementation is dispatching via methods which directly iterate over the underlying arrays.

These implementations aren't the prettiest or most idiomatic but they're efficient. If this gets implemented it would look different in Java anyway.

I tried these alternative implementations and found dramatic speed ups:

Vector

(let [die (clojure.lang.Reduced. false)]
  (defn vec-eq
    [^PersistentVector v ^Iterable y]
    (let [iy (.iterator y)]
      (.reduce v (fn [_ x] (if (= x (.next iy)) true die)) true))))

This works well when comparing vectors and for vector x list
Current implementation goes through a loop from 0 to count and calls nth for every element. nth calls arrayFor() every time, while both reduce and an iterator get the backing array once per array.

Map

(let [o (Object.)
      die (clojure.lang.Reduced. false)
      eq (fn [m2] (fn [b k v]
                   (let [v' (.valAt ^IPersistentMap m2 k o)]
                     (if (.equals o v')
                       die
                       (if (= v v') true die)))))]
  (defn map-eq
    [m1 m2]
    (.kvreduce ^IKVReduce m1 (eq m2) true)))

Here, too, the implementation iterates directly over the underlying array structure.
Current implementation casts the array to seq then iterates over it while getting entries from the other map via the Map interface.
This implementation avoids casting the map to a sequence and does not allocate entries.

Sequences

When the receiver is a list the object compared against it and the receiver will be cast to a seq.

It could be more efficient to compare it with other collections via an iterator

(defn iter-eq
  [^Iterable x ^Iterable y]
  (let [ix (.iterator x)
        iy (.iterator y)]
    (loop []
      (if (.hasNext ix)
        (if (= (.next ix) (.next iy))
          (recur)
          false)
        true))))

Benchmarking

With criterium, vec-eq wins both cases. There are diminishing returns with size increase but still at n=64 vec-eq is twice as fast as =.
map-eq is also 2-3x faster for bigger maps and up to 10x faster for smaller maps

(doseq [n [1 2 4 8 16 32 64]
        :let [v1 (vec (range n))
              v2 (vec (range n))]]
  (println 'iter-eq n (iter-eq v1 v2))
  (cc/quick-bench (iter-eq v1 v2))
  (println 'vec-eq n (vec-eq v1 v2))
  (cc/quick-bench (vec-eq v1 v2))
  (println '= n (= v1 v2))
  (cc/quick-bench (= v1 v2)))


(doseq [n [1 2 4 8 16 32 64]
        :let [v1 (vec (range n))
              v2 (list* (range n))]]
  (println 'iter-eq n (iter-eq v1 v2))
  (cc/quick-bench (iter-eq v1 v2))
  (println 'vec-eq n (vec-eq v1 v2))
  (cc/quick-bench (vec-eq v1 v2))
  (println '= n (= v1 v2))
  (cc/quick-bench (= v1 v2)))

(doseq [n [1 2 4 8 16 32 64]
        :let [m1 (zipmap (range n) (range n))
              m2 (zipmap (range n) (range n))]]
  (cc/quick-bench (map-eq m1 m2))
  (cc/quick-bench (= m1 m2)))

Addendum:
Also checked the following cases:

(doseq [n [10000 100000]
        :let [v1 (vec (range n))
              v2 (assoc v1 (dec (count v1)) 7)]]
  (cc/quick-bench (vec-eq v1 v2))
  (cc/quick-bench (iter-eq v1 v2))
  (cc/quick-bench (= v1 v2)))

(doseq [n [100000]
        :let [m1 (zipmap (range n) (range n))
              m2 (assoc m1 (key (last m1)) 7)]]
  (cc/quick-bench (map-eq m1 m2))
  (cc/quick-bench (= m1 m2)))

Optimized implementations still win by huge margins

get/find/assoc on vectors overflows key when passed large longs

Wed, 22 Sep 2021 17:19:46 +0000

Is this expected behavior?

clj
Clojure 1.10.3
user=> (get [1 2 42] 4294967295)
nil
user=> (get [1 2 42] 4294967296)
1
user=> (get [1 2 42] 4294967297)
2
user=> (get [1 2 42] 4294967298)
42
user=> (assoc [1 2 42] 4294967298 :wow)
[1 2 :wow]
user=> (find [1 2 42] 4294967298)
[4294967298 42]
user=> (.intValue 4294967298)
2

(Note this is with Clojure 1.10.3, but the syntax highlighting seems to truncate the last part).

AFAICT the get is related to this line in APersistentVector. The other two are caused by similar surrounding lines.

Would it be funner if clojure.set/difference worked better with sequences?

Wed, 25 Aug 2021 22:15:08 +0000

Thanks as ever for Clojure- we love it, and use it all day every day to solve worthy problems in ways that delight us as programmers and wow our users as well.

I was naively taking set differences, hoping for this, and was pleased:

> (clojure.set/difference #{:a :b :c} '(:a 1 2))
#{:c :b}

Then I did this, and was surprised:

> (clojure.set/difference #{:a :b :c} '(:a 1 2 3))
Execution error (IllegalArgumentException)
contains? not supported on type: clojure.lang.PersistentList

Then I read the implementation of clojure.set/difference, and I get it now:

https://github.com/clojure/clojure/blob/a29f9b/src/clj/clojure/set.clj#L49-62

But, I do wonder if it would be funner if this just worked. And since funner is probably a more important criteria to me than it is to you, I wonder what the real tradeoffs are here.

Is it performance?
Or, what would be the downside of implicitly converting to something that implements contains? at the critical moment?

Could we expose the data behind ranges?

Sat, 10 Jul 2021 12:45:09 +0000

dtype-next, libpython-clj, and tech.ml.dataset all assign special meaning to ranges but they often have to deconstruct them to do so. For libpython, we create actual python ranges in some cases. For dtype-next and friends, a range with increment of 1 may indicate a sub-buffer operation as opposed to an indexed-buffer operation -- sub-buffer retains the ability for System/arraycopy and the like to work correctly and is thus a major optimization in some cases.

I would like to be able to get the start, step, and end (when it exists) from any object created via clojure.core/range. I can currently do this via reflection or via subtraction of the first and second members of the sequence generated via the range along with count but I think it may be a safe and reasonable change to be able to query this information directly.

Most of this could be achieved by simply making the members start, step, and end members of LongRange and Range public as they are already final.

As a separate discussion it may be worth considering having finite ranges work like persistent vectors. I don't want to confuse the above point with this one but it sometimes more efficient to index into ranges via nth or an IFn invoke pathway than it is to use Clojure's sequence abstraction. Finite ranges are logically to me more like persistent vectors in nature and potentially they could behave precisely like persistent vectors including w/r/t hashcode, deriving from java.util.RandomAccess, and having conj produce either a new range or a persistent vector depending on if the value fits properly in the sequence.

bug in walker with array-maps

Sat, 27 Feb 2021 00:06:52 +0000

The walker doesn't handle array-maps properly, IMO. It converts them to regular maps and loses the ordering information.

> (def x (array-map :a 1 :b 2 :c 3 :d 4 :e 5 :f 6 :g 7 :h 8 :i 9 :j 10))
#'x
> (class x)
clojure.lang.PersistentArrayMap
> x
{:e 5, :g 7, :c 3, :j 10, :h 8, :b 2, :d 4, :f 6, :i 9, :a 1}
> (def xw (clojure.walk/postwalk identity x))
#'xw
> (class xw)
clojure.lang.PersistentHashMap
> xw
{:e 5, :g 7, :c 3, :j 10, :h 8, :b 2, :d 4, :f 6, :i 9, :a 1}

Why does (conj) returns vector but (conj nil 1) returns list and (conj nil) returns nil?

Sun, 17 Jan 2021 19:57:13 +0000

Why does (conj) returns vector but (conj nil 1) returns list and (conj nil) returns nil?

(conj)
;=> []
(conj nil)
;=> nil
(conj nil 1)
;=> (1)

It seems a little inconsistent no?

Couples of consecutive elements in a coll

Thu, 24 Dec 2020 13:22:59 +0000

What's the most idiomatic way of writing a function that, given a coll (1 2 3 4 5)

Returns

((1 2) (2 3) (3 4) (4 5))

i.e. consecutive couples of elements in a collection.

My naive solution would be:

(map (fn [a b] [a b]) a (rest a))

How do I update map elements conditionally?

Mon, 20 Jul 2020 05:59:14 +0000

I have some code that's massaging a data structure from one format to another (SVG hiccup attributes to CLJFX attributes). The format are quite similar but when an attribute is present I need to occasionally futz with it a bit. For instance, maybe I need to (read-string ..) a string into a number, or update a keyword so it's spelled a bit different, etc.

Right now my code is of the form

(-> svg-attributes
  (update :stroke #(case %
                     nil :black
                     "none" :black
                     (if (= \# %)
                       %
                       (keyword %))))
  (update :fill #(case %
                   nil "transparent"
                   "none" "transparent"
                   (if (= \# %)
                     %
                     (keyword %))))
  (update :points #(case %
                     nil []
                     [] []
                     (map read-string (-> %
                                          (clojure.string/split #"[ ,]")))))
  (update :stroke-width #(case %
                           nil 1.0
                           %))
  (update :stroke-dasharray #(case %
                               nil []
                               (map read-string (-> %
                                                    (clojure.string/split #"[ ,]")))))
  (clojure.set/rename-keys {:stroke-dasharray :stroke-dash-array})
  (update :font-size #(case %
                        nil 10
                        "none" 10
                        %))))

The issue is that I keep having to catch the nil case and jamming in some defaults... which works most of the time.. but I'd rather skip updating when there is nothing to update.

I saw this question, which is quite similar: https://ask.clojure.org/index.php/8387/how-to-avoid-nil-values-in-maps

Is the idiomatic solution to do a (cond-> with a (some? (:somekeyword %)) on each line ? I feel like I'm not reaching for the right tool here! Maybe someone can give me a better suggestion :)

Strange behavior of apply vector with a function as element

Sun, 26 Apr 2020 09:07:26 +0000

I don't understand why apply vector on a list doesn't return the same collection than vector or [], when at least one of the elements of the list is a function

(= [1 2 3] (apply vector '(1 2 3)))
=> true   
(= [int 2 3] (apply vector '(int 2 3)))
=> false
(= [int 2 3] (vector int 2 3))
=> true
(apply vector '(int 2 3))
=> [int 2 3]

Clojure 1.8.0 Java HotSpot(TM) 64-Bit Server VM 1.8.0_91-b14

Retrieving the domain/range of finite functions

Sun, 19 Apr 2020 18:32:29 +0000

One of the great features of Clojure is that it considers collections as functions which means that it is possible to invoke them

({:a 1, :b 2} :a) => 1
(#{:a :b} :a) => :a
([:a :b :c] 1) => :b

However it would be nice to be able to query the domain and range of those functions independently of their specific type.

The keys and vals functions currently implement this but only for Maps. I think it would be a good idea to make those functions more generic by accepting sets and vectors like in the following examples:

(keys  #{:a :b}) => (:a :b)
(vals #{:a :b}) => (:a :b)
(keys [:a :b :c]) => (0 1 2)
(vals [:a :b :c]) => (:a :b :c)

The specific use case I have in mind where this feature could be useful is to define a shorter syntax for explicit foreign key mapping in a data oriented DSL for relational data modeling, where #{:attr1 :attr2} could be used as a shortcut for {:attr1 :attr1, :attr2 :attr2}.

Clojure functions to implement the Map interface. Why or Why not?

Thu, 31 Oct 2019 17:13:37 +0000

From a theoretical standpoint, there really isn't a difference between a Map and a 1-arity function

What do you think of having a Function implement the Map interface? Will it be useful? If this was thought and you guys decided not to go with it, why not? Would love to learn the thinking

Should I prefer nil to empty collections?

Wed, 14 Aug 2019 09:12:35 +0000

This is a question about intersection of 3 different clojure aspects:

Collection functions treat nil as empty collection. Functions like filter, map, and even assoc happily accept nil for coll.
if and derived macros (when, and, or etc.) treat nil as falsey value.
empty? coerces it's input to seq, which leads to unnecessary allocations, and it's idiomatic to check for not-emptyness using seq instead of (not (empty? xs)). This is a common source of confusion, because (not (empty? xs))'s intent feels more clear then seq

With that said, I feel like consciously representing every empty collection in application as nil might be useful, because this:

(let [xs (get-non-empty-coll-or-nil)]
  (if xs
    (do-stuff-with xs)
    (do-nothing)))

...is more performant and clear than this:

(let [xs (get-possibly-empty-coll)]
  (if (seq xs)
    (do-stuff-with xs)
    (do-nothing)))

Two downsides I see to this approach:

having to use (fnil conj []) or (fnil conj #{}) instead of conj to ensure collections are vectors/sets, because conj-ing to nil creates a list, which I personally almost never use.
having to run all incoming collections on the boundaries of a system through not-empty.

What do you think?

= - doc perhaps not precise enough

Sun, 11 Aug 2019 08:57:06 +0000

looking at =

(defn ^boolean =
  "Equality. Returns true if x equals y, false if not. Compares
  numbers and collections in a type-independent manner.  Clojure's immutable data
  structures define -equiv (and thus =) as a value, not an identity,
  comparison."
  ([x] true)
  ([x y]
    (if (nil? x)
      (nil? y)
      (or (identical? x y)
        ^boolean (-equiv x y))))
  ([x y & more]
     (if (= x y)
       (if (next more)
         (recur y (first more) (next more))
         (= y (first more)))
       false)))

evaluating the following seams surprising to me:

(= [1] #{1})

because a vector is a collection and a set is also a collection, so when it says

Compares numbers and collections in a type-independent manner.

i think it should say:

Compares numbers, sequentials, maps and sets in a type-independent
manner.

peek does not support transient vectors

Tue, 08 Jan 2019 05:25:53 +0000

Appears to be an oversight, given that peek is a read operation.

Input:

(peek (transient [:a :b :c]))

Expected outcome:

:c

Actual outcome:

Unhandled java.lang.ClassCastException clojure.lang.PersistentVector$TransientVector cannot be cast to clojure.lang.IPersistentStack