Clojure Q&A - Recent questions in data.int-map

Document that clojure.data.int-map is sorted

Fri, 17 Mar 2023 08:58:14 +0000

Recently a question about it was asked on Slack.

This DIMAP-4 Jira ticket has made the map and the set deliberately sorted, but it was never mentioned in the docs.
Seems like it could be explicitly mentioned in the docstrings?

Should data.int-map's allow assoc'ing non-integer keys?

Fri, 27 Jan 2023 03:03:54 +0000

Here's a question, in Clojure there are our typical HashMaps which allow assoc'ing any kind of keys on them, and then we have some more specialized kinds of maps for specific keys. Most of them are however also open to allow assoc'ing keys outside their usual range, gracefully reverting to hashmaps when the operation would break the data structure's invariants. Examples below:

Records are open to assoc, and revert to hashmaps when dissoc'ing a record field

(defrecord Person [name lang])
;; => clojure.data.int_map_test.Person
(let [p (->Person "john" "en_CA")]
  [p (assoc p :foo :bar) (dissoc p :lang)])
;; =>
;; [#clojure.data.int_map_test.Person{:name "john", :lang "en_CA"}
;; #clojure.data.int_map_test.Person{:name "john", :lang "en_CA", :foo :bar}
;; {:name "john"}]

On the other hand, struct-maps are open to assoc, but closed to dissoc

(let [sm (struct-map (create-struct :name :lang) :name "John" :lang "en_CA")]
  [(assoc sm :foo "bar")
   (try (dissoc sm :lang) (catch Exception e (.toString e)))])
    => [{:name "John", :lang "en_CA", :foo "bar"} 
        "java.lang.RuntimeException: Can't remove struct key"]

An other example is ArrayMaps which eventually turn into HashMaps as they grow. That's not exactly the same, there's no specific key outside the map's range, but similar in a more general sense of, when an operation either violates invariants or performance expectations, fall back to the general HashMap.

That makes me wonder, how do people feel about int-map/int-set which instead throw exception if you try and add a non-integer key?

(assoc (i/int-map 1 2) :a "b")
ClassCastException class clojure.lang.Keyword cannot be cast to class java.lang.Number

Should it be modified to fall back to a hashmap instead? It seems like that is more in line with Clojure philosophy of open data structures.

It could be implemented similar to PersistentStructMap, which holds an extra map in which it stores extra key/value pairs which are not part of the struct's definition:

final IPersistentMap ext;

The same thing could be done in PersistentIntMap

(deftype PersistentIntMap
  [^INode root
   ^long epoch
   meta
   ^IPersistentMap ext]
  ;; ...

What do you think?

Also, the same question but for int-sets

clojure.data.int-map Branch nodes allocate unused ArrayLists

Mon, 16 Jan 2023 07:33:46 +0000

I see here https://github.com/clojure/data.int-map/blob/master/src/main/java/clojure/data/int_map/Nodes.java#L376

There is an unused List<INode> above = new ArrayList<INode>(); being created, it should probably be removed.

clojure.data.int-map Branch nodes could use bitmaps to save space

Mon, 16 Jan 2023 06:31:31 +0000

In clojure.data.int-map contrib library, the INode impelementation Branch stores its children always in an INode[] of length 16, even though often/usually, not all 16 of the children's spots are taken up. For instance, the representation of

(int-map 0 0 1 0)

Will be represented with a Branch node with an INode[] where the 0-position contains 0->0, the 1-position contains 1->0, and the other 14 spaces in the array are null, see below:

This is a difference to the core hashmap/hashset implementation which uses a (in their case, 32-bit, but the branching factor in int-map is only 16) bitmap to keep track of which positions in the array are present and then at read time you can see how many bits are present (set to 1) in the bitmap, to the right of the index you want to read, and that tells you what the true index in the compressed array is.

It would probably be worth taking a look at if use of bitmaps could positively effect performance and memory use of int-map's here.

Could also try compressing further by combining the bitmap and/or mask and/or prefix and/or offset because these fields are large enough that they are not using all of their bits. mask (long), for instance, is only ever one of 15, 15 << 4, 15 << 8, 15 << 12, etc. offset (int) is only ever 0 ... 60 I think. And prefix (long) similarly is only storing a binary representation of a bitmask between 0...64 in length so in principle could be compressed quite a bit too. Among these I bet there would be a field to store some 16 bits of a bitmask without having to add an other field.

clojure.data.int-map could use shared empty singleton instances

Mon, 16 Jan 2023 05:05:11 +0000

Shared empty int-map

I notice that in the contrib library clojure.data.int-map, a new instance of the empty PersistentIntMap is created every time (int-map) is called. This is different to how all the other persistent collections work, where there's only one single empty list/vector/map/set exists, and it is reused.

It's also the same situation with PersistentIntSet, both dense and non-dense.

(empty my-int-map) doesn't preserve meta

the IPersistentCollection#empty() implementation on PersistentIntMap is:

  (empty [this]
      (PersistentIntMap. Nodes$Empty/EMPTY 0 nil))

So the metadata from this is lost in the returned empty int-map. Other strict collection implementations retain metadata in the returned collection:

(mapv (fn [coll] (meta (empty (with-meta coll {:a 1})))) 
    [() [] {} #{} (sorted-map) (sorted-set)])
=> [{:a 1} {:a 1} {:a 1} {:a 1} {:a 1} {:a 1}]

The same is true for PersistentIntSet

`intersection` and `union` do not support `nil` punning

Tue, 31 Aug 2021 16:51:39 +0000

This is for org.clojure/data.int-map v 1.0.0.
At the same time clojure.set/intersection and clojure.set/union handle the case well.

(ns sandbox
  (:require
   [clojure.data.int-map :as int-map]
   [clojure.set]))

(clojure.set/intersection #{1 3} #{1 2 3}) ; => #{1 3}
(clojure.set/intersection #{1 3} nil) ; => nil
(clojure.set/intersection nil #{1 2 3}) ; => nil

(int-map/intersection (int-map/int-set #{1 3}) (int-map/int-set #{1 2 3})) ; => #{1 3}
(int-map/intersection (int-map/int-set #{1 3}) nil) ; => NullPointerException
(int-map/intersection nil (int-map/int-set #{1 2 3})) ; => NullPointerException

(clojure.set/union #{1 3} #{2}) ; => #{1 3 2}
(clojure.set/union #{1 3} nil) ; => #{1 3}
(clojure.set/union nil #{2}) ; => #{2}

(int-map/union (int-map/int-set #{1 3}) (int-map/int-set #{2})) ; => #{1 2 3}
(int-map/union (int-map/int-set #{1 3}) nil) ; => NullPointerException
(int-map/union nil (int-map/int-set #{2})) ; => NullPointerException

Why does PersistentPriorityMap kv-reduce call the reducing function with item and priority?

Mon, 01 Feb 2021 16:45:15 +0000

Why does reduce-kv on a priority-map call the reducing function with item as the key and priority as the value (source) instead of the map's key and value. Calling reduce will be passed the map's key and value. I'm assuming it has something to do with this particular data structure's origins, but I cannot find any documentation pointing me to in in the clojure.data.priority-map ns. Here is an example.

(def m
  (clojure.data.priority-map/priority-map-keyfn-by first compare
    :a [2 :apples] :b [1 :bananas]))

(reduce-kv
  (fn [acc k v]
    (assoc acc k v))
  {} m)
=> {:b 1, :a 2}

(reduce
  (fn [acc [k v]]
    (assoc acc k v))
  {} m)
=> {:b [1 :bananas], :a [2 :apples]}

Is this the expected behavior? If so, could we add some documentation that this is how reduce-kv on a PersistentPriorityMap should work?

int sets return false for .equals and = when comparing against java.util.Set, and similarly for int maps against java.ut

Wed, 09 May 2018 03:51:15 +0000

Most implementations of Clojure immutable maps and sets can return true when being compared against java.util.Map and java.util.Set instances (even mutable ones) if they currently have the same contents.

The proposed patch to enable this for int sets and int maps has a test case that demonstrates this is not true for the latest library code.

I would expect to get true for all of the .equals and = calls in the sample REPL session below.

`
user=> (def jset1 (java.util.HashSet. [1]))

'user/jset1

user=> (def intset1 (imap/int-set [1]))

'user/intset1

user=> (def cset1 #{1})

'user/cset1

user=> (.equals cset1 jset1)
true
user=> (= cset1 jset1)
true
user=> (.equals jset1 cset1)
true
user=> (= jset1 cset1)
true
user=> (.equals intset1 jset1)
false
user=> (= intset1 jset1)
false
user=> (.equals jset1 intset1)
true
user=> (= jset1 intset1)
false
`

PersistentIntMap equals/equiv gives wrong result

Wed, 27 Sep 2017 11:12:42 +0000

{{PersistentIntMap}} {{equals}} and {{equiv}} are currently broken for maps that have the same keys, but with different values:

`
clojure.data.int-map> (= (int-map 1 :foo)

                     (int-map 1 :bar))

;; => true
clojure.data.int-map> (.equals (int-map 1 :foo)

                           (int-map 1 :bar))

;; => true
`

The bug was introduced in (link: https://github.com/clojure/data.int-map/commit/fccd5e4bd1dc42b8ea50c80081e785ea68f36dd6 text: this commit) addressing (link: https://dev.clojure.org/jira/browse/DIMAP-11 text: DIMAP-11).

Cause/fix is simple, see attached patch.

support element lookup on transient sets

Mon, 26 Jun 2017 07:41:18 +0000

This is related to https://dev.clojure.org/jira/browse/CLJ-700

contains? doesnt work on transient collections

Now of course the expectation of how data.int-map should work is the same as with a general set so I dont complain there. However, a workaround shown in this thread: https://groups.google.com/forum/#!topic/clojure/lQVmZ-jcdiU is to use the set directly as a function:

;; this works
((transient #{1 2 3}) 2)
;=> 2

;; this doesnt :(
((transient (int-set #{1 2 3})) 2)
;=> CompilerException java.lang.IllegalArgumentException: contains? not supported on type: clojure.data.int_map.TransientIntSet

I checked the code and the TransientIntSet supports the contain method of the TransientIntSet interface which could be used directly and return the correct value but it is definitely not idiomatic to recur to interop nor have to look that around for this specific case.

Cannot merge int-sets of different density.

Wed, 21 Sep 2016 12:09:08 +0000

`
(iset/union (iset/int-set)

        (iset/dense-int-set))

throws java.lang.IllegalArgumentException: Cannot merge int-sets of different density.

Now this might not be a bug per se, but the README on GitHub claims that

bq. dense-int-set behaves the same as int-set, the difference is only in their memory efficiency.

I think it should either be possible to mix dense/normal sets, or state explicitly in the documentation that it isn't and also possibly update the error message since it wasn't obvious to me that this is the cause (I had totally forgotten I had used dense sets in this one place and believed the problem to be caused by transients).

Unlike regular transient sets, data.int-map transient sets do not support the IFn interface

Fri, 01 Jul 2016 09:22:04 +0000

=> ((transient (clojure.data.int-map/dense-int-set 1)) 1)
IllegalArgumentException Don't know how to create ISeq from: java.lang.Long clojure.lang.RT.seqFrom (RT.java:545)

=> ((transient #{1}) 1)
1

Union, intersection and difference have only 2 arguments versions

Sun, 01 Nov 2015 01:28:14 +0000

The arities of the special union, intersection and difference operations don't match the arities of those in clojure.set.

|operation |clojure.set|int-set|
| :-- | :-- | :-- | :-- |
|union | 0, 1, 2, n | 2 |
|intersection | 1, 2, n | 2 |
|difference | 1, 2, n | 2 |

This prevents using int-sets as a drop-in replacement for sets.

Looking at the source of clojure.set, those seem to be implemented via simple reduce for difference, and bubbled-sets (via bubble-max-key) for union and intersection.

Could a similar approach be used here? I'm willing to work on this.

Minor typos in README and docstring

Wed, 30 Sep 2015 13:27:08 +0000

Fixes two very minor typos.

int-map doc string is overly restrictive?

Tue, 29 Sep 2015 23:44:23 +0000

Doc string for int-map is "Creates an integer map that can only have non-negative integers as keys.", but it seems that the implementation may work fine for any integer in the range of a Java Long, positive or negative?

Proper seq on int-sets

Tue, 28 Oct 2014 15:48:48 +0000

Empty intersection of int-sets return () from clojure.lang.Seqable.seq().

user> (dorun (map #(println "set contains:" %) (intersection (int-set [1]) (int-set [2])))) set contains: nil nil

Support efficient rseq on int maps

Sat, 20 Sep 2014 16:16:55 +0000

As discussed previously in https://github.com/ztellman/immutable-int-map/pull/1

I note that the existing Seq implementation is not lazy, so i've just implemented a basic reverse iterator around this rather than walking the tree right-first - I was actually fairly surprised core clojure doesn't provide Reversible for ArrayLists already.

Patch taken from https://github.com/glenjamin/data.int-map/tree/rseq attached, benchmarks below:

user=> (require '[clojure.data.benchmark :refer (entries)]
  #_=>          '[clojure.data.int-map :as i]
  #_=>          '[criterium.core :as c])
nil
user=>

user=> (let [m (into (i/int-map) entries)]
  #_=>   (println "seq")
  #_=>   (c/quick-bench (dorun (seq m)))
  #_=>   (println "rseq")
  #_=>   (c/quick-bench (dorun (rseq m))))
seq
WARNING: Final GC required 8.9263203426305 % of runtime
WARNING: Final GC required 48.03202811726098 % of runtime
Evaluation count : 12 in 6 samples of 2 calls.
             Execution time mean : 75.794581 ms
    Execution time std-deviation : 421.571821 µs
   Execution time lower quantile : 75.476998 ms ( 2.5%)
   Execution time upper quantile : 76.483561 ms (97.5%)
                   Overhead used : 2.127421 ns

Found 1 outliers in 6 samples (16.6667 %)
low-severe 1 (16.6667 %)
Variance from outliers : 13.8889 % Variance is moderately inflated by outliers
rseq
WARNING: Final GC required 54.64906330819225 % of runtime
Evaluation count : 6 in 6 samples of 1 calls.
             Execution time mean : 101.840831 ms
    Execution time std-deviation : 723.601456 µs
   Execution time lower quantile : 100.881998 ms ( 2.5%)
   Execution time upper quantile : 102.496873 ms (97.5%)
                   Overhead used : 2.127421 ns

CLA has been signed.