folding over hash-map nested hash-map throws exception

Question

I got a baffling exception in a recursive function that folds. REPL transcript below:


nREPL server started on port 57818 on host 127.0.0.1 - nrepl://127.0.0.1:57818
REPL-y 0.3.5, nREPL 0.2.6
Clojure 1.7.0-alpha5
Java HotSpot(TM) 64-Bit Server VM 1.7.0_76-b13
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

user=> (use 'foldtest.core)
nil
user=> (source leafs)
(defn leafs [xs]
  (->> (r/mapcat (fn [k v]
                   (if (map? v)
                     (leafs v)
                     [[k v]])) xs)
       (r/foldcat)))
nil
user=> (leafs (hash-map :a (hash-map :b 1 :c 2)))

ClassCastException clojure.lang.PersistentHashMap$1 cannot be cast to clojure.lang.IFn  clojure.core.reducers/fjinvoke (reducers.clj:48)
user=> (pst)
ClassCastException clojure.lang.PersistentHashMap$1 cannot be cast to clojure.lang.IFn
    clojure.core.reducers/fjinvoke (reducers.clj:48)
    clojure.lang.PersistentHashMap.fold (PersistentHashMap.java:207)
    clojure.core.reducers/eval1347/fn--1348 (reducers.clj:367)
    clojure.core.reducers/eval1220/fn--1221/G--1211--1232 (reducers.clj:81)
    clojure.core.reducers/folder/reify--1247 (reducers.clj:130)
    clojure.core.reducers/fold (reducers.clj:98)
    clojure.core.reducers/fold (reducers.clj:96)
    clojure.core.reducers/foldcat (reducers.clj:318)
    foldtest.core/leafs (core.clj:5)
    foldtest.core/leafs/fn--1367 (core.clj:7)
    clojure.core.reducers/mapcat/fn--1277/fn--1280 (reducers.clj:185)
    clojure.lang.PersistentHashMap$NodeSeq.kvreduce (PersistentHashMap.java:1127)
nil
user=>


Note that it *must* be a hash-map nested in a hash-map. Other combinations of array and hash maps seem fine:


user=> (leafs (array-map :a (hash-map :b 1 :c 2)))
[[:c 2] [:b 1]]
user=> (leafs (hash-map :a (array-map :b 1 :c 2)))
[[:b 1] [:c 2]]
user=> (leafs (hash-map :a (hash-map :b 1 :c 2)))

ClassCastException clojure.lang.PersistentHashMap$1 cannot be cast to clojure.lang.IFn  clojure.core.reducers/fjinvoke (reducers.clj:48)
user=> (leafs (array-map :a (array-map :b 1 :c 2)))
[[:b 1] [:c 2]]
user=>


Possibly related: CLJCLR-63

It took me a while to discover this because of this inconsistency (which I am not sure is a bug):


user=> (def a {:a 1})
#'user/a
user=> (type a)
clojure.lang.PersistentHashMap
user=> (let [a {:a 1}] (type a))
clojure.lang.PersistentArrayMap
user=> (type {:a 1})
clojure.lang.PersistentArrayMap
user=>


(I had put test input in a def, but using the defed var always failed but literals always worked!)

Answer 1

Comment made by: favila

I confirmed this issue still exists in CLJ 1.8 and master (1.9) with Java 8.

I was able to diagnose the cause. The problem is that the {{.fold}} method of PHMs create a {{Callable}} (not an {{IFn}}) for {{fjinvoke}} to invoke, but {{fjinvoke}} will call its argument as a normal clojure function ({{IFn .invoke}}). This is the only Clojure-core foldable data structure that provides a raw {{Callable}} to {{fjinvoke}}. Normally this problem is not hit because the task isn't invoked within a running task, but a nested fold executes inside an already-running task and will reach the {{(f)}} line in {{fjinvoke}} and blow up.

Since Clojure functions are also {{Callable}} anyway, my fix is to tighten the contract of {{fjinvoke}} to receive a {{Callable}} and {{.call}} the argument instead of invoking it. An alternative fix would be to change PHM's {{.fold}} to create something more Clojure-function like (both invokeable and callable) instead.

Answer 2

Reference: https://clojure.atlassian.net/browse/CLJ-1662 (reported by favila)