defrecord should efficiently implement IReduceInit

Question

In one my the TMD pathways (that came from mastodon) we expand the dataset size using defrecords. Surprisingly considering everything else that performance case is doing, implementing IReduceInit spend up the timings considerably.

tech.v3.dataset.reductions-test> (require '[criterium.core :as crit])
nil
tech.v3.dataset.reductions-test> (defrecord YMC [year-month ^long count]
  ;; clojure.lang.IReduceInit
  ;; (reduce [this rfn init]
  ;;   (let [init (reduced-> rfn init
  ;;                  (clojure.lang.MapEntry/create :year-month year-month)
  ;;                  (clojure.lang.MapEntry/create :count count))]
  ;;     (if (and __extmap (not (reduced? init)))
  ;;       (reduce rfn init __extmap)
  ;;       init)))
  )
tech.v3.dataset.reductions_test.YMC
tech.v3.dataset.reductions-test> (let [yc (YMC. :a 1)]
                                   (crit/quick-bench (reduce (fn [acc v] v) nil yc)))
Evaluation count : 6729522 in 6 samples of 1121587 calls.
             Execution time mean : 87.375170 ns
    Execution time std-deviation : 0.173728 ns
   Execution time lower quantile : 87.104982 ns ( 2.5%)
   Execution time upper quantile : 87.550708 ns (97.5%)
                   Overhead used : 2.017589 ns
nil
tech.v3.dataset.reductions-test> (defrecord YMC [year-month ^long count]
   clojure.lang.IReduceInit
   (reduce [this rfn init]
     (let [init (reduced-> rfn init
                    (clojure.lang.MapEntry/create :year-month year-month)
                    (clojure.lang.MapEntry/create :count count))]
       (if (and __extmap (not (reduced? init)))
         (reduce rfn init __extmap)
         init)))
  )
tech.v3.dataset.reductions_test.YMC
tech.v3.dataset.reductions-test> (let [yc (YMC. :a 1)]
                                   (crit/quick-bench (reduce (fn [acc v] v) nil yc)))
Evaluation count : 43415358 in 6 samples of 7235893 calls.
             Execution time mean : 11.775423 ns
    Execution time std-deviation : 0.197683 ns
   Execution time lower quantile : 11.594695 ns ( 2.5%)
   Execution time upper quantile : 12.079668 ns (97.5%)
                   Overhead used : 2.017589 ns
nil
tech.v3.dataset.reductions-test> (defmacro reduced->
  [rfn acc & data]
  (reduce (fn [expr next-val]
            `(let [val# ~expr]
               (if (reduced? val#)
                 val#
                 (~rfn val# ~next-val))))
          acc
          data))

#'tech.v3.dataset.reductions-test/reduced->
tech.v3.dataset.reductions-test> 

Talking this over with other members it appears the value lookup pathway could also be optimized in the case when __extmap is not nil (it uses clojure.core/get as opposed to a direct getorDefault call.

• https://github.com/techascent/tech.ml.dataset/blob/master/test/tech/v3/dataset/reductions_test.clj#L197

Answer 1

That's interesting. I actually looked at this in the context of other reduce changes in 1.12.0-alpha1 but we didn't have any evidence that it was an issue. I'll make a ticket for this tomorrow.

Comments

Interesting!  I also spent a great deal of time making reduce absolutely as fast as possible with ham-fisted and dtype-next.  I profiled it heavily in nearly all situations I could and worked on some detailed things like making map and filter operations (and transducers) respect the runtime type of the map function or filter function so you can construct chains of primitive-typed ops and the entire chain will be primitive typed with no boxing in between.  

Odd coincidental bit a research.

Answer 2

Logged as https://clojure.atlassian.net/browse/CLJ-2750