`doseq` and `for` expands body twice

Wed, 26 Feb 2025 23:15:06 +0000

The bodies of doseq and for seem to be duplicated in their expansions.

$ clojure
user=> (defmacro a [] (prn :expand))
#'user/a
user=> (doseq [_ nil] (a))
:expand
:expand
nil
user=> (for [_ nil] (a))
:expand
:expand
()

I found the same problem in ClojureScript. I could not find an existing discussion about this so I'm not sure if this is by design, apologies if so. I'm aware that these macros have a large code footprint and that doseq cannot use closures (the usual way to prevent exponential expansion), but I missed this detail.

This leads to exponential code blowup if these forms are nested. Artificial demonstration:

$ clojure
Clojure 1.12.0
user=> (def counter (atom -1))
#'user/counter
user=> (defmacro a [] (prn :expand (swap! counter inc)))
#'user/a
user=> #(doseq [_ 1] (doseq [_ 2] (doseq [_ 3] (doseq [_ 4] (a)))))
:expand 0
:expand 1
:expand 2
:expand 3
:expand 4
:expand 5
:expand 6
:expand 7
:expand 8
:expand 9
:expand 10
:expand 11
:expand 12
:expand 13
:expand 14
:expand 15
#object[user$eval926$fn__927 0x76563d26 "user$eval926$fn__927@76563d26"]

This might be especially relevant to core.async, where placing go under doseq is idiomatic, since go is expensive in both expansion time and code size.

user=> (time (eval '(doseq [_ nil] (a/go (doseq [_ nil] (a/go (doseq [_ nil] (a/go))))))))
"Elapsed time: 837.230917 msecs"
nil

It also leads to duplicated reflection warnings from the fully expanded forms, which is how I found this problem.

user=> (doseq [_ nil] (Thread/sleep (identity 1)))
Reflection warning, NO_SOURCE_PATH:1:16 - call to static method sleep on java.lang.Thread can't be resolved (argument types: unknown).
Reflection warning, NO_SOURCE_PATH:1:16 - call to static method sleep on java.lang.Thread can't be resolved (argument types: unknown).
nil

I believe the exponential expansion for doseq was introduced in Clojure 1.1.0 with support for chunked seqs with this commit.

$ clojure -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.0.0"}}}' 
Downloading: org/clojure/clojure/1.0.0/clojure-1.0.0.pom from central
Downloading: org/clojure/clojure/1.0.0/clojure-1.0.0.jar from central
Clojure 1.0.0-
user=> (defmacro a [] (prn :expand))
#'user/a
user=> (doseq [_ nil] (a))
:expand
nil
user=> ^D
$ clojure -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.1.0"}}}'
Downloading: org/clojure/clojure/1.1.0/clojure-1.1.0.pom from central
Downloading: org/clojure/clojure/1.1.0/clojure-1.1.0.jar from central
Clojure 1.1.0
user=> (defmacro a [] (prn :expand))
#'user/a
user=> (doseq [_ nil] (a))
:expand
:expand
nil
user=>

At first I wasn't sure if this was hopeless, but I think doseq is fixable fusing the chunked and non-chunked cases like so:

(defmacro doseq
  "Repeatedly executes body (presumably for side-effects) with
  bindings and filtering as provided by \"for\".  Does not retain
  the head of the sequence. Returns nil.
  
  Unlike clojure.core/doseq, does not cause exponential macro expansion
  of expressions in bindings or body."
  [seq-exprs & body]
  (#'clojure.core/assert-args
     (vector? seq-exprs) "a vector for its binding"
     (even? (count seq-exprs)) "an even number of forms in binding vector")
  (let [step (fn step [recform exprs]
               (if-not exprs
                 [true `(do ~@body)]
                 (let [k (first exprs)
                       v (second exprs)]
                   (if (keyword? k)
                     (let [steppair (step recform (nnext exprs))
                           needrec (steppair 0)
                           subform (steppair 1)]
                       (cond
                         (= k :let) [needrec `(let ~v ~subform)]
                         (= k :while) [false `(when ~v
                                                ~subform
                                                ~@(when needrec [recform]))]
                         (= k :when) [false `(if ~v
                                               (do
                                                 ~subform
                                                 ~@(when needrec [recform]))
                                               ~recform)]))
                     (let [seq- (gensym "seq_")
                           chunk- (with-meta (gensym "chunk_")
                                             {:tag 'clojure.lang.IChunk})
                           count- (gensym "count_")
                           i- (gensym "i_")
                           in-chunk- (gensym "in-chunk_")
                           recform `(if ~in-chunk-
                                      (recur ~seq- ~chunk- ~count- (unchecked-inc ~i-))
                                      (recur (next ~seq-) nil 0 0))
                           steppair (step recform (nnext exprs))
                           needrec (steppair 0)
                           subform (steppair 1)]
                       [true
                        `(loop [~seq- (seq ~v), ~chunk- nil,
                                ~count- 0, ~i- 0]
                           (let [~in-chunk- (< ~i- ~count-)
                                 ~seq- (if ~in-chunk- ~seq- (seq ~seq-))]
                             (when (if ~in-chunk- true ~seq-)
                               (let [chunked?# (if ~in-chunk- false (chunked-seq? ~seq-))
                                     ~k (if ~in-chunk-
                                          (.nth ~chunk- ~i-)
                                          (if chunked?# nil (first ~seq-)))]
                                 (if (if ~in-chunk- false chunked?#)
                                   (let [c# (chunk-first ~seq-)]
                                     (recur (chunk-rest ~seq-) c#
                                            (int (count c#)) (int 0)))
                                   (do ~subform
                                       ~@(when needrec [recform])))))))])))))]
    (nth (step nil (seq seq-exprs)) 1)))

Missing documentation for `:while` modifier of `for` and `doseq`

Mon, 01 Jul 2024 13:56:08 +0000

I believe the for and doseq documentation (at least docstrings) fail to explain the semantics of the :while modifier. The existence is documented, but not the behavior.

I would like to request that the doctoring be updated, especially since it behaves in what I believe is a reasonable but counter-intuitive way. I believe a reasonable person would expect that :while false would trigger the for or doseq to exit; however it does not. Rather it causes the inner-most loop containing the :while to exit, but most notably the outer loops continue to iterate.

Another discussion of this can be found here: clojurians

Clojure Q&A - Recent questions tagged doseq

`doseq` and `for` expands body twice

Missing documentation for `:while` modifier of `for` and `doseq`