Should the threading macros handle lambdas?

Question

I recently thought that I ran into a bug in clojure.spec because it threw an exception when I put a lambda like (fn [x] x) or #(identity %) into the -> threading macro without wrapping it first in an extra parens like ((fn [x] x)). This is the most common repeated "mistake" I've made in Clojure over the years, and I've heard from others that it's a common misconception about the threading macros. This "mistake" in my case is based on my assumption that the lambda is a fundamental "thing" or value in the language. But, in fact, Clojure is just data, mostly macros, and each macro handles its input data in different ways.

The -> threading macro treats a lambda as a list instead of as a fundamental "function value." When one does (-> :hmm (fn [x] x)) it is expanded like: (fn :hmm [x] x) which obviously is not what is intended. However, it does have a special convenience case for a non-list thing like a symbol, keyword, etc, which it assumes is something that can behave as a function with one argument, and wraps it in parens before threading. This special case adds value because it interprets the intention of the user and avoids forcing them to add parens when they're unnecessary.

I assert that the basic lambda forms (fn) and #() should and easily can be handled similarly to a symbol, by interpreting the intention of the user. The value from doing so would make the threading macros simpler and more intuitive to use, would remove an unnecessary exception that affects numerous users, and would advance the language towards treating the lambda forms as core "values." This last point is a philosophical one, but important in my opinion for making the language "just work" the way a new or non-expert user expects when getting stuff done.

The existing special case of wrapping a non-list form already demonstrates the value of providing some interpretation of the intention of the user, but there now exists an inconsistency between this special case, and the absence of it for two of the most common forms of arguably one of the most foundational "values" in the language, the function itself.

Have you been bitten by this "mistake"?

If you'd like to try it side by side with the existing -> macro, here's a version called t->:

(defmacro t->
  "Threads the expr through the forms. Inserts x as the
  second item in the first form, making a list of it if it is a lambda or not a
  list already. If there are more forms, inserts the first form as the
  second item in second form, etc."
  {:added "1.0"}
  [x & forms]
  (loop [x x, forms forms]
    (if forms
      (let [form (first forms)
            threaded (if (and (seq? form) (not (#{'fn 'fn*} (first form))))
                       (with-meta `(~(first form) ~x ~@(next form)) (meta form))
                       (list form x))]
        (recur threaded (next forms)))
      x)))

I've prepared a patch for -> and ->> that handles this use case and has no breaking impact that I can conceive of, keeping the existing behavior, while adding the new. It passes all of the core Clojure tests, but I have not yet tested it on outside code beyond my own. That would obviously be necessary before it could be considered a possible change in core.

From 686831062a574486413022af31e8c7a07b78cd24 Mon Sep 17 00:00:00 2001
From: Thomas Spellman <thos37@gmail.com>
Date: Mon, 13 Jan 2020 20:39:45 -0800
Subject: [PATCH] thread macros

---
 src/clj/clojure/core.clj             | 12 ++++++------
 test/clojure/test_clojure/macros.clj | 12 ++++++++++++
 2 files changed, 18 insertions(+), 6 deletions(-)

diff --git a/src/clj/clojure/core.clj b/src/clj/clojure/core.clj
index 8e98e072..fe43289b 100644
--- a/src/clj/clojure/core.clj
+++ b/src/clj/clojure/core.clj
@@ -1670,42 +1670,42 @@
   (. (. System (getProperties)) (get \"os.name\"))
 
   but is easier to write, read, and understand."
   {:added "1.0"}
   ([x form] `(. ~x ~form))
   ([x form & more] `(.. (. ~x ~form) ~@more)))
 
 (defmacro ->
-  "Threads the expr through the forms. Inserts x as the
-  second item in the first form, making a list of it if it is not a
+  "Threads the expr through the forms. Inserts x as the second item
+  in the first form, making a list of it if it is a lambda or not a
   list already. If there are more forms, inserts the first form as the
   second item in second form, etc."
   {:added "1.0"}
   [x & forms]
   (loop [x x, forms forms]
     (if forms
       (let [form (first forms)
-            threaded (if (seq? form)
+            threaded (if (and (seq? form) (not (#{'fn 'fn*} (first form))))
                        (with-meta `(~(first form) ~x ~@(next form)) (meta form))
                        (list form x))]
         (recur threaded (next forms)))
       x)))
 
 (defmacro ->>
-  "Threads the expr through the forms. Inserts x as the
-  last item in the first form, making a list of it if it is not a
+  "Threads the expr through the forms. Inserts x as the last item
+  in the first form, making a list of it if it is a lambda or not a
   list already. If there are more forms, inserts the first form as the
   last item in second form, etc."
   {:added "1.1"}
   [x & forms]
   (loop [x x, forms forms]
     (if forms
       (let [form (first forms)
-            threaded (if (seq? form)
+            threaded (if (and (seq? form) (not (#{'fn 'fn*} (first form))))
               (with-meta `(~(first form) ~@(next form)  ~x) (meta form))
               (list form x))]
         (recur threaded (next forms)))
       x)))
 
 (def map)
 
 (defn ^:private check-valid-options
diff --git a/test/clojure/test_clojure/macros.clj b/test/clojure/test_clojure/macros.clj
index ce17bb38..9fb1fa9e 100644
--- a/test/clojure/test_clojure/macros.clj
+++ b/test/clojure/test_clojure/macros.clj
@@ -106,8 +106,20 @@
   (is (nil? (loop []
               (as-> 0 x
                 (when-not (zero? x)
                   (recur))))))
   (is (nil? (loop [x nil] (some-> x recur))))
   (is (nil? (loop [x nil] (some->> x recur))))
   (is (= 0 (loop [x 0] (cond-> x false recur))))
   (is (= 0 (loop [x 0] (cond->> x false recur)))))
+
+(deftest ->lambda-test
+  (is (= 'a (-> 'a ((fn [x] x)))))
+  (is (= 'a (-> 'a (fn [x] x))))
+  (is (= 'a (-> 'a #(identity %))))
+  (is (= 'a (-> 'a (#(identity %))))))
+
+(deftest ->>lambda-test
+  (is (= 'a (->> 'a ((fn [x] x)))))
+  (is (= 'a (->> 'a (fn [x] x))))
+  (is (= 'a (->> 'a #(identity %))))
+  (is (= 'a (->> 'a (#(identity %))))))
-- 
2.21.0 (Apple Git-122.2)

Logged: https://clojure.atlassian.net/browse/CLJ-2553

Answer 1

This is a list.

(fn [x] x)

This is a reader macro that evaluates to an anonymous function form, also a list:

#(identity %)

;;becomes
(fn* [p1__152#] (identity p1__152#))

The expansion for #() happens at read time, so it's transforming structure of the code (the form following #) into an s-expression. So, it's like a shortcut (e.g. syntactic sugar) that expands into a common Clojure form.

->

Is a macro that operates on s-expressions. Its purpose is to restructure s-expressions so that we can write more linear, pipeline versions of the typical inside-out nested s-expression flow for complex expressions. This is very handy in general.

Your proposal changes the fundamental behavior of -> since it now no longer operates on mere structural forms (aside from elevating non-list forms to singleton lists), we have new implicit rules. In fact, there are many times where I "want" to structurally modify lists (including ones that define function forms like (fn [] blah) directly, e.g. when writing macros or other meta programming. It's both more general and simpler to restrict the behavior of -> to purely structural modification of lists, and leave any special higher rules up to the caller (e.g. you need to wrap functions in an extra () if you want them to be invoked, or come up with another means to do so).

The good news is....it's not hard to deviate from -> and write your own variant for your own needs. This is the perfect use case for a macro in a library. Just like there are variants of -> that include some notion of continuity/failure, like some->, you could easily define your own lift-functions-> macro or something that would implement the semantics you described. This solution clears up the semantics for your use (and possibly others if they opt in) while retaining the simplicity and generality of the original macro. I think this is a great example of leveraging macros to extend the language to suit your needs.

I do not think the proposed semantic gain is worth the deviation in consistency and simplicity, since you're applying more magic to infer what the expansions of reader macros and fn forms mean. I'd rather just manipulate lists, and where necessary, write macros to implement custom semantics. I think the existing -> and ->> style accomplishes list manipulation in the simplest, least-magic way possible.

Answer 2

-> already supports "lambdas", but it's not via fn, it is via as->

(-> 42
    inc
    (fn [x] (/ x x)) ;; << will not work
    inc)

Simple switch from fn to as->, and it will work

(-> 42
    inc
    (as-> x (/ x x)) ;; switch from fn to as->, remove extra []
    inc)

I think that is great to have two simple macros that operate together. It is better then a great macro that handle all possible cases.

Comments

Yeah, that's pretty sweet. `as->` is one of my favorite. Certainly is handy in some situations.

I'd argue though that two macros is not better than one, if simply fixing one would make the other superfluous in most common situations.

```
(-> 42 inc #(/ % %) inc)
```

That's what new-comers to the language expect to work. They think, "Yeah, that's gotta work... there's no way Clojure would be letting me thread into the _actual_ lambda definition, so this probably makes sense to the compiler."

Then they get bit and think, "Oh, well there's probably some good reason why that's not possible."

But after a few years, they start to realize that there actually is no good reason why they couldn't use the simpler form.

---

Nice, this would also work with `doto` which I often _almost_ use and then abandon because of similar issues.

Answer 3

Would you post an example of the thing that did not work as you expected? Anonymous functions can be very useful with -> and ->>.

The threading macros "thread" a running result through a bunch of forms at a certain spot, first argument or last argument. So, to start with an example that involves a function represented by a symbol:

user=> (-> 42 (+ 1))
43

We can see exactly how it works, by looking at the macro expansion:

user=> (macroexpand-1 '(-> 42 (+ 1)))
(+ 42 1)

Going back to the threading, we can use (fn...) in place of + by the rule of referential transparency:

user=> (-> 42 ((fn [a b] (+ a b)) 1))
43

This example makes -> thread the running value into the invocation of the anonymous function, which we can see more clearly in the macro expansion:

user=> (macroexpand-1 '(-> 42 ((fn plus [a b] (+ a b)) 1)))
((fn plus [a b] (+ a b)) 42 1)

I find one particular idiom with anonymous functions inside -> or ->> very useful, now and then: to log the running value. I can insert something like the following between two steps:

...
            ((fn [x] (println "running value:" x) x)) 
...

Comments

Hi @pbwolf, thanks for your response.  I've updated my original post with more clarification.  I agree with all your ideas of why anonymous functions (what I call lambdas) are useful and important within the threading macros.  My point is simply that it should be unnecessary to have to double wrap them in parens for a lambda with one argument.  For example, one of your examples:

    (-> 42 ((fn [a b] (+ a b)) 1))

That takes 2 arguments and so would be necessary to wrap in parens.  However, regarding your second example:

    (-> "hello" ((fn [x] (println "running value:" x) x)))

My suggestion is that with a small modification to the -> macro, you could simply do:

    (t-> "hello" (fn [x] (println "running value:" x) x))

Another example:

    (t-> {:some :data} #(assoc % :some-other :data))

In other words it makes working with the threading macro more intuitive and enjoyable and eliminates a common error caused by a common "mistake" that I think should not be treated as a mistake, but rather as a reasonable use case.

Please try it out and see what you think:

    (defmacro t->
      "Threads the expr through the forms. Inserts x as the second item
      in the first form, making a list of it if it is a lambda or not a
      list already. If there are more forms, inserts the first form as the
      second item in second form, etc."
      {:added "1.0"}
      [x & forms]
      (loop [x x, forms forms]
        (if forms
          (let [form (first forms)
                threaded (if (and (seq? form) (not (#{'fn 'fn*} (first form))))
                           (with-meta `(~(first form) ~x ~@(next form)) (meta form))
                           (list form x))]
            (recur threaded (next forms)))
          x)))

---

I myself have delusionally put #(...) into a threading form.  I have some sympathy for the cause.  But, by the light of day, I even more strongly cherish the (objective) simplicity of threading macros that apply the same pattern to every form regardless of its contents.  Also, someday, I will make my own "enhanced" fn form, and I will get offended if the core threading macros demote it to second class by favoring core's fn and fn* only.

---

I think it's not delusional to expect that a core macro would have a special case for two core function forms of arguably the most core thing -> the function.  It's funny that you're protective of your hypothetical future possibility of being resentful of your theoretical non-core enhanced fn being deprived of a core special case!  That's great!  But really, when you're that much of a power coder, I'm sure you'll already have an enhanced threading macro to handle it too!  

My main point is that this change actually makes the use of the language **simpler**, especially for new-comers and non-experts, and it deprives no one of something they already have, it breaks nothing, and it only *adds value!*

---

> My main point is that this change actually makes the use of the language **simpler**,

I disagree, it may make it easier or more convenient in most situations, but this is a more complex threading macro, and not simpler. To my point, which of the two can be used to implement the other? The answer reveals which one is simpler.

Having said that, making it more convenient in this case could be a good thing, and warrant the additional complexity. I doubt core would embrace it though, so I'd just roll your own if I was you.

---

Didier A. wrote: > I disagree, it may make it easier or more convenient in most situations, but this is a more complex threading macro, and not simpler

We are referring to two different kinds of "simple" and "easy."  I'm referring to simpler usage and you're referring to simpler implementation.  I'm aiming for simpler (and easier in my opinion) usage, which makes the implementation more complex.  You're aiming for simpler implementation which makes usage harder and more complex.

I appreciate that you think it might be worth it to add a bit of complexity to make usage simpler.  I agree that it's a grey area.  I think that each case should be analyzed individually, and in this particular case it's worth doing.  However, I think the biggest down side to making this change would be a lack of backwards compatibility with older versions of Clojure.  But is this a significant down-side?

Answer 4

I actually love this idea, so I added it to my injest library.

Looking back, I can remember feeling this pain and wishing it was more ergonomic wrt anonymous functions. Then I developed muscle memory and forgot I felt the pain. There was a lot of new ideas around threads going around when they first came out - thread libraries and even joke libraries about them. I shied away from most of them though, unsure of their longevity, wrt code maintainability.

It's been a few years now though, they're less shiny and we all have a lot more experience with them now. We have better intuitions of what works and what doesn't and how we'd like them to behave. We know what can be added to them and how their associated limitations will affect code usability over time.

At this point, I think I can confidently say that wrapping lambdas will produce no significant negative consequences for code maintainability over time, and won't limit future semantic growth for the language in any desirable direction.

Additionally, lambda wrapping has the additional benefit of conveying to the reader that the author intends for the anonymous function to only take one parameter. In the classical thread syntax, the reader would have to scan all the way to the end of (#(... in order to know if an extra parameter is being passed in. It also prevents people from creating unmaintainable abstractions involving the threading of values into a literal lambda definition, which I would rather not have to maintain.

That being said, -> and ->> are such core constructs that adding semantics to them should be done very carefully and with intense scrutiny. I'd imagine it'd be easier for the core Clojure team to evaluate such a change if folks in the community at large had the opportunity to kick the tires on the new semantics for a while. injest provides +> and +>> that allow you to do lookups into maps, indexing into sequences and wrap lambdas, like:
(+> {"init" 10} "init" range rest 2 #(- 10 % 1)) ;=> 6
I feel confident at this point that all these additional behaviors are, in fact, orthogonally simplistic semantics, in the sense that they don't complect with each other, past features or future features. But again, one of Clojure's greatest features is the core team's refusal to add complecting features - sometimes adding less worse things now let’s you add more better things later. That wisdom has borne dividends for Clojure again and again over time, so I prefer the core team pushes back on these kinds of changes, in general. So, while I believe these new semantics are orthogonally non-complecting and suspicions to the contrary are healthy positions to have, I recommend folks check out the injest library and kick the tires, so folks can build a better intuition of their impact and we can all talk about their merits or lacktherof with more confidence.

As a bonus, you get the auto-transducifying x> and x>> - they're like training wheels for transducers :)