[CLJS] Should macros support "lifting" vars to the ns-level?

Question

I'm by no means a compiler expert so I don't know if the name "lifting" is even in the right ballpark for this. Kinda hard to google when you don't know the term you are looking for.

I'm writing a pure ClojureScript implementation to do DOM processing. Kinda like a mix between Svelte and React. In it I have a macro that rewrites a common Hiccup form to create a "create" function and an "update" function. For this macro to work efficiently I want to be able to create ns-level "vars".

An example of what this may look like

(defn card [{:keys [title body]}]
  (<> [:div.card
       [:div.card-title title]
       [:div.card-body body]
       [:div.card-footer
        [:div.card-actions
         [:button "ok"]
         [:button "cancel"]]]]))

So the <> macro (called a "fragment") rewrites this to two functions. One that creates the initial DOM and one that updates the parts that can actually change. This is done to avoid as much "diffing" as possible.

Without going into too much more detail on this it basically generates

(defn card [{:keys [title body]}]
  (fragment-create "dummy-id" [title body]
    (fn create-fn [state] ...)
    (fn update-fn [state] ...)))

The problem with this is that it will re-create the functions each time the card function is called. Since functions do not have "identity" in JS the runtime can't tell if that is still the same identical? fragment from before or if it was updated (either via the REPL or hot-reload). It has to rely on some unique identifier for that. The JS runtime would also always allocate the functions on each call although the runtime may decide to re-use the previous fragment and throw them away immediately.

So what I'd like to be able to generate is something closer to this from the original code

(def fragment-123
  (fragment-create
    (fn create-fn [state] ...)
    (fn update-fn [state] ...)))

(defn card [{:keys [title body]}]
  (fragment-use fragment-123 [title body]))

So the fragment "handler" is only created once (currently using a deftype but could just be a map). And it is then "used" whenever the card function is called.

Without "lifting" support the code can only generate the def inline

(defn card [{:keys [title body]}]
  (do (def fragment-123
        (fragment-create
          (fn create-fn [state] ...)
          (fn update-fn [state] ...)))
      (fragment-use fragment-123 [title body])))

That is obviously problematic. So it would need to check if the fragment is already defined and so on. reify currently does this and the code really isn't pleasent to look at.

I implemented this as an experiment in shadow-cljs where macros wanting to use this can call a special function (from &env currently, could be a binding) that basically lets them "prepend" a form to whatever top-level form it is currently working on.

Should this be something the compiler natively supports?

I know that Clojure doesn't have it either but it is sort of doing it by creating classes and accessing them whenever so it kinda does this.

The implementation could be different. I'm just trying to determine if I'm crazy for wanting to do this in the first place. The previous implementation worked fine just ended up being more code with a bunch of extra additional checks since it can't rely on identical?.

Answer 1

Thinking within the current language/compiler - the desired behavior looks like defonce, but with support for REPL/live-reload changes. If the exists? check in defonce is too slow I think one could write an analogous macro with a per-invocation cost of about one javascript object property lookup (either on top of def or using a registry, as Tom mentioned.)

Figuring out how & when to invalidate previous values during dev seems tricker. One could use something like gensym during macroexpansion to get a fresh entry whenever the top-level form is recompiled. are there cases where you'd re-evaluate something from your editor, and shadow would not recompile but use a cached version, so you'd be stuck with a stale value?

"lifting" sounds similar to hoisting or loop-invariant code motion but unlike these compiler optimizations, the desire here is to change the semantics, so it feels like quite a different thing to me (even if the mechanics are similar).

Comments

"Hoisting" sounds similar I guess. My desire is not to change semantics, which this doesn't IMHO.

I'm not really looking for alternative suggestions. I've been through plenty of them and they all do their job fine when needed. The solution I presented here however requires the least amount of code and should in theory be the most performant since no checks need to be performed anywhere. Benchmarks still need to be written so I guess I should have done that first.

I created an [example gist](https://gist.github.com/thheller/8688986f8c7a1f6aef1b4a411ed3bf0f#file-reify-current-js) to show the `reify` output. Note that everything is nested in `make_thing`. IMHO that shows that there is a definite use-case for this in `cljs.core` itself.

---

Fun fact when writing the benchmark the lifted variant outperforms everything else by an order of magnitude ... BUT not because of the code actually being faster. Closure can just analyze it better and decided to remove most of the code from that benchmark while the others kept all their code.

---

Interesting results though. My instincts were correct that the lifted variant is fastest but by less than expected.

node out/bench-fragment.js
fragment-lift x 71,860,782 ops/sec ±2.76% (83 runs sampled)
fragment-with-checks x 24,358,151 ops/sec ±1.08% (93 runs sampled)
fragment-always x 62,388,334 ops/sec ±1.50% (85 runs sampled)

with-checks (checking if in registry) is by far the slowest (kinda expected) but I did not expect the variant that always blindly allocates both functions to be so close.

---

Interesting - what does the code for the checked variant look like?

---

https://github.com/thheller/shadow-experiments/blob/c60281cad708928a4f3d4b9b55fca94b133b9644/src/main/shadow/experiments/arborist/fragments.clj#L393-L400

fragment-get would do a atom deref/map lookup which is probably the cause of the slowdown.

---

> a atom deref/map lookup which is probably the cause of the slowdown.

I think you are right here. I'm seeing little difference between reading a def, reading a single object property (how a performant registry could be implemented), and doing the exists? check (what reify does), though clearly exists? is doing more work than necessary for this particular case. But reading an atom is way slower.

```
exists?         x 927,315,604 ops/sec ±1.02% (91 runs sampled)
cache           x 932,548,414 ops/sec ±1.18% (90 runs sampled)
def             x 933,832,352 ops/sec ±1.13% (92 runs sampled)

read-from-atom  x 70,540,398 ops/sec ±0.26% (97 runs sampled)
```

source w compiled js: https://gist.github.com/mhuebert/2781f9d1b2481301a8eb17ce2c5d0e3e

> "Hoisting" sounds similar I guess. My desire is not to change semantics, which this doesn't IMHO.

If we hoist an expression like `42`, no program will break, we can statically determine that it's a safe optimization, what the cljs constants table does. I would think hoisting an expression that returns a function is not a safe optimization to make unless we can prove that no part of the program relies on the existing, different behaviour, which is that each function is a new unique object that can be mutated and compared independently. ie the semantics may be equivalent for the particular way in which you are using these functions, not generally equivalent, lots of js relies on how it works currently. (I am also no compiler expert.)

---

I'm not sure I follow your comments on hoisting. This isn't something that would be used in a lot of places. The only thing I can think of in core is reify. The entire point is to get the exact semantics of a ns-level def. Any code using this for something else would be incorrect. It can't even access the locals from where the macro is running.

Seeing what the Closure Compiler does to the code in :advanced has already convinced me to keep this in shadow-cljs. It doesn't need to make it into core, since the fallback I added works too and just runs a little slower and generates a little more code.

---

Did not follow everything you are doing, but can't you perform the def at macro-expansion time? That's what I would do in Clojure.

Instead of returning

    (def fragment-123
            (fragment-create
              (fn create-fn [state] ...)
              (fn update-fn [state] ...)))

I would execute it in the macro, and only return:

    (fragment-use fragment-123 [title body])

But maybe this is not possible in CLJS given macros aren't available at runtime, not sure.

Also, for checking if the var is already defined, wouldn't using defonce instead of def just do it, without making the code any more complicated? Though I understand the possible performance concerns:

    (defn card [{:keys [title body]}]
      (do (defonce fragment-123
            (fragment-create
              (fn create-fn [state] ...)
              (fn update-fn [state] ...)))
          (fragment-use fragment-123 [title body])))

---

Just looking at the defonce inside the defn makes my skin crawl. :P

But after a few more tweaks I got this to a point where there is no measurable difference in Chrome and Firefox is only ~10% slower when using the non-def variant. Although I don't really trust the benchmark results given that they vary too much between each run.

The non-def variant produces slightly more code but only about 50 bytes per fragment and probably gzips well enough so it doesn't matter. Dunno about parse/load performance, still need to figure out how to benchmark that.

Hiccup is also surprisingly competetive in Chrome and only 10% slower to create and 2x slower to update. But 5x slower to create in Firefox and 10x slower to update, differences widen however in more complex scenarios. Not to mention the 100x difference in :none due to all the non-optimized keyword construction.

Enough time wasted trying to benchmark this. I'm satisfied with the results showing that it isn't really worth worrying about this overall.

---

Haha, sure, but I don't know why the defonce makes your skin crawl anymore than seeing a def in there :p

My macros always return monstrous things, but its hidden away behind the nice macro interface :p I say what your users don't know doesn't hurt them :p

Answer 2

I've used lexical bindings with a defn, built via macro, to get at the use case you describe.

(defmacro fragment [name args & body]
  `(let [frag# (fragment-create ~args ~@body)]
     (defn ~name ~args
       (frag# ~@args))))

Not sure about the exact plumbing, but it seems like that would work identically to the def version you showed, without having to pollute the global ns. Basically using a let-over-lambda approach facilitated by a helper macro to eliminate the boiler plate.

If you "need" some global registry of functions (like def), there's nothing stopping this approach from side-effecting some registry (an atom) and inserting the fragment function there. You could then reference that in other fragments via a lookup function (or perhaps define some syntactic sugar through the macro to translate stuff to a lookup). Again, unsure what the exact use cases are, but I think you can avoid lifting vars.

Comments

I can avoid lifting vars if I add a conditional check or some other "glue" code. The point is removing that conditional check and doing it only once. The fragment macro itself can be used anywhere so I can't make that itself just a defn.