Require is not thread safe

Question

We're seeing this problem quite often, now that we scaled up a fleet of a system using Clojure quite heavily, I guess the numbers are just playing against us, but regardless this hinders the use of Clojure in every multi-threaded environment. I have put together a minimal example to reproduce the issue. So, the questions are,

• Is there a workaround for this issue? (akin some dynamic var that we set, I couldn't find any but there might be)
• Why is this considered a "heavy hammer"? (i.e., requiring is seldom done, and mostly in a single-threaded environment (think repl), while most prod env will benefit from this)
• Any hints as of why is this happening? (Reading the source code I don't quite understand why is this happening, might be that some ns requires (and therefore adds an ns), adds all symbols, and while there, another thread replaces that ns while requiring it)

Answer 1

There is a private function serialized-require that simply acquires a global lock before doing require, which you could either use, or do the same thing in your Java code around any uses of clojure.core/require, i.e. wrap it with the acquisition of a global lock, e.g. by making all require calls from inside of a synchronized method on a single object.

Comments

Thanks for your answer, I know that, however, that will not work in our case `:(` since at least one of those classes is AOT' compiled, which in turns uses `loadWithClass` which in turns calls `require` (to load the `clj` file), and that we cannot control.

---

If you cannot change that AOT compiled code, because it truly is not under your control, is it under your control to evaluate some code like shown below, and somehow ensure that it is evaluated before `require` can be called from multiple threads when your system is starting up?  The println's are optional, of course.


(def original-require clojure.core/require)

(defn my-serialized-require [& args]
  (locking clojure.lang.RT/REQUIRE_LOCK
    (println "my-serialized-require acquired the lock...")
    (apply original-require args)
    (println "my-serialized-require releasing lock...")))

(alter-var-root #'clojure.core/require (fn [& args] my-serialized-require))


If that isn't possible for you, are you allowed to compile a modified version of Clojure source code and use that in your project?  If so, you can modify the definition of `require` to be the locking version.

---

For what it's worth, requiring-resolve is the public API to the thread-safe (locked) require and my understanding is that the plan is to make the regular require thread-safe using this mechanism at some point (perhaps Clojure 1.11? Alex might have some insight on that) but that there is still some analysis and maybe research to be done before such a change can be made to a fundamental part of Clojure.

---

Thank you again, now that I do understand where is the problem I think I can put in place a workaround for our case, however, this does not solve the root issue, and I'd love to see a long-term, general solution for it.

---

Thank you again, now that I do understand where is the problem I think I can put in place a hack^H^H^H^H workaround for our case, however, this does not solve the root issue, and I'd love to see a long-term, general solution for it.

---

I understand your desire for a long term general solution.  I am an interested volunteer responding to the part of your question where you ask for a workaround.  I have zero control over the contents of the released versions of Clojure.  The official Clojure maintainers also read these issues, and they can respond with whatever they believe is a long term general solution.

---

I do appreciate that, thank you!

---

Hello, just echo'ing things above - yes, this is a known issue and is on the candidate list for inclusion in Clojure 1.11. Parallel (conflicting) loads are relatively rare. `requiring-resolve` was added in 1.10 to cover the most common dynamic load use case (which can easily race).

As Andy mentioned above, clojure.lang.RT/REQUIRE_LOCK was intentionally left available so that user programs that needed to could participate in this lock. This should still be considered implementation and subject to future removal but in the meantime the suggested workaround can be used in cases like you mention.

The expectation is that the long-term solution is to make normal `require` safer, but this will require significant analysis and testing that has not yet been completed.

---

gen-class could be another common use case. If you generate a class from Clojure, and Java creates instances of that class inside multiple threads, it's likely that the generated class initializer will require the implementing Clojure namespace under the hood in a non thread safe manner. We seem to have this issue currently.