how to require clojure libs as classes instead of clj file to reduce repl start up time during development?

Question

Hi!

i read about reducing repl start up time during dev and tried to apply these knowledge to start REPL using Leiningen (in the article Alex shows this practice using deps.edn).

My steps:

I overrided clojure.core/load var to track time elapsed on loading module with:

(alter-var-root #'clojure.core/load (fn [origin-fn]
                                  (if (get (meta origin-fn) ::track-load?)
                                    origin-fn
                                    (vary-meta (fn [& args]
                                                 (let [start# (System/nanoTime)
                                                       ret# (apply origin-fn args)
                                                       elapsed-ms# (/ (double (- (System/nanoTime) start#)) 1000000.0)]
                                                   (newline)
                                                   (pr {:args (vec args)
                                                        :ms   elapsed-ms#})
                                                   (newline)
                                                   ret#))
                                               merge
                                               {::track-load? true}))))

I noticed that loading core clojure parts takes time ~0.10-0.20 ms and loading third party libs modules (cheshire for example) , takes ~200-400ms.

As i understand, main idea of reducing REPL start up time is the priority of compiled class before clj module if clj module was not changed. So we can compile third party libs and get benefit of not parsing their clj files but use their compiled classes.

I compiled all clojure files (my and libs) via leiningen uberjar and added folder with compiled class files to :checkout-deps-shares, i guess it is a way to add folder with class files to classpath:

:checkout-deps-shares [:source-paths :test-paths
                     ~(fn [p] (str (:root p) "/target/uberjar/classes/*"))]

Then i restarted repl, overrided clojure.core/load to track modules load time and as experiment do following:

(time (require 'runtime.account :reload-all))

where runtime.account is one of my root modules.

As a result i cannot see any difference in loading time of third party libs modules :/.

Maybe you can give me some details about mechanism deciding to load class file or clj file and some way to monitor that activity?

Thanks!

Answer 1

lein uberjar may not aot compile everything. One way to be sure is to set the :aot key to :all
Can you post your lein project.clj?

Comments

ok!
got it,  run my experiment with uberjar compiled with :aot :all.

No expected results :/


here is my project.clj, sorry for formatting

(defproject myproject "1.0.0"
            :dependencies [(comment "deps")]
            :uberjar-name "my-project.jar"
            :main runtime.run
            :source-paths ["src-clj"]
            :resource-paths ["resources"]
            :test-paths ["test-clj"]
            :target-path "target/%s"
            :env {(comment "ENV VARS")}
            :eftest {:multithread?    :namespaces
                           :capture-output? true
                           :test-warn-time  1000}
            :profiles {:default   [:base :system :user :provided :local-dev]
                              :user      {:dependencies         [(comment "dev deps")]
            :checkout-deps-shares [:source-paths :test-paths
                                                          ~(fn [p] (str (:root p) "/target/uberjar/classes/*"))]}
                       :local-dev {}
                       :uberjar   {:aot      :all
                                          :jvm-opts ["-Dclojure.compiler.direct-linking=true" "-Dclojure.compiler.elide-meta=[:doc :file :line :added]"]}}
            :aliases {"run-tests"  ["trampoline" "run" "-m" "test.core"]
                            "run-server" ["trampoline" "run" "-m" "runtime.run"]}
            :jvm-opts [(comment "JVM OPTS")]
            :repl-options {:timeout 180000
                                       :init-ns runtime.repl})

---

I discovered that putting "target/uberjar/classes" to :src-path speed up repl start time from 25 sec to 15 sec

but after that
(time (require 'runtime.account :reload-all))

(clojure.tools.namespace.repl/refresh) cannot resolve dependencies

clojure.tools.namespace.repl is configured to track following dirs: "src-clj", "test-clj"

---

I added :compile profile for lein project with :aot :all option configured
now with 'lein with-profile compile compile' i can generate class files for each namespace in
folder "/Users/user-name/core/project-name/target/default/classes"

i check through (clojure.java.classpath/classpath-directories) that classpath contains this folder

lein repl is starting 15 seconds instead of 25 seconds

but using (clojure.tools.namespace.repl/refresh)  fails with resolving dependency, file cannot be found

i guess there is some magic with class loaders

investigating