`clj` behaves different in the REPL as opposed to from a file

Question

I'm down a rabbit hole of trying to write a new "literate mode" for clojure

I basically just want to launch clj in an inferior mode and then pipe in the code from the code blocks. I'm trying to leverage add-lib so I can dynamically add libraries in literate blocks (and eshew the whole deps.edn project structure). However I've hit a weird issue

This is my test code

(use 'clojure.tools.deps.alpha.repl)
(add-lib 'thi.ng/geom {:mvn/version "1.0.0-RC4"})
(require `thi.ng.geom.svg.core)
(thi.ng.geom.svg.core/circle [0 1] 6)

If I launch a repl with

clj -Sdeps '{:deps {org.clojure/tools.deps.alpha {:git/url "https://github.com/clojure/tools.deps.alpha.git" :sha "d492e97259c013ba401c5238842cd3445839d020"}}}'

(hash is from the official Clojure blog post here ) I can then copy paste the test code in and I get the desired result - it prints: [:circle {:cx "0.00", :cy "1.00", :r 6}]

However if I I feed the same exact code through an intermediary file add-lib-test.clj

clj -Sdeps '{:deps {org.clojure/tools.deps.alpha {:git/url "https://github.com/clojure/tools.deps.alpha.git" :sha "d492e97259c013ba401c5238842cd3445839d020"}}}'  add-lib-test.clj 

then it blows up

WARNING: When invoking clojure.main, use -M
Syntax error (IllegalAccessError) compiling at (add-lib-test.clj:2:1).
Context classloader is not a DynamicClassLoader

Full report at:
/tmp/clojure-1788839394237911817.edn

geokon@ux305-kde:~$ cat /tmp/clojure-1788839394237911817.edn
{:clojure.main/message
 "Syntax error (IllegalAccessError) compiling at (add-lib-test.clj:2:1).\nContext classloader is not a DynamicClassLoader\n",
 :clojure.main/triage
 {:clojure.error/phase :compile-syntax-check,
  :clojure.error/line 2,
  :clojure.error/column 1,
  :clojure.error/source "add-lib-test.clj",
  :clojure.error/path "add-lib-test.clj",
  :clojure.error/class java.lang.IllegalAccessError,
  :clojure.error/cause
  "Context classloader is not a DynamicClassLoader"},
 :clojure.main/trace
 {:via
  [{:type clojure.lang.Compiler$CompilerException,
    :message
    "Syntax error compiling at (/home/geokon/add-lib-test.clj:2:1).",
    :data
    {:clojure.error/phase :compile-syntax-check,
     :clojure.error/line 2,
     :clojure.error/column 1,
     :clojure.error/source "/home/geokon/add-lib-test.clj"},
    :at [clojure.lang.Compiler load "Compiler.java" 7648]}
   {:type java.lang.IllegalAccessError,
    :message "Context classloader is not a DynamicClassLoader",
    :at
    [clojure.tools.deps.alpha.repl$add_loader_url
     invokeStatic
     "repl.clj"
     32]}],
  :trace
  [[clojure.tools.deps.alpha.repl$add_loader_url
    invokeStatic
    "repl.clj"
    32]
   [clojure.tools.deps.alpha.repl$add_loader_url invoke "repl.clj" 22]
   [clojure.core$run_BANG_$fn__8790 invoke "core.clj" 7715]
   [clojure.lang.ArrayChunk reduce "ArrayChunk.java" 58]
   [clojure.core.protocols$fn__8154 invokeStatic "protocols.clj" 136]
   [clojure.core.protocols$fn__8154 invoke "protocols.clj" 124]
   [clojure.core.protocols$fn__8114$G__8109__8123
    invoke
    "protocols.clj"
    19]
   [clojure.core.protocols$seq_reduce invokeStatic "protocols.clj" 31]
   [clojure.core.protocols$fn__8146 invokeStatic "protocols.clj" 75]
   [clojure.core.protocols$fn__8146 invoke "protocols.clj" 75]
   [clojure.core.protocols$fn__8088$G__8083__8101
    invoke
    "protocols.clj"
    13]
   [clojure.core$reduce invokeStatic "core.clj" 6828]
   [clojure.core$run_BANG_ invokeStatic "core.clj" 7710]
   [clojure.core$run_BANG_ invoke "core.clj" 7710]
   [clojure.tools.deps.alpha.repl$add_lib invokeStatic "repl.clj" 58]
   [clojure.tools.deps.alpha.repl$add_lib invoke "repl.clj" 35]
   [clojure.tools.deps.alpha.repl$add_lib invokeStatic "repl.clj" 48]
   [clojure.tools.deps.alpha.repl$add_lib invoke "repl.clj" 35]
   [user$eval796 invokeStatic "add-lib-test.clj" 2]
   [user$eval796 invoke "add-lib-test.clj" 2]
   [clojure.lang.Compiler eval "Compiler.java" 7177]
   [clojure.lang.Compiler load "Compiler.java" 7636]
   [clojure.lang.Compiler loadFile "Compiler.java" 7574]
   [clojure.main$load_script invokeStatic "main.clj" 475]
   [clojure.main$script_opt invokeStatic "main.clj" 535]
   [clojure.main$script_opt invoke "main.clj" 530]
   [clojure.main$main invokeStatic "main.clj" 664]
   [clojure.main$main doInvoke "main.clj" 616]
   [clojure.lang.RestFn applyTo "RestFn.java" 137]
   [clojure.lang.Var applyTo "Var.java" 705]
   [clojure.main main "main.java" 40]],
  :cause "Context classloader is not a DynamicClassLoader",
  :phase :compile-syntax-check}}

I don't really know enough JVM to understand this error. It looks to me that it's trying to compile the file - and this doesn't play nice with add-lib. Is there some way I can have clj evaluate without compiling?

Answer 1

FYI, that version of add-lib is quite old now and is unlikely to work well with the current clojure CLI. You can probably crib the latest from https://github.com/seancorfield/dot-clojure/blob/develop/deps.edn#L202-L228

The way this stuff works is to climb up the classloader structure to find a DynamicClassLoader, then use the facilities there to add new jars to the classloader. For that to work, you actually have to have a DynamicClassLoader available to you, which is not necessarily the case in every way you can load and run Clojure code. You can see in that link above a trick Sean is playing to install one - you can do similar in your own code if you like.

Comments

Thank you so much for trying to help @alexmiller. Is there some place you'd recommend to read up on ClassLoaders and this JVM background? I'm a bit embarassed I can't diagnose this on my own

I've modified `add-test-lib.clj` to work with the latest version of `add-lib` (now `add-libs`) and to set the `DynamicClassLoader` as per the example you linked

    (->>(Thread/currentThread)
        (.getContextClassLoader)
        (clojure.lang.DynamicClassLoader.)
        (.setContextClassLoader (Thread/currentThread)))

    (use 'clojure.tools.deps.alpha.repl)
    (add-libs {'thi.ng/geom {:mvn/version "1.0.0-RC4"}})
    (require `thi.ng.geom.svg.core)
    (thi.ng.geom.svg.core/circle [0 1] 6)

And now I run with the latest version

    clj -Sdeps '{:deps {org.clojure/tools.deps.alpha {:git/url "https://github.com/clojure/tools.deps.alpha.git" :sha "241cd24c35ba770aea4773ea161d45276e5d3a73"}}}' add-lib-test.clj

Again, it works in the REPL but not through a file. However it does get a bit further now :) I'm guessing the ClassLoader is still have issues, or there is something wrong with the non-REPL classpath (another term I'm a bit hazy on).

    {:clojure.main/message
    "Syntax error (FileNotFoundException) compiling at (add-lib-test.clj:4:1).\nCould not locate thi/ng/geom/svg/core__init.class, thi/ng/geom/svg/core.clj or thi/ng/geom/svg/core.cljc on classpath.\n",
    :clojure.main/triage
    {:clojure.error/phase :compile-syntax-check,
    :clojure.error/line 4,
    :clojure.error/column 1,
    :clojure.error/source "add-lib-test.clj",
    :clojure.error/path "add-lib-test.clj",
    :clojure.error/class java.io.FileNotFoundException,
    :clojure.error/cause
    "Could not locate thi/ng/geom/svg/core__init.class, thi/ng/geom/svg/core.clj or thi/ng/geom/svg/core.cljc on classpath."},
    :clojure.main/trace
    {:via
    [{:type clojure.lang.Compiler$CompilerException,
        :message
        "Syntax error compiling at (/home/geokon/add-lib-test.clj:4:1).",
        :data
        {:clojure.error/phase :compile-syntax-check,
        :clojure.error/line 4,
        :clojure.error/column 1,
        :clojure.error/source "/home/geokon/add-lib-test.clj"},
        :at [clojure.lang.Compiler load "Compiler.java" 7648]}
    {:type java.io.FileNotFoundException,
        :message
        "Could not locate thi/ng/geom/svg/core__init.class, thi/ng/geom/svg/core.clj or thi/ng/geom/svg/core.cljc on classpath.",
        :at [clojure.lang.RT load "RT.java" 462]}],
    :trace
    [[clojure.lang.RT load "RT.java" 462]
    [clojure.lang.RT load "RT.java" 424]
    [clojure.core$load$fn__6839 invoke "core.clj" 6126]
    [clojure.core$load invokeStatic "core.clj" 6125]
    [clojure.core$load doInvoke "core.clj" 6109]
    [clojure.lang.RestFn invoke "RestFn.java" 408]
    [clojure.core$load_one invokeStatic "core.clj" 5908]
    [clojure.core$load_one invoke "core.clj" 5903]
    [clojure.core$load_lib$fn__6780 invoke "core.clj" 5948]
    [clojure.core$load_lib invokeStatic "core.clj" 5947]
    [clojure.core$load_lib doInvoke "core.clj" 5928]
    [clojure.lang.RestFn applyTo "RestFn.java" 142]
    [clojure.core$apply invokeStatic "core.clj" 667]
    [clojure.core$load_libs invokeStatic "core.clj" 5985]
    [clojure.core$load_libs doInvoke "core.clj" 5969]
    [clojure.lang.RestFn applyTo "RestFn.java" 137]
    [clojure.core$apply invokeStatic "core.clj" 667]
    [clojure.core$require invokeStatic "core.clj" 6007]
    [clojure.core$require doInvoke "core.clj" 6007]
    [clojure.lang.RestFn invoke "RestFn.java" 408]
    [user$eval4834 invokeStatic "add-lib-test.clj" 4]
    [user$eval4834 invoke "add-lib-test.clj" 4]
    [clojure.lang.Compiler eval "Compiler.java" 7177]
    [clojure.lang.Compiler load "Compiler.java" 7636]
    [clojure.lang.Compiler loadFile "Compiler.java" 7574]
    [clojure.main$load_script invokeStatic "main.clj" 475]
    [clojure.main$script_opt invokeStatic "main.clj" 535]
    [clojure.main$script_opt invoke "main.clj" 530]
    [clojure.main$main invokeStatic "main.clj" 664]
    [clojure.main$main doInvoke "main.clj" 616]
    [clojure.lang.RestFn applyTo "RestFn.java" 137]
    [clojure.lang.Var applyTo "Var.java" 705]
    [clojure.main main "main.java" 40]],
    :cause
    "Could not locate thi/ng/geom/svg/core__init.class, thi/ng/geom/svg/core.clj or thi/ng/geom/svg/core.cljc on classpath.",
    :phase :compile-syntax-check}}