How to control the transient deps that get compiled/assembled into an uberjar?

Question

I am having trouble understanding how to control the contents of an uberjar using tools.build. In this scenario, the runtime environment (a Spark cluster) provides some deps and thus those deps should not be in the uberjar. Let's say I have 2 classpaths (represented as aliases):

• :compile The classpath with all deps/classes that are required for compilation.

• :uber The classpath with only the deps/classes that should be in the final jar.

In theory, if you take the :compile classpath and subtract the :uber classpath, we should get the set of classes that are provided by the runtime environment.

Consider the following simplified dependency graph:

We only need the blue deps to be in the jar. The white "provided" deps need to be used during compilation but should not be in the final uberjar.

I looked into using the :filter-nses option of compile-clj but it creates problems with transient deps.

• If we filter to only my.project we will lose the important deps of some.other/dep and some.transient/dep.
• If we filter to exclude org.apache (not possible with :filter-nses, but in theory it could be done) we would still have org.scala-lang/scala-library hanging around.

I also experimented using different calls to create-basis during compile and jar building. I'm not sure if that makes any sense.

Is it possible to leverage the dependency graph to control either 1) what gets compiled or 2) which classes get put into an uberjar? Any advice would be much appreciated.

Answer 1

Using multiple calls to create-basis is exactly right - one for compile and one for uberjar.

Comments

Good to know. Is there an example of this documented somewhere? Or perhaps a known open source project that uses this pattern that I could read from?

This gist contains my attempt at using multiple bases but it throws exceptions that I can't explain:
https://gist.github.com/erp12/7ea123fe4d623f61dcda94e3cbd03845#file-build-clj

Interestingly, the root cause exception is not consistent. Below are two examples.
Even more strange is that these exceptions are raised even if `jar-basis` is never referenced in any entry-point function. It seems just calling `create-basis` multiple times is causing the exceptions.

```
Exception in thread "Thread-7" java.lang.ExceptionInInitializerError
    at clojure.tools.deps.alpha.util.S3TransporterFactory.triggerLoad(S3TransporterFactory.java:44)
    at clojure.tools.deps.alpha.util.S3TransporterFactory.access$100(S3TransporterFactory.java:29)
    at clojure.tools.deps.alpha.util.S3TransporterFactory$1.run(S3TransporterFactory.java:49)
    at java.lang.Thread.run(Thread.java:748)
Caused by: Syntax error compiling at (clojure/tools/reader.clj:1:1).
    at clojure.lang.Compiler.load(Compiler.java:7652)
    at clojure.lang.RT.loadResourceScript(RT.java:381)
    ...
Caused by: java.lang.IllegalAccessError: indexing-reader? does not exist
    at clojure.core$refer.invokeStatic(core.clj:4237)
    at clojure.core$refer.doInvoke(core.clj:4205)
```

Another version of the error.

```
Exception in thread "Thread-7" java.lang.ExceptionInInitializerError
    at clojure.tools.deps.alpha.util.S3TransporterFactory.triggerLoad(S3TransporterFactory.java:44)
    at clojure.tools.deps.alpha.util.S3TransporterFactory.access$100(S3TransporterFactory.java:29)
    at clojure.tools.deps.alpha.util.S3TransporterFactory$1.run(S3TransporterFactory.java:49)
    at java.lang.Thread.run(Thread.java:748)
Caused by: Syntax error compiling at (clojure/tools/reader/reader_types.clj:1:1).
    at clojure.lang.Compiler.load(Compiler.java:7652)
    at clojure.lang.RT.loadResourceScript(RT.java:381)
    ...
Caused by: java.lang.IllegalAccessError: whitespace? does not exist
    at clojure.core$refer.invokeStatic(core.clj:4237)
    at clojure.core$refer.doInvoke(core.clj:4205)
```

---

This is a race condition on code loading.

calling create-basis ultimately causes maven code to spin off another thread which ends up loading some code if it isn't loaded yet. That code loading is racing with other code loading happening in your build.clj, and causing these errors.

if you require the namespace clojure.tools.deps.alpha.util.s3-transporter at the top of your build.clj it should preload the code and eliminate the race. It will make your script slower to launch though.

---

Thanks for the explanation of the issue! That is very helpful.

---

Also ran into the race condition with https://github.com/clj-easy/graal-build-time.

---

The race condition on load here should be fixed in latest release of tools.build btw.