Welcome! Please see the About page for a little more info on how this works.

Ask a Question

closed clojure throws (various exceptions) immediately - AdoptOpenJDK J9 JVM

+1 vote
in Clojure by
closed by

I am having trouble with Clojure on the J9 JVM.

I'm interested in J9 in hopes Clojure won't limit one's choice of virtual machine.

The trouble happens in Windows and Linux, and both the "11" and "15" versions of the J9 JVM, and numerous (all?) Clojure versions including 1.10.1, 1.10.2, 1.7, and 1.5.1.

The trouble is an Exception or Error, of one kind or another, usually originating in clojure.core; frequently it is an absurd NullPointerException in reduce1 that is also thrown by Leiningen on the J9 JVM.

The problem is erratic. But something reliably goes wrong if J9 "always compiles" (-Xcomp) and it seems to work fine when J9 "always interprets" (-Xint). The clojure commands given in this report, for some reason, don't seem to honor the -J option so I modified the clojure script to pass along the necessary JVM options for this experiment.

I installed clojure with

./linux-install-1.10.2.774.sh --prefix ${HOME}/clo

I got AdoptOpenJDK, version 11, J9 JVM, for Linux, from adoptopenjdk.org.

On the first run of clojure

${HOME}/clo/bin/clojure

it threw NullPointerException. More about this below.

Trying various JVM options, I found that the -Xint JVM option allowed clojure to download various things and display the REPL prompt. According to java -X this option means "interpreted mode execution only". To use -Xint, I modified clojure to pass along the JVM option. (The -J option is honored by not all of clojure's invocations of Java.)

After the first successful run, subsequent runs of clojure usually succeeded in producing the REPL prompt.

However, the -Xcomp JVM option makes the error happen every time. It means "forces compilation of methods on first invocation".

I first ran into a J9 problem when trying to use the Windows version of Leiningen 2.9.5. Some of my discovery path is in a Github issue there: https://github.com/technomancy/leiningen/issues/2725

The NullPointerException stack trace was:

Exception in thread "main" java.lang.NullPointerException
	at clojure.core.protocols$naive_seq_reduce.invokeStatic(protocols.clj:62)
	at clojure.core.protocols$interface_or_naive_reduce.invokeStatic(protocols.clj:72)
     ...
	at clojure.pprint$pprint.invoke(pprint_base.clj:241)
	at clojure.lang.Var.invoke(Var.java:384)
	at clojure.main$report_error$fn__9186$fn__9187.invoke(main.clj:603)
	at clojure.main$report_error$fn__9186.invoke(main.clj:602)
	at clojure.main$report_error.invokeStatic(main.clj:601)
	at clojure.main$main.invokeStatic(main.clj:666)
	at clojure.main$main.doInvoke(main.clj:616)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.lang.Var.applyTo(Var.java:705)
	at clojure.main.main(main.java:40)

On a subsequent run, I got a different error:

Syntax error macroexpanding clojure.core/defn- at (clojure/tools/cli.cljc:124:1).
null
 Full report at:
/tmp/clojure-12652663517358293277.edn

and that file said:

{:clojure.main/message
 "Syntax error macroexpanding clojure.core/defn- at (clojure/tools/cli.cljc:124:1).\nnull\n",
 :clojure.main/triage
 {:clojure.error/phase :macro-syntax-check,
  :clojure.error/line 124,
  :clojure.error/column 1,
  :clojure.error/source "cli.cljc",
  :clojure.error/symbol clojure.core/defn-,
  :clojure.error/path "clojure/tools/cli.cljc",
  :clojure.error/class java.lang.NullPointerException},
 :clojure.main/trace
 {:via
  [{:type clojure.lang.Compiler$CompilerException,
    :message
    "Syntax error macroexpanding clojure.core/defn- at (clojure/tools/cli.cljc:124:1).",
    :data
    {:clojure.error/phase :macro-syntax-check,
     :clojure.error/line 124,
     :clojure.error/column 1,
     :clojure.error/source "clojure/tools/cli.cljc",
     :clojure.error/symbol clojure.core/defn-},
    :at [clojure.lang.Compiler checkSpecs "Compiler.java" 6976]}
   {:type java.lang.NullPointerException,
    :at [clojure.core$every_QMARK_ invokeStatic "core.clj" 2683]}],
  :trace
  [[clojure.core$every_QMARK_ invokeStatic "core.clj" 2683]
   [clojure.core$every_QMARK_ invoke "core.clj" 2676]
   [clojure.spec.alpha$pcat_STAR_ invokeStatic "alpha.clj" 1388]
   [clojure.spec.alpha$pcat_STAR_ invoke "alpha.clj" 1387]
   [clojure.spec.alpha$deriv invokeStatic "alpha.clj" 1542]
   [clojure.spec.alpha$deriv invoke "alpha.clj" 1528]
   [clojure.spec.alpha$re_conform invokeStatic "alpha.clj" 1669]
   [clojure.spec.alpha$re_conform invoke "alpha.clj" 1660]
   [clojure.spec.alpha$regex_spec_impl$reify__2510
    conform_STAR_
    "alpha.clj"
    1710]
   [clojure.spec.alpha$conform invokeStatic "alpha.clj" 171]
   [clojure.spec.alpha$conform invoke "alpha.clj" 167]
   [clojure.spec.alpha$macroexpand_check invokeStatic "alpha.clj" 708]
   [clojure.spec.alpha$macroexpand_check invoke "alpha.clj" 704]
   [clojure.lang.AFn applyToHelper "AFn.java" 156]
   [clojure.lang.AFn applyTo "AFn.java" 144]
   [clojure.lang.Var applyTo "Var.java" 705]
   [clojure.lang.Compiler checkSpecs "Compiler.java" 6974]
   [clojure.lang.Compiler macroexpand1 "Compiler.java" 6992]
   [clojure.lang.Compiler macroexpand "Compiler.java" 7079]
   [clojure.lang.Compiler eval "Compiler.java" 7165]
   [clojure.lang.Compiler load "Compiler.java" 7640]
   [clojure.lang.RT loadResourceScript "RT.java" 381]
   [clojure.lang.RT loadResourceScript "RT.java" 372]
   [clojure.lang.RT load "RT.java" 459]
   [clojure.lang.RT load "RT.java" 424]
   [clojure.core$load$fn__6857 invoke "core.clj" 6115]
   [clojure.core$load invokeStatic "core.clj" 6114]
   [clojure.core$load doInvoke "core.clj" 6098]
   [clojure.lang.RestFn invoke "RestFn.java" 408]
   [clojure.core$load_one invokeStatic "core.clj" 5897]
   [clojure.core$load_one invoke "core.clj" 5892]
   [clojure.core$load_lib$fn__6797 invoke "core.clj" 5937]
   [clojure.core$load_lib invokeStatic "core.clj" 5936]
   [clojure.core$load_lib doInvoke "core.clj" 5917]
   [clojure.lang.RestFn applyTo "RestFn.java" 142]
   [clojure.core$apply invokeStatic "core.clj" 669]
   [clojure.core$load_libs invokeStatic "core.clj" 5974]
   [clojure.core$load_libs doInvoke "core.clj" 5958]
   [clojure.lang.RestFn applyTo "RestFn.java" 137]
   [clojure.core$apply invokeStatic "core.clj" 669]
   [clojure.core$require invokeStatic "core.clj" 5996]
   [clojure.core$require doInvoke "core.clj" 5996]
   [clojure.lang.RestFn invoke "RestFn.java" 619]
   [clojure.tools.deps.alpha.script.make_classpath2$eval140$loading__6738__auto____141
    invoke
    "make_classpath2.clj"
    9]
   [clojure.tools.deps.alpha.script.make_classpath2$eval140
    invokeStatic
    "make_classpath2.clj"
    9]
   [clojure.tools.deps.alpha.script.make_classpath2$eval140
    invoke
    "make_classpath2.clj"
    9]
   [clojure.lang.Compiler eval "Compiler.java" 7181]
   [clojure.lang.Compiler eval "Compiler.java" 7170]
   [clojure.lang.Compiler load "Compiler.java" 7640]
   [clojure.lang.RT loadResourceScript "RT.java" 381]
   [clojure.lang.RT loadResourceScript "RT.java" 372]
   [clojure.lang.RT load "RT.java" 459]
   [clojure.lang.RT load "RT.java" 424]
   [clojure.core$load$fn__6857 invoke "core.clj" 6115]
   [clojure.core$load invokeStatic "core.clj" 6114]
   [clojure.core$load doInvoke "core.clj" 6098]
   [clojure.lang.RestFn invoke "RestFn.java" 408]
   [clojure.core$load_one invokeStatic "core.clj" 5897]
   [clojure.core$load_one invoke "core.clj" 5892]
   [clojure.core$load_lib$fn__6797 invoke "core.clj" 5937]
   [clojure.core$load_lib invokeStatic "core.clj" 5936]
   [clojure.core$load_lib doInvoke "core.clj" 5917]
   [clojure.lang.RestFn applyTo "RestFn.java" 142]
   [clojure.core$apply invokeStatic "core.clj" 669]
   [clojure.core$load_libs invokeStatic "core.clj" 5974]
   [clojure.core$load_libs doInvoke "core.clj" 5958]
   [clojure.lang.RestFn applyTo "RestFn.java" 137]
   [clojure.core$apply invokeStatic "core.clj" 669]
   [clojure.core$require invokeStatic "core.clj" 5996]
   [clojure.main$main_opt invokeStatic "main.clj" 514]
   [clojure.main$main_opt invoke "main.clj" 510]
   [clojure.main$main invokeStatic "main.clj" 664]
   [clojure.main$main doInvoke "main.clj" 616]
   [clojure.lang.RestFn applyTo "RestFn.java" 137]
   [clojure.lang.Var applyTo "Var.java" 705]
   [clojure.main main "main.java" 40]],
  :phase :macro-syntax-check}}

On yet another run, I got yet a different error message:

Exception in thread "main" java.lang.NullPointerException
	at clojure.core$reduce1.invokeStatic(core.clj:944)
	at clojure.core$reduce1.invokeStatic(core.clj:936)
	at clojure.core$merge.invokeStatic(core.clj:3052)
	at clojure.core$Throwable__GT_map$base__7474.invoke(core_print.clj:491)
	at clojure.core$map$fn__5885.invoke(core.clj:2757)
	at clojure.lang.LazySeq.sval(LazySeq.java:42)
	at clojure.lang.LazySeq.seq(LazySeq.java:51)
	at clojure.lang.RT.seq(RT.java:535)
	at clojure.lang.LazilyPersistentVector.create(LazilyPersistentVector.java:44)
	at clojure.core$vec.invokeStatic(core.clj:379)
	at clojure.core$Throwable__GT_map.invokeStatic(core_print.clj:492)
	at clojure.main$report_error.invokeStatic(main.clj:593)
	at clojure.main$main.invokeStatic(main.clj:666)
	at clojure.main$main.doInvoke(main.clj:616)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.lang.Var.applyTo(Var.java:705)
	at clojure.main.main(main.java:40)

The problem is also reproducible in Clojure 1.5.1:

$ java -Xcomp -cp clojure-1.5.1.jar clojure.main
Exception in thread "main" java.lang.ExceptionInInitializerError
	at java.base/java.lang.J9VMInternals.ensureError(J9VMInternals.java:185)
	at java.base/java.lang.J9VMInternals.recordInitializationFailure(J9VMInternals.java:174)
	at java.base/java.lang.Class.forNameImpl(Native Method)
	at java.base/java.lang.Class.forName(Class.java:413)
	at clojure.lang.RT.loadClassForName(RT.java:2098)
	at clojure.lang.RT.load(RT.java:430)
	at clojure.lang.RT.load(RT.java:411)
	at clojure.core$load$fn__5018.invoke(core.clj:5530)
	at clojure.core$load.doInvoke(core.clj:5529)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invoke(core.clj:5336)
	at clojure.core$load_lib$fn__4967.invoke(core.clj:5375)
	at clojure.core$load_lib.doInvoke(core.clj:5374)
	at clojure.lang.RestFn.applyTo(RestFn.java:142)
	at clojure.core$apply.invoke(core.clj:619)
	at clojure.core$load_libs.doInvoke(core.clj:5413)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.core$apply.invoke(core.clj:619)
	at clojure.core$require.doInvoke(core.clj:5496)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.lang.Var.invoke(Var.java:415)
	at clojure.main.main(main.java:36)
Caused by: java.lang.NullPointerException
	at clojure.core$reduce1.invoke(core.clj:896)
	at clojure.core$expand_method_impl_cache.invoke(core_deftype.clj:474)
	at clojure.core$_cache_protocol_fn.invoke(core_deftype.clj:542)
	at clojure.core.protocols$fn__5979$G__5974__5992.invoke(protocols.clj:13)
	at clojure.core$reduce.invoke(core.clj:6177)
	at clojure.core$into.invoke(core.clj:6229)
	at clojure.main__init.load(Unknown Source)
	at clojure.main__init.<clinit>(Unknown Source)
closed with the note: was a bug in J9, now fixed

3 Answers

+2 votes
by
selected by
 
Best answer

If anyone's interested, J9 bug report here for this: https://github.com/eclipse/openj9/issues/12191 (thanks Ghadi!)

by
The issue was traced to a bad change in j9 and reverted (see https://github.com/eclipse/openj9/issues/12191#issuecomment-799695842).

From the resolution:

"The fix delivered #12221 backports this to the 0.26 release which will be generated for Java 8, 11, and 16. We use the same JIT compiler for all of those JVM levels so the fix will be there in 0.26 releases.

This release is scheduled for April 23, 2021:
https://projects.eclipse.org/projects/technology.openj9/releases/0.26.0 "
0 votes
by

Problems with the J9 JVM have been discussed quite a bit in the last week on Slack and the consensus seemed to be that Clojure is not compatible with it (it is not tested against it). I just searched for J9 and found half a dozen folks reporting pretty much random behavior on the J9 JVM across a bunch of channels.

by
In case of fire, it's super to be in a place with 2 exits.  Having two mainstream certified JVMs is like that.  You don't try the fire escape every year.  But, if you try the fire escape, and it's locked, uh-oh.  Especially if everyone else uses the fire escape fine, it seems to be locked only for you, presumably because of your special brain implant.

I bring the matter here to provide details to start a Jira issue.  That's our way of establishing conditions favorable to discovery of a workaround, or a crisp upstream bug ticket.

Having `reduce1` cited as the immediate cause, noticeably often, in J9 versions 11 & 15, for Clojure versions as far back as 1.5.1, makes me hope that the failure can be narrowed and either worked around or brought to the attention of the J9 bug fixers.
by
I doubt reduce1 is special, more likely it shows up because it's almost the first lazy seq code to be run in the JVM. Would not surprise me if the npe is related to locals clearing bytecode, which is emitted by the Clojure compiler but unusual from a Java->JVM standpoint and involved in most functions traversing a lazy seq.
by
A promising theory. There's another stack trace where the point-of-problem was in `some`.
by
I linked this above but not sure what people get notified about exactly - https://github.com/eclipse/openj9/issues/12191
by
They just found one concrete issue causing this on their side: https://github.com/eclipse/openj9/issues/12191#issuecomment-797840254

Alex's initial guess turned out to be in the money :-) - and the issue isn't with Clojure itself.

There are other more subtle issues such as slight differences between hash functions - see the end of this article: https://www.royvanrijn.com/blog/2018/05/openj9-jvm-shootout/

Those subtle issues are going to be tough to root out of each dependency.  My opinion is that focusing on some of the important pieces of OpenJDK-16 - which also shows some impressive perf benefits at least with our stuff - will yield more ROI.
by
Clojure makes no guarantees on specific hash values so I don't see hash differences as a problem per se (as long as general guarantees hold - two things that are equal should have the same hash code).
by
Oh I agree that Clojure itself would work fine (and for that matter so does every project in the techascent system); that was one of my points above.  It is every *other* possible dependency in the JVM ecosystem that would concern me, personally.  

Mainly I was just impressed that you nailed it on the first try :-).
0 votes
by
by
They fixed it in openj9-0.26.0!  Thank you Clojure Squad for narrowing the problem and pursuing it with the Eclipse compiler writers!
Maintained by Cognitect on behalf of the Clojure community
...