What is a good place to start for implementing a new Clojure backend?

Question

I've done some digging around for a spec, test suite, or some scraps of documentation to inform the development of a Clojure backend. Most of my searching has only turned up other implementations that are mostly doing their own thing - even ClojureScript doesn't seem to have any way to verify conformance with what Clojure should be. And from what I understand, ClojureCLR works largely because it was a (mostly) one-to-one clone of the Java source code in C#.

Is there a preferred, or maintainer recommended, way to go about developing a new backend for Clojure? Or is that considered 'wild' territory?

Answer 1

I think this is wild (or at least under documented) territory. There are no books, documents ,etc. There are working experience reports from existing projects (CLJS, Clojure JVM/CLR, clojerl, joker, maybe others). Some other libraries have leveraged the cljs compiler to target gambit scheme, C in one case, etc.

The basic strategy appears to be to target the cljs implementation, since it's very amenable to bootstrapping due to a design based on protocols as opposed to clojure jvm/clr somewhat tighter integration with the host via the respective RT classes. That is effectively what the clojerl developer did. The benefit therein is that you now have access to the clojure testing suite. Since there is not an official specification, conformance with the tests are the next best thing. There are some recommendations in the core documentation about implementing platform specific reader conditions (e.g. use fully qualified keys for your implementation, reserved unqualified keys for official implementations).

There are readers, analyzers, and emitters written entirely in clojure (clojure jvm, not cljc though). It's conceivable to target a minimal subset of clojure necessary to bootstrap these things, then once they are running, load the rest of clojure (e.g. cljs.core). The only tricky part is that the reader, analyzer, and emitter themselves leverage a bit more advanced clojure idioms, which means you have to have those implemented first (e.g. multimethods and protocols and persistent structures) to use the existing tool chain as is. There could be motivation for simplifying the existing tooling into even simpler implementations that can stand atop even less infrastructure, to make the host implementation experience simpler/faster. There may be an obvious clojure0 that provides a simple bootstrapping layer that the higher level stuff can be defined on top of. I think those are open research topics at the moment (community drive seems to have accepted the existing implementations as sufficient for the time being).

While there are not specific "clojure" references, there are many generic Lisp references for implementing varieties of lisps. Lisp in Small Pieces, Lisp System Implementation, some excerpts from PAIP by Norvig, Make A Lisp, Scheme 48, PicoLisp, FemtoLisp, and broader research probably provide excellent insight into this kind of endeavor. I am sure there are far more excellent references than listed as well (lisp has a very long tradition of this kind of stuff...).