Clojure-generated class names length exceed file-system limit

Question

Class names generated by the Clojure compiler can be arbitrarily long, exceeding the file system's maximum allowed file name length. For example it happens when you nest functions a bit too deeply:

(defmacro nestfn [n & body]
  (if (> n 0)
    `(fn [] (nestfn ~(- n 1) ~@body))
    body))

(def myf (nestfn 100 "body"))

Compiling this produces a {{java.io.IOException: File name too long}} exception.

Comments

This question was surprisingly hard to find, possibly because it doesn't mention, or tag, core.match, even though that's one of the more common culprits.

Has there been any progress? The ticket was marked "Critical".

---

I tagged it. It's not on our list for 1.12 but I have it in our list to look at in 1.13.

---

Just to add another use case, or cause for this bug: we hit the same limit by having too many nested `(for [] ,,,)` loops.

Answer 1

Comment made by: martinraison

The Scala community found this issue a while ago, and now the compiler has a {{max-classfile-name}} parameter (defaulting to 255). Hashing is used when the limit is exceeded. Maybe we should consider something similar?

Answer 2

Comment made by: panewman

I tried clojure.core.match with 13 patterns and the compiliation failed under Windows. I assume this problem is the root cause of it.

Answer 3

_Comment made by: chrisbetz

Some more info on that:

A colleague of mine just ran into that problem because he's using Linux / eCryptfs (where the limit of 143 is rather small, compared to our FileVault encrypted macOS used otherwise): see https://bugs.launchpad.net/ecryptfs/ bug/344878.

However, Clojure's not alone with that problem, Scala is also hit hard: https://issues.scala-lang.org/browse/SI-3623

There's no "easy" solution to this, and truncating the filename (as Scala does) bears a lot of other problems, obviously.

For all of you bitten by this problem, one possible workaround might be the one proposed by Mario Pastorelli (https://issues.scala-lang.org/secure/ViewProfile.jspa?name=melrief) in comment https://issues.scala-lang.org/browse/SI-3623?focusedCommentId=76104&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-76104:

Use an unencrypted filesystem to temporarily store classfiles, maybe by having a tempFS to keep stuff in memory. Not the best way, because we do not like leaving important stuff unencrypted, ...

Answer 4

Comment made by: alexvong1995

Hello,

I run into the same problem while using clojure.core.match. I macroexpand the function definition and observe that the macro-expanded definition is deeply nested.

I think one way to solve it is to provide an option to wrap the class file as a jar with some shorter file name so that the the class name can remain the same (zipped file name can be as long as you like, right?). WDYT?

Btw, I am using clojure 1.9.0 so I think we should say that this bug affects 1.9 as well.

Answer 5

Comment made by: alexvong1995

I come up with a proof-of-concept patch. The compiler now outputs jar instead of class if the class name is longer than 255. The jar name is simply a (left) truncation of the class name.

Suppose **compile-path* is set to "build", then you need to add build/ to your class path, so that the jars can be found.

This patch is only a proof of concept, ideally all classes with long name should be put into one big jar to avoid having to decompress many files. Also, the user should be able to specify **compile-name-max** and **compile-jar-name**. Finally, the code is quite ugly, I should have spitted things into several functions.

Answer 6

Comment made by: alexmiller

Alex - we're not going to output jars. This is at odds with many facets of the Clojure runtime.

Answer 7

Comment made by: steveminer@gmail.com

Sorry if this comment is off topic, but the original example is a bit confusing to me. Maybe it should be:

`
(defmacro nestfn [n & body]
(if (> n 0)

`(fn [] (nestfn ~(dec n) ~@body))
`(do ~@body)))

`

That way {{(trampoline (nestfn 10 "foo"))}} would return "foo". However, I do get a CompilerException java.lang.StackOverflowError for n=1000 on macOS.

Answer 8

Comment made by: gsnewmark

@Alex Miller Hello! Would a patch with Scala-like approach (hashing of name in case it's too long) be considered? As it breaks binary compatibility this workaround would definitely be disabled by default, but it would be possible to enable name hashing by using new compiler option. This approach would definitely help in our particular case, but I'm not sure if it's useful/generic enough to be included in compiler, so I haven't started working on patch yet.

Answer 9

_Comment made by: alexmiller_

Not going to make any breaking changes. Plan should be to switch second strategy when current strategy doesn’t work.

Answer 10

Reference: https://clojure.atlassian.net/browse/CLJ-1852 (reported by alex+import)