Could tools.namespace refresh get more sophisticated directory/file filtering?

Question

TL;DR:
currently the only way of affecting which source files are loaded by clojure.tools.namespace.repl/refresh is setting refresh dirs, which is an allowlist-based system. There is no way currently to block specific directories/files/patterns from being loaded but allowing everything else. This presents an issue when there are clojure source files in source directories on the classpath, which are not meant to be loaded at all.

Long story:

We bumped into an interesting issue with clojure.tools.namespace/refresh vs clj-kondo hooks .

A one-liner repro case is:

clj -Srepro -Sdeps '{:deps {org.clojure/tools.namespace {:mvn/version "1.2.0"} seancorfield/next.jdbc {:git/url "https://github.com/seancorfield/next-jdbc/" :git/sha "24bf1dbaa441d62461f980e9f880df5013f295dd"}}}' -M -e "((requiring-resolve 'clojure.tools.namespace.repl/refresh-all))"

This will fail with:

:error-while-loading hooks.com.github.seancorfield.next-jdbc
Could not locate hooks/com/github/seancorfield/next_jdbc__init.class, hooks/com/github/seancorfield/next_jdbc.clj or hooks/com/github/seancorfield/next_jdbc.cljc on classpath. Please check that namespaces with dashes use underscores in the Clojure file name.

For context, clj-kondo is a static analyzer/linter. In order to be able to analyze custom macros properly, it lets libraries distribute clj files (describing how these macros should be analyzed) as resources under a specific directory.

The above example fails due to the combination of the following:

• referring a lib by git coordinates, which means that it will appear as a directory on the classpath
• said lib exporting clj-kondo hooks
• clj-kondo hooks themselves being .clj files
• as these are for clj-kondo to use, they are not meant to be loaded in the app using the lib
• as a result these files normally having a namespace inside them that does not appear in code proper, for example: https://github.com/seancorfield/next-jdbc/blob/24bf1dbaa441d62461f980e9f880df5013f295dd/resources/clj-kondo.exports/com.github.seancorfield/next.jdbc/hooks/com/github/seancorfield/next_jdbc.clj#L1
• tools.namespace by default reloads all clj source files found in classpath-directories (referring back to the first point)

As there is currently no way to tell tools.namespace to not load certain files I had to come up with a somewhat hacky workaround, which tries to set the refresh dirs to classpath-directories minus problematic ones, but it would be much much nicer if there was a way to be able to set this with a blocklist or predicate.

Workaround in case anyone else bumps into it:

(defn remove-clj-kondo-exports-from-tools-ns-refresh-dirs
  "A potential issue from using this is that if the directory containing the clj-kondo.exports folder
  also directly contains to-be-reloaded clojure source files, those will no longer be reloaded."
  []
  (->> (clojure.java.classpath/classpath-directories)
       (mapcat
        (fn [^File classpath-directory]
          (let [children   (.listFiles classpath-directory)
                directory? #(.isDirectory ^File %)
                clj-kondo-exports?
                           #(= "clj-kondo.exports" (.getName ^File %))
                has-clj-kondo-exports
                           (some (every-pred clj-kondo-exports? directory?) children)]
            (if has-clj-kondo-exports
              (->> children
                   (filter directory?)
                   (remove clj-kondo-exports?))
              [classpath-directory]))))
       (apply clojure.tools.namespace.repl/set-refresh-dirs)))

;; call in user.clj
(remove-clj-kondo-exports-from-tools-ns-refresh-dirs)

Answer 1

This presents an issue when there are clojure source files in source directories on the classpath, which are not meant to be loaded at all.

This seems like a problem of your own construction? Aren't source files in the source directories inherently there to be loaded? Can't you just not do that?

Comments

Well, not really my own per se; I'm just trying to refresh in a project that pulls in next.jdbc through a git dep... And this is how next.jdbc provides hints to clj-kondo.

While I agree the primary case for clj files in source folders is to be loaded, I can imagine other scenarios too, like distributing example code files etc that way, which won't necessarily be able to be loaded in the app itself.

---

But why are those in :paths? The whole meaning of :paths is "paths containing source to be put on the classpath". Seems like these source files should not be there if they are not source. Projects could create aliases that add those paths if needed for some tool-specific purpose.

---

Thank you, that makes sense to me. I'll circle back to the authors of the tools in question.

That being said, I do like the possibility of setting up :exclusions in deps.edn in case a lib pulls in a dep I don't want/need - it sometimes helps protect my workflow/product from certain mistakes made by the lib author. It would be great if tools.namespace had that ability too but I understand if you don't think this current case is strong enough for that.

---

The answer to "why are they on the path" is in this Clojurians slack thread if anyone wonders: https://clojurians.slack.com/archives/CHY97NXE2/p1641423463308400?thread_ts=1641398954.304500&cid=CHY97NXE2

"The requirements are that these files are on the classpath under a clj-kondo.exports directory so clj-kondo knows what to copy to the local config directory"

---

Alex, do you think a patch providing this functionality would be accepted?

I was thinking of adding predicate-based path filtering to the end of this function: https://github.com/clojure/tools.namespace/blob/c0b333e127e14c2ac6d5b04d14d0e714d08bfdbb/src/main/clojure/clojure/tools/namespace/dir.clj#L28

similarly to how set-refresh-dirs works: https://github.com/clojure/tools.namespace/blob/master/src/main/clojure/clojure/tools/namespace/repl.clj#L164

It wouldn't be specific to my problem, it would be a way of saying "out of all the files found in refresh-dirs, don't consider the ones where this predicate applies"

---

No, this still doesn't make sense to me.

---

> Aren't source files in the source directories inherently there to be loaded?

> The whole meaning of :paths is "paths containing source to be put on the classpath". Seems like these source files should not be there if they are not source.

There appears to be a gap in my understanding, circling back to these earlier comments from you;

If I am a library author and I want to distribute some .clj files with my lib (it's important that they are included in the lib and not something auxiliary), which aren't meant to be loaded, what's the best way to do that when my lib is distributed as a) a jar b) a git dep?

---

All jar files (and the classpath defined by the :paths of a git dep) ARE on the classpath and any clj files in them ARE loadable by Clojure.

If you want to distribute other things you could (in Maven) publish artifacts via a different classifier (not "jar", but same mechanism that Java libs publish "source" and "javadoc" artifacts). If you don't want them to be loadable as clj source, you could also use a different file extension (like .edn or whatever you want - you can still to choose to load them as Clojure via the reader but they won't be see as Clojure namespaces).

---

Thank you for that, Alex.

Answer 2

There is now a clj-kondo issue to try and solve this from that side.