Clojure Q&A - Recent questions tagged tools.namespace

tools.namespace remove-node does not remove outgoing dependencies from :dependents

Mon, 20 Jan 2025 02:32:44 +0000

Problem

I noticed that when a dependency, example.one, was removed from another namespace, example.two, example.one still showed up under [::track/deps :dependents 'example.two] in the tracker after an invocation of repl/scan.

After digging into this for some time, I saw that the implementation of remove-node in MapDependencyGraph didn't make any changes to the :dependents attribute.

This seems like a mistake to me, as the remove-node docstring reads:

Removes the node from the dependency graph without removing it as a
dependency of other nodes. That is, removes all outgoing edges from
node.

The implementation of remove-node follows:

(remove-node [graph node]
  (MapDependencyGraph.
    (dissoc dependencies node)
    dependents))

This removes outgoing edges from dependencies, but if dependents is effectively an inverse of dependencies, then I'd expect to see some sort of removal from dependents as well. Instead, dependents remains unmodified.

Proposed Fix

If this is indeed a bug, and not a misunderstanding on my part, I have a fix here along with a couple tests on my behavioral-tests branch that removes node from values in the dependents map.

Does this align with the expected behavior of remove-node, or am I misunderstanding the intended behavior of this function?

CLR behavior of tools.namespace does not match the JVM implementation

Thu, 05 Dec 2024 14:44:05 +0000

I noticed a few behavioral differences between the CLR and JVM implementations of tools.namespace.

As an example, the following tests will pass in the JVM implementation and fail in the CLR implementation. In CLR, files-2 is empty and ::dir/time is an instant.

(ns clojure.tools.namespace.repl-test
  (:require [clojure.test :refer [deftest is]]
            [clojure.tools.namespace.dir :as dir]
            [clojure.tools.namespace.find :as find]
            [clojure.tools.namespace.repl :as repl]
            [clojure.tools.namespace.test-helpers :as help]))

(deftest t-repl-scan
  (let [dir        (help/create-temp-dir "t-repl-scan")
        _main-clj  (help/create-source dir 'example.main :clj '[example.one])
        _one-cljc  (help/create-source dir 'example.one :clj)
        _other-dir (help/create-temp-dir "t-repl-scan-other")
        files-1    (::dir/files (repl/scan {:platform find/clj}))
        files-2    (::dir/files (repl/scan {:platform find/clj}))]
    (is (not-empty files-1))
    (is (not-empty files-2))
    (is (= files-1 files-2))))

(deftest t-repl-scan-time
  (let [scan (repl/scan {:platform find/clj})]
    (is (integer? (::dir/time scan)))))

After discussing with other contributors, the question of intent came up and this test was proposed as something that correctly captures the intent of repl/scan. However, if this is the intended behavior, then a new problem arises as this test fails in the JVM implementation.

(deftest t-repl-scan
  (try
    (let [dir       (help/create-temp-dir "t-repl-scan")
          _main-clj (help/create-source dir 'example.main :clj '[example.one])
          _one-cljc (help/create-source dir 'example.one :clj)
          other-dir (help/create-temp-dir "t-repl-scan-other")
          _         (repl/set-refresh-dirs dir other-dir)
          scan1     (repl/scan {:platform find/clj})
          scan2     (repl/scan {:platform find/clj})
          files-1   (::dir/files scan1)
          files-2   (::dir/files scan2)]
      (is (= (count files-1) 2))
      (is (= (count files-2) 0)))
    (finally (repl/clear))))

A couple things I think would help here:
1. Flesh out the tools.namespace (JVM) test suite to capture all the intended behavior.
2. Realign clr.tools.namespace with tools.namespace.

Point 1 will provide a good baseline for point 2, as well as any other existing/future ports of tools.namespace and help prevent things from diverging in the future as changes are made.

Note

As of writing, repl/scan will throw an InvalidCastException on CLR. To fix this, replace clojure.tools.namespace.dir/modified-files with this:

(defn- modified-files [tracker files]
  (filter #(DateTime/op_LessThan ^DateTime (::time tracker System.DateTime/UnixEpoch) (.get_LastWriteTimeUtc ^FileSystemInfo %)) files))

clojure.tools.namespace throws casting error when calling find-ns-decls-in-jarfile

Tue, 07 Mar 2023 17:15:20 +0000

If I call find-ns-decls-in-jarfile jarfile (example below) with any jar I get the exception:

Error printing return value (ClassCastException) at clojure.tools.namespace.find/read-ns-decl-from-jarfile-entry (find.clj:158).
class java.util.jar.JarFile cannot be cast to class java.io.File (java.util.jar.JarFile and java.io.File are in module java.base of loader 'bootstrap')

I can reproduce this with this example:


    (let [filename (format "%s/.m2/repository/org/clojure/tools.namespace/1.4.2/tools.namespace-1.4.2.jar" (System/getenv "HOME"))
      jarfile (JarFile. (io/file filename))]
  (clojure.tools.namespace.find/find-ns-decls-in-jarfile jarfile))

I think the issue is the type hint on line 159. I think this should instead be ^java.util.jar.JarFile.

How to restrict namespaces reloaded by tools.namespace to only dependencies of a root ns?

Fri, 03 Mar 2023 20:17:19 +0000

Hey,

At my organisation, we use Component and would like to start using the reloaded workflow to reload the code more easily.

The problem we’re facing is that our dev classpath contains too much. In addition to the app proper (loaded by (require 'our-org.repl)), there are tests and some one-off code: migrations and illustrative REPL sessions that we keep around as documentation.

Some of these auxiliary namespaces have names that don’t match their location in the filesystem, so they are not loaded correctly by tools.namespace. We’d like to prevent them from being reloaded at all, even though they are present on the classpath.

We could use clojure.tools.namespace.repl/set-refresh-dirs, but I feel this is clunky (need to explicitly opt-in for directories) and not quite expressive enough. What we’d really want is to reload only the namespaces that are (possibly indirect) dependencies of our-org.repl.

To this end, I’ve monkey-patched tools-namespace:

(defn relevant-deps [deps ns]
  (conj
   (ns.dependency/transitive-dependencies deps ns)
   ns))

(alter-var-root
 #'ns.repl/remove-disabled
 (fn [orig-impl]
   (fn [{:keys [::ns.track/deps] :as tracker}]
     (-> tracker
         (orig-impl)
         (update ::ns.track/unload (partial filter (loaded-libs)))
         (update ::ns.track/load
                 #(set/intersection (set %1) %2)
                 (relevant-deps deps 'our-org.repl))))))

This works, but is obviously kludgy. Is there a better way to achieve the same goal? Perhaps the functionality of restricting dependencies could be added to tools.namespace?

Should tools.namespace find still cope with namespace-less sources?

Tue, 21 Feb 2023 20:17:47 +0000

Issue
Tools.namespace find 1.4.1 does not handle sources that don't include a ns declaration.

Reproduction
Given file ./src/foo.clj:

;; no ns declaration here

And test script ./test.clj:

(require '[clojure.tools.namespace.find :as f]
         '[clojure.java.io :as io])

(println "found nses:" (f/find-ns-decls-in-dir (io/file "src")))

We can see that tools.namespace 1.4.0 returns an empty sequence for this scenario:

$ clj -Sdeps '{:deps {org.clojure/tools.namespace {:mvn/version "1.4.0"}}}' -M --report stderr test.clj
found nses: ()

But version 1.4.1 throws an exception for this same test:

$ clj -Sdeps '{:deps {org.clojure/tools.namespace {:mvn/version "1.4.1"}}}' -M --report stderr test.clj
{:clojure.main/message
 "Execution error (NullPointerException) at clojure.tools.namespace.find/find-ns-decls-in-dir$fn (find.clj:93).\nCannot invoke \"clojure.lang.IObj.withMeta(clojure.lang.IPersistentMap)\" because \"x\" is null\n",
 :clojure.main/triage
 {:clojure.error/class java.lang.NullPointerException,
  :clojure.error/line 93,
  :clojure.error/cause
  "Cannot invoke \"clojure.lang.IObj.withMeta(clojure.lang.IPersistentMap)\" because \"x\" is null",
  :clojure.error/symbol
  clojure.tools.namespace.find/find-ns-decls-in-dir$fn,
  :clojure.error/source "find.clj",
  :clojure.error/phase :execution},
 :clojure.main/trace
 {:via
  [{:type clojure.lang.Compiler$CompilerException,
    :message
    "Syntax error macroexpanding at (/home/lee/proj/oss/-verify/tns-test/test.clj:4:1).",
    :data
    {:clojure.error/phase :execution,
     :clojure.error/line 4,
     :clojure.error/column 1,
     :clojure.error/source
     "/home/lee/proj/oss/-verify/tns-test/test.clj"},
    :at [clojure.lang.Compiler load "Compiler.java" 7665]}
   {:type java.lang.NullPointerException,
    :message
    "Cannot invoke \"clojure.lang.IObj.withMeta(clojure.lang.IPersistentMap)\" because \"x\" is null",
    :at [clojure.core$with_meta__5485 invokeStatic "core.clj" 220]}],
  :trace
  [[clojure.core$with_meta__5485 invokeStatic "core.clj" 220]
   [clojure.core$with_meta__5485 invoke "core.clj" 219]
   [clojure.tools.namespace.find$find_ns_decls_in_dir$fn__1437
    invoke
    "find.clj"
    93]
   [clojure.core$keep$fn__8649 invoke "core.clj" 7409]
   [clojure.lang.LazySeq sval "LazySeq.java" 42]
   [clojure.lang.LazySeq seq "LazySeq.java" 51]
   [clojure.lang.RT seq "RT.java" 535]
   [clojure.core$seq__5467 invokeStatic "core.clj" 139]
   [clojure.core$print_sequential invokeStatic "core_print.clj" 53]
   [clojure.core$fn__7391 invokeStatic "core_print.clj" 174]
   [clojure.core$fn__7391 invoke "core_print.clj" 174]
   [clojure.lang.MultiFn invoke "MultiFn.java" 234]
   [clojure.core$pr_on invokeStatic "core.clj" 3675]
   [clojure.core$pr invokeStatic "core.clj" 3678]
   [clojure.core$pr invoke "core.clj" 3678]
   [clojure.lang.AFn applyToHelper "AFn.java" 154]
   [clojure.lang.RestFn applyTo "RestFn.java" 132]
   [clojure.core$apply invokeStatic "core.clj" 667]
   [clojure.core$pr invokeStatic "core.clj" 3691]
   [clojure.core$pr doInvoke "core.clj" 3678]
   [clojure.lang.RestFn applyTo "RestFn.java" 139]
   [clojure.core$apply invokeStatic "core.clj" 667]
   [clojure.core$prn invokeStatic "core.clj" 3715]
   [clojure.core$prn doInvoke "core.clj" 3715]
   [clojure.lang.RestFn applyTo "RestFn.java" 137]
   [clojure.core$apply invokeStatic "core.clj" 667]
   [clojure.core$println invokeStatic "core.clj" 3734]
   [clojure.core$println doInvoke "core.clj" 3734]
   [clojure.lang.RestFn invoke "RestFn.java" 421]
   [user$eval1480 invokeStatic "test.clj" 4]
   [user$eval1480 invoke "test.clj" 4]
   [clojure.lang.Compiler eval "Compiler.java" 7194]
   [clojure.lang.Compiler load "Compiler.java" 7653]
   [clojure.lang.Compiler loadFile "Compiler.java" 7591]
   [clojure.main$load_script invokeStatic "main.clj" 475]
   [clojure.main$script_opt invokeStatic "main.clj" 535]
   [clojure.main$script_opt invoke "main.clj" 530]
   [clojure.main$main invokeStatic "main.clj" 664]
   [clojure.main$main doInvoke "main.clj" 616]
   [clojure.lang.RestFn applyTo "RestFn.java" 137]
   [clojure.lang.Var applyTo "Var.java" 705]
   [clojure.main main "main.java" 40]],
  :cause
  "Cannot invoke \"clojure.lang.IObj.withMeta(clojure.lang.IPersistentMap)\" because \"x\" is null",
  :phase :execution}}

Execution error (NullPointerException) at clojure.tools.namespace.find/find-ns-decls-in-dir$fn (find.clj:93).
Cannot invoke "clojure.lang.IObj.withMeta(clojure.lang.IPersistentMap)" because "x" is null

Usage Scenario

Cljdoc uses tools.namespace to find namespaces during API analysis.

Various libs include all sorts of weirdnesses including namespace-less sources.

Previous versions of tools.namespace handled this weirdness for us.

We can certainly work around this if it is decided that tools.namespace should not handle namespace-less sources.

Exposing namespaces that will be reloaded from c.t.n.repl

Fri, 23 Sep 2022 22:42:10 +0000

Hello! I've recently copied some logic out of clojure.tools.namespace.repl that determines the next set of namespaces that need refreshing: https://github.com/clojure/tools.namespace/blob/master/src/main/clojure/clojure/tools/namespace/repl.clj#L91 This information is highly useful to external consumers who may need to manage some stateful resources prior to refreshing, depending on their state management solution.

I opened a PR here, which was closed because I apparently can't read contribution guidelines: https://github.com/clojure/tools.namespace/pull/13 This just pulls out functions that are invoked via alter-var-root to a standalone function. (vemv also helpfully pointed out that I forgot to propagate an argument).

Would this be a reasonable, small change to make to c.t.n.repl?

Thank you :)

tools.namespace does not recognise ns deps implied by data_readers.cljc

Fri, 29 Apr 2022 14:52:37 +0000

i found an interesting issue in our codebase around some tagged-literals seemingly violating the principle of least surprise

we have been creating some pure-data descriptions of processes including some defrecord objects from a custom tagged-literal reader declared in data_readers.cljc, and having those defrecord objects participate in a protocol. it's all been working just fine until i ran across this issue

someone had put one of these tagged literals into a def (which seemed like a reasonable thing to do at first blush):

(def foo #ctx/event-path [:blah])

which was ok... until c.t.n.r/refresh was called, after which point everything broke

it turned out that the defrecord object foo had a stale class (presumably because the namespace dependencies induced by the tagged-literal parsers declared in data_readers.cljc are not recognised by tools.namespace, and namespaces were therefore recompiled out of order), and so no longer participated in the protocol

should tools.namespace be able to recognise data_readers.cljc induced dependencies ? or is that a step too far for it ?

Could tools.namespace refresh get more sophisticated directory/file filtering?

Wed, 05 Jan 2022 18:03:20 +0000

TL;DR:
currently the only way of affecting which source files are loaded by clojure.tools.namespace.repl/refresh is setting refresh dirs, which is an allowlist-based system. There is no way currently to block specific directories/files/patterns from being loaded but allowing everything else. This presents an issue when there are clojure source files in source directories on the classpath, which are not meant to be loaded at all.

Long story:

We bumped into an interesting issue with clojure.tools.namespace/refresh vs clj-kondo hooks .

A one-liner repro case is:

clj -Srepro -Sdeps '{:deps {org.clojure/tools.namespace {:mvn/version "1.2.0"} seancorfield/next.jdbc {:git/url "https://github.com/seancorfield/next-jdbc/" :git/sha "24bf1dbaa441d62461f980e9f880df5013f295dd"}}}' -M -e "((requiring-resolve 'clojure.tools.namespace.repl/refresh-all))"

This will fail with:

:error-while-loading hooks.com.github.seancorfield.next-jdbc
Could not locate hooks/com/github/seancorfield/next_jdbc__init.class, hooks/com/github/seancorfield/next_jdbc.clj or hooks/com/github/seancorfield/next_jdbc.cljc on classpath. Please check that namespaces with dashes use underscores in the Clojure file name.

For context, clj-kondo is a static analyzer/linter. In order to be able to analyze custom macros properly, it lets libraries distribute clj files (describing how these macros should be analyzed) as resources under a specific directory.

The above example fails due to the combination of the following:

referring a lib by git coordinates, which means that it will appear as a directory on the classpath
said lib exporting clj-kondo hooks
clj-kondo hooks themselves being .clj files
as these are for clj-kondo to use, they are not meant to be loaded in the app using the lib
as a result these files normally having a namespace inside them that does not appear in code proper, for example: https://github.com/seancorfield/next-jdbc/blob/24bf1dbaa441d62461f980e9f880df5013f295dd/resources/clj-kondo.exports/com.github.seancorfield/next.jdbc/hooks/com/github/seancorfield/next_jdbc.clj#L1
tools.namespace by default reloads all clj source files found in classpath-directories (referring back to the first point)

As there is currently no way to tell tools.namespace to not load certain files I had to come up with a somewhat hacky workaround, which tries to set the refresh dirs to classpath-directories minus problematic ones, but it would be much much nicer if there was a way to be able to set this with a blocklist or predicate.

Workaround in case anyone else bumps into it:

(defn remove-clj-kondo-exports-from-tools-ns-refresh-dirs
  "A potential issue from using this is that if the directory containing the clj-kondo.exports folder
  also directly contains to-be-reloaded clojure source files, those will no longer be reloaded."
  []
  (->> (clojure.java.classpath/classpath-directories)
       (mapcat
        (fn [^File classpath-directory]
          (let [children   (.listFiles classpath-directory)
                directory? #(.isDirectory ^File %)
                clj-kondo-exports?
                           #(= "clj-kondo.exports" (.getName ^File %))
                has-clj-kondo-exports
                           (some (every-pred clj-kondo-exports? directory?) children)]
            (if has-clj-kondo-exports
              (->> children
                   (filter directory?)
                   (remove clj-kondo-exports?))
              [classpath-directory]))))
       (apply clojure.tools.namespace.repl/set-refresh-dirs)))

;; call in user.clj
(remove-clj-kondo-exports-from-tools-ns-refresh-dirs)

Resolution for bug TNS-6 in clojure.tools.namespace

Mon, 24 May 2021 17:39:12 +0000

Hi there,

So I believe I have a resolution for TNS-6 in clojure.tools.namespace. I have a consistent repro and a resolution in the form of a one-line patch (see the patch for more detail). All the tests still pass, though I'm not sure they cover the changes. I have also tested the resolution in a number of local dependent repos, and it resolves the issue.

However, I am not a Clojure contributor so I cannot comment or submitt the patch or comment on relevant Jira ticket.

From b5a62f894e049a09d5fcf099ac7b4d6591fb4f18 Mon Sep 17 00:00:00 2001
From: zalky <zalan.k@gmail.com>
Date: Mon, 24 May 2021 13:08:30 -0400
Subject: [PATCH] Fix TNS-6: fully remove ns from deps graph

The clojure.tools.namespace.dependency/remove-node fn removes a
namespace's :dependencies set from the graph (outgoing edges), but
does not remove that namespace from the :dependents set of other
namespaces (ingoing edges). If a namespace is removed and one of its
dependencies is changed at the same time, as commonly occurs when
switching branches, then the removed namespace is still in the
:dependents of the changed namespace when it is reloaded. As all the
of the dependents of a changed namespace are also reloaded, this
throws a missing namespace error.

The resolution is to use clojure.tools.namespace.depedency/remove-all
to remove both outgoing (:dependencies) and ingoing (:dependents)
edges in the dependency graph.

However, care must be taken when choosing the point of change, because
clojure.tools.namespace.track/add relies on the partial removal
behaviour of clojure.tools.namespace.track/remove-deps. It uses
remove-deps to wipe the dependencies of a node, and cannot know how to
re-insert dependents if lost. Therefore the point of change should be
in clojure.tools.namespace.track/remove.
---
 src/main/clojure/clojure/tools/namespace/track.cljc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/main/clojure/clojure/tools/namespace/track.cljc b/src/main/clojure/clojure/tools/namespace/track.cljc
index 683c35d..ce666d5 100644
--- a/src/main/clojure/clojure/tools/namespace/track.cljc
+++ b/src/main/clojure/clojure/tools/namespace/track.cljc
@@ -96,7 +96,7 @@
          :or {load (), unload (), deps (dep/graph)}} tracker
         known (set (dep/nodes deps))
         removed-names (filter known names)
-        new-deps (remove-deps deps removed-names)
+        new-deps (reduce dep/remove-all deps removed-names)
         changed (affected-namespaces deps removed-names)]
     (assoc tracker
       ::deps new-deps
-- 
2.30.1 (Apple Git-130)

[Patch updated after further testing]

Would be nice if tools.namespace.find functions return file info in ns-decl meta

Tue, 05 Nov 2019 13:35:01 +0000

Currently calling tools.namespace.find/find-ns-decls returns a seq of ns-decl for a given dir or collection of paths but there is no way to know where did they came from (can also be coming from paths inside jar files).

Looks pretty straight forward to add file info as meta of the ns-decl form. Something like https://github.com/jpmonettas/tools.namespace/commit/1431215324ba7f76f78342742ce3716ec8c5abb8

What do you guys think?

(Filed jira at https://clojure.atlassian.net/browse/TNS-55)

How does tools.namespace.track/add is supposed to work?

Fri, 18 Oct 2019 21:22:25 +0000

I'm trying to figure out if I can use tools.namespace.track for a project but I don't understand the :unload :load functionality.

(require '[clojure.tools.namespace.track :as ns-track])

;; define a empty tracker and add some dependencies
(def tracker (-> (ns-track/tracker)
                 (ns-track/add '{alpha #{beta}
                                 beta  #{gamma delta}})))

;; now add a new dependency and check what we need to unload and load
(-> tracker
    (ns-track/add '{epsilon #{}})
    (select-keys [:clojure.tools.namespace.track/unload
                  :clojure.tools.namespace.track/load]))

#:clojure.tools.namespace.track{:unload (epsilon alpha beta),
                                :load (epsilon beta alpha)}

I don't understand why alpha and beta are required to unload/load but maybe I'm missing something.

I'm using the latest version.

Thanks!

Java 11 and Tools Namespace

Wed, 31 Jul 2019 14:50:56 +0000

I've run into an issue where refresh from tools.namespace (v. 0.3.0) seems to work with Java 8 but not Java 11. I tried with both Clojure 1.10.0 and 1.10.1. I was able to reproduce the problem on a clean virtual machine. In both cases, I was able to reload the namespaces on Java 8 but not Java 11.

It's weird though -- in some of my other projects, I can reload namespaces on Java 11. I haven't been able to track down the difference. (I'm also new to tools.deps, so it could be I'm doing something wrong there.)

I put together a reproduction here: https://github.com/thomascothran/namespace-repro

I'm assuming it's likely an error on my end, but the only thing I've been able to narrow it down to is the Java version.

Reference: https://clojure.atlassian.net/browse/TNS-54