Clojure Q&A - Recent activity in tools.namespace

Commented: tools.namespace remove-node does not remove outgoing dependencies from :dependents

Sat, 08 Feb 2025 19:42:53 +0000

Sorry for the late reply here, things got busy the past couple weeks.

These are great ideas – I hadn't considered using grep on the git history.

Looks like there are only two usages of remove-node in tools.namespace (I still need to check on other clojure projects):
1. dependency/topo-sort
  - The change I made doesn't have any effect here, since the function already calls remove-edge elsewhere with each node and its own dependencies.
2. track/remove-deps > track/add
  - There are other functions much higher up the chain (dir/scan-files, dir/scan-dirs, repl/scan) where I noticed something off with the behavior. These are what I created the "proposed fix" tests against.

A search on "remove-node" in git history brought back a commit from 2021. Looks more related to some upstream functionality than to remove-node itself, but the author did provide a good bit of context:

commit 5aaa6a328c8aaf0ae8ea7e5c48736eb2f1825392
Author: zalky <zalan.k@gmail.com>
Date:   Mon May 24 13:08:30 2021 -0400

    Fix TNS-6: fully remove ns from deps graph

    The clojure.tools.namespace.dependency/remove-node fn removes a
    namespace's :dependencies set from the graph (outgoing edges), but
    does not remove that namespace from the :dependents set of other
    namespaces (ingoing edges). If a namespace is removed and one of its
    dependencies is changed at the same time, as commonly occurs when
    switching branches, then the removed namespace is still in the
    :dependents of the changed namespace when it is reloaded. As all the
    of the dependents of a changed namespace are also reloaded, this
    throws a missing namespace error.

    The resolution is to use clojure.tools.namespace.depedency/remove-all
    to remove both outgoing (:dependencies) and ingoing (:dependents)
    edges in the dependency graph.

    However, care must be taken when choosing the point of change, because
    clojure.tools.namespace.track/add relies on the partial removal
    behaviour of clojure.tools.namespace.track/remove-deps. It uses
    remove-deps to wipe the dependencies of a node, and cannot know how to
    re-insert dependents if lost. Therefore the point of change should be
    in clojure.tools.namespace.track/remove.

I am now leaning toward your suggestion of a new operation since there is this precedence of not changing the remove-node function.

Another possible solution that would solve the heart of the problem (repl/scan) would be to update the remove-deps function in clojure.tools.namespace.track with something like this:

(defn- remove-edges [deps node]
  (->> (dep/immediate-dependencies deps node)
       (reduce #(dep/remove-edge %1 node %2) deps)))

(defn- remove-deps [deps names]
  (reduce remove-edges deps names))

Retagged: tools.namespace no longer returns declared metadata on namespace

Tue, 30 May 2023 04:56:06 +0000

Problem

v1.4.0 of tools.namespace returns metadata from namespace declarations. Newer versions do not.

Reproduction

Given ./src/foo.clj:

(ns ^{:doc "foo" :no-doc true} foo)

And ./test.clj:

(require '[clojure.tools.namespace.find :as f]
         '[clojure.java.io :as io])

(binding [*print-meta* true]
  (pr (f/find-ns-decls-in-dir (io/file "src"))))

If I use v1.4.0, I get the foo ns :doc and :no-doc metadata:

❯ clj -Sdeps '{:deps {org.clojure/tools.namespace {:mvn/version "1.4.0"}}}' -M test.clj
((ns ^{:doc "foo", :no-doc true} foo))

But if I use v1.4.4, the ns metadata includes the new generated :dir and :file metadata from tools.namespace, but not the :doc and :no-doc metadata:

❯ clj -Sdeps '{:deps {org.clojure/tools.namespace {:mvn/version "1.4.4"}}}' -M test.clj
((ns ^{:dir "src", :file "foo.clj"} foo))

This has been true since v1.4.1 when the jar/dir and file metadata feature was introduced:

❯ clj -Sdeps '{:deps {org.clojure/tools.namespace {:mvn/version "1.4.1"}}}' -M test.clj
((ns ^{:dir "src", :file "foo.clj"} foo))

Additional Info

The same issue should exhibit for find-ns-decls-in-jarfile.

Usage Scenario

Cljdoc uses tools.namespace to learn about namespaces before attempting to load them. It skips namespaces that have :no-doc and/or :skip-wiki metadata.

Commented: clojure.tools.namespace throws casting error when calling find-ns-decls-in-jarfile

Mon, 13 Mar 2023 23:00:11 +0000

Great, thanks!

Answered: How to restrict namespaces reloaded by tools.namespace to only dependencies of a root ns?

Sat, 04 Mar 2023 22:36:00 +0000

Retagged: Should tools.namespace find still cope with namespace-less sources?

Wed, 01 Mar 2023 16:37:51 +0000

Issue
Tools.namespace find 1.4.1 does not handle sources that don't include a ns declaration.

Reproduction
Given file ./src/foo.clj:

;; no ns declaration here

And test script ./test.clj:

(require '[clojure.tools.namespace.find :as f]
         '[clojure.java.io :as io])

(println "found nses:" (f/find-ns-decls-in-dir (io/file "src")))

We can see that tools.namespace 1.4.0 returns an empty sequence for this scenario:

$ clj -Sdeps '{:deps {org.clojure/tools.namespace {:mvn/version "1.4.0"}}}' -M --report stderr test.clj
found nses: ()

But version 1.4.1 throws an exception for this same test:

$ clj -Sdeps '{:deps {org.clojure/tools.namespace {:mvn/version "1.4.1"}}}' -M --report stderr test.clj
{:clojure.main/message
 "Execution error (NullPointerException) at clojure.tools.namespace.find/find-ns-decls-in-dir$fn (find.clj:93).\nCannot invoke \"clojure.lang.IObj.withMeta(clojure.lang.IPersistentMap)\" because \"x\" is null\n",
 :clojure.main/triage
 {:clojure.error/class java.lang.NullPointerException,
  :clojure.error/line 93,
  :clojure.error/cause
  "Cannot invoke \"clojure.lang.IObj.withMeta(clojure.lang.IPersistentMap)\" because \"x\" is null",
  :clojure.error/symbol
  clojure.tools.namespace.find/find-ns-decls-in-dir$fn,
  :clojure.error/source "find.clj",
  :clojure.error/phase :execution},
 :clojure.main/trace
 {:via
  [{:type clojure.lang.Compiler$CompilerException,
    :message
    "Syntax error macroexpanding at (/home/lee/proj/oss/-verify/tns-test/test.clj:4:1).",
    :data
    {:clojure.error/phase :execution,
     :clojure.error/line 4,
     :clojure.error/column 1,
     :clojure.error/source
     "/home/lee/proj/oss/-verify/tns-test/test.clj"},
    :at [clojure.lang.Compiler load "Compiler.java" 7665]}
   {:type java.lang.NullPointerException,
    :message
    "Cannot invoke \"clojure.lang.IObj.withMeta(clojure.lang.IPersistentMap)\" because \"x\" is null",
    :at [clojure.core$with_meta__5485 invokeStatic "core.clj" 220]}],
  :trace
  [[clojure.core$with_meta__5485 invokeStatic "core.clj" 220]
   [clojure.core$with_meta__5485 invoke "core.clj" 219]
   [clojure.tools.namespace.find$find_ns_decls_in_dir$fn__1437
    invoke
    "find.clj"
    93]
   [clojure.core$keep$fn__8649 invoke "core.clj" 7409]
   [clojure.lang.LazySeq sval "LazySeq.java" 42]
   [clojure.lang.LazySeq seq "LazySeq.java" 51]
   [clojure.lang.RT seq "RT.java" 535]
   [clojure.core$seq__5467 invokeStatic "core.clj" 139]
   [clojure.core$print_sequential invokeStatic "core_print.clj" 53]
   [clojure.core$fn__7391 invokeStatic "core_print.clj" 174]
   [clojure.core$fn__7391 invoke "core_print.clj" 174]
   [clojure.lang.MultiFn invoke "MultiFn.java" 234]
   [clojure.core$pr_on invokeStatic "core.clj" 3675]
   [clojure.core$pr invokeStatic "core.clj" 3678]
   [clojure.core$pr invoke "core.clj" 3678]
   [clojure.lang.AFn applyToHelper "AFn.java" 154]
   [clojure.lang.RestFn applyTo "RestFn.java" 132]
   [clojure.core$apply invokeStatic "core.clj" 667]
   [clojure.core$pr invokeStatic "core.clj" 3691]
   [clojure.core$pr doInvoke "core.clj" 3678]
   [clojure.lang.RestFn applyTo "RestFn.java" 139]
   [clojure.core$apply invokeStatic "core.clj" 667]
   [clojure.core$prn invokeStatic "core.clj" 3715]
   [clojure.core$prn doInvoke "core.clj" 3715]
   [clojure.lang.RestFn applyTo "RestFn.java" 137]
   [clojure.core$apply invokeStatic "core.clj" 667]
   [clojure.core$println invokeStatic "core.clj" 3734]
   [clojure.core$println doInvoke "core.clj" 3734]
   [clojure.lang.RestFn invoke "RestFn.java" 421]
   [user$eval1480 invokeStatic "test.clj" 4]
   [user$eval1480 invoke "test.clj" 4]
   [clojure.lang.Compiler eval "Compiler.java" 7194]
   [clojure.lang.Compiler load "Compiler.java" 7653]
   [clojure.lang.Compiler loadFile "Compiler.java" 7591]
   [clojure.main$load_script invokeStatic "main.clj" 475]
   [clojure.main$script_opt invokeStatic "main.clj" 535]
   [clojure.main$script_opt invoke "main.clj" 530]
   [clojure.main$main invokeStatic "main.clj" 664]
   [clojure.main$main doInvoke "main.clj" 616]
   [clojure.lang.RestFn applyTo "RestFn.java" 137]
   [clojure.lang.Var applyTo "Var.java" 705]
   [clojure.main main "main.java" 40]],
  :cause
  "Cannot invoke \"clojure.lang.IObj.withMeta(clojure.lang.IPersistentMap)\" because \"x\" is null",
  :phase :execution}}

Execution error (NullPointerException) at clojure.tools.namespace.find/find-ns-decls-in-dir$fn (find.clj:93).
Cannot invoke "clojure.lang.IObj.withMeta(clojure.lang.IPersistentMap)" because "x" is null

Usage Scenario

Cljdoc uses tools.namespace to find namespaces during API analysis.

Various libs include all sorts of weirdnesses including namespace-less sources.

Previous versions of tools.namespace handled this weirdness for us.

We can certainly work around this if it is decided that tools.namespace should not handle namespace-less sources.

Answered: Exposing namespaces that will be reloaded from c.t.n.repl

Thu, 29 Sep 2022 20:21:35 +0000

Created jira at https://clojure.atlassian.net/browse/TNS-58

Answered: Could tools.namespace refresh get more sophisticated directory/file filtering?

Mon, 09 May 2022 15:24:00 +0000

There is now a clj-kondo issue to try and solve this from that side.

Commented: tools.namespace does not recognise ns deps implied by data_readers.cljc

Mon, 09 May 2022 12:00:22 +0000

hmm - that's a killer point @alexmiller - which means that my understanding of why things are failing after `c.t.n.r/refresh` (but never in prod) is flawed

back to the drawing board

Answer selected: Resolution for bug TNS-6 in clojure.tools.namespace

Thu, 27 May 2021 03:17:11 +0000

Are you interested in becoming a contributor and submitting the patch? If so, please sign the Contributor Agreement and file a support request to get a jira account per https://clojure.org/dev/dev#_becoming_a_contributor.

If not, then I can't actually look at your patch, but perhaps you could describe the problem and solution and a contributor can write a clean patch.

Commented: Preventing `clojure.tools.namespace.repl/refresh` from loading namespaces that haven't been loaded before

Fri, 02 Apr 2021 17:07:59 +0000

> It's not gone at all if the two files are in the same dir.

It is poor practice to place non-loadable namespace under `src/` or `test/`. It directly affects other users and tools.

If you have non-loadable namespaces, it's better to place them under `examples/`, `drafts/` or such.

Answered: Should tools.namespace.repl/refresh exclude code found in ~/.gitlibs?

Fri, 02 Apr 2021 02:12:45 +0000

It is a really good idea to set-refresh-dirs in advance. Examples, production-oriented scripts, drafted code, etc - none of those tend to be safe to require blindly (which is what (refresh) does when unconfigured).

Edited: Would be nice if tools.namespace.find functions return file info in ns-decl meta

Thu, 14 Nov 2019 15:01:40 +0000

Currently calling tools.namespace.find/find-ns-decls returns a seq of ns-decl for a given dir or collection of paths but there is no way to know where did they came from (can also be coming from paths inside jar files).

Looks pretty straight forward to add file info as meta of the ns-decl form. Something like https://github.com/jpmonettas/tools.namespace/commit/1431215324ba7f76f78342742ce3716ec8c5abb8

What do you guys think?

(Filed jira at https://clojure.atlassian.net/browse/TNS-55)

Commented: How does tools.namespace.track/add is supposed to work?

Sun, 27 Oct 2019 14:42:55 +0000

Already figured out that you need to dissoc unload and loads when processing them, I was confused because I was expecting a tracker api for this. Thanks!

Commented: Java 11 and Tools Namespace

Fri, 13 Sep 2019 21:11:14 +0000

The problem, as Juraj Martinka has identified, is with `(clojure.java.classpath/classpath)` in certain contexts. I have added my own comments at https://github.com/clojure-emacs/cider/issues/2686#issuecomment-531366583

There have been multiple workarounds in java.classpath for different environments. The current behavior was added in https://clojure.atlassian.net/browse/CLASSPATH-8 to work around changes to the classloader hierarchy in Java 9. The relevant lines are https://github.com/clojure/java.classpath/blob/68f972cd47adcc248d205265ae352ca4caaa117a/src/main/clojure/clojure/java/classpath.clj#L86-L87

At first glance, Juraj Martinka's suggestion — to combine the two classpath sources — looks like a viable solution, although I do have questions about possible duplication of paths. I would also like to understand why the CIDER jack-in process produces this result before making changes. I will wait to see if someone on the CIDER issue thread can help with an explanation.

Answer selected: Is it usual to require all application namespaces via tools.namespace?

Tue, 13 Aug 2019 05:30:27 +0000

As asked: "Is it usual to require all application namespaces via tools.namespace?"

No, definitely not "usual".

It's not particularly usual to have multimethod/protocol implementations spread across so many namespaces -- that are otherwise not required by code that uses those multimethods/protocols -- that a list of requires of them is particularly "noisy".

tools.deps has a number of provider implementations spread across namespaces but it requires all the built-in ones in the "entry point" namespace for other code that uses the library. See https://github.com/clojure/tools.deps.alpha/blob/860bc7e3e310380b3baf9b43b05d343f3ae812f6/src/main/clojure/clojure/tools/deps/alpha.clj#L17-L22 -- I don't know whether you'd consider that "noisy"?

If you really do have a "lot" of implementation namespaces that are not likely to be required by any other code, and therefore need loading somewhere near startup, then I would first consider creating an interim namespace that does the requiring and then just require that in your initial namespace.

If that still seems too noisy, I might consider using tools.namespace to automatically find and load just the implementation namespaces, assuming they follow an appropriate pattern. But I've never run across that need in eight years of production Clojure so far.

Answered: scan-dirs / scan-all can return incorrect dependencies when both clj and cljc files define same namespace