Clojure Q&A - Recent questions tagged tools.deps

Concurrency bug in tools.deps use of Aether/Maven libs?

Tue, 21 Jul 2026 18:08:46 +0000

Occasionally, I've been seeing this error:

[org.eclipse.aether.internal.impl.DefaultTrackingFileManager] [] {} Failed to read tracking file '/home/sean/.m2/repository/org/apache/maven/maven/3.8.8/_remote.repositories' java.nio.channels.ClosedChannelException

which leads to

Expected HashMap$TreeNode, but was given HashMap$Node, at java.util.HashMap$TreeNode/moveRootToFront (HashMap.java:2020) - runtime error (unexpected type).

And the only solution seems to be to blow away part of the ~/.m2/repository and use -Sforce to recompute/redownload.

Any idea what might cause this sort of error? (Windows/WSL2, Ubuntu 20.04, all the very latest CLI/Clojure etc). See Slack thread for more discussion: https://clojurians.slack.com/archives/C6QH853H8/p1784653752175769

More of the stacktrace:
`
at java.base/sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:169)

    at java.base/sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1676)
    at org.eclipse.aether.internal.impl.DefaultTrackingFileManager.fileLock(DefaultTrackingFileManager.java:171)
    at org.eclipse.aether.internal.impl.DefaultTrackingFileManager.read(DefaultTrackingFileManager.java:65)
    at org.eclipse.aether.internal.impl.EnhancedLocalRepositoryManager.readRepos(EnhancedLocalRepositoryManager.java:220)
    at org.eclipse.aether.internal.impl.EnhancedLocalRepositoryManager.checkFind(EnhancedLocalRepositoryManager.java:151)
    at org.eclipse.aether.internal.impl.EnhancedLocalRepositoryManager.find(EnhancedLocalRepositoryManager.java:128)
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:360)
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:261)
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:243)
    at org.apache.maven.repository.internal.DefaultModelResolver.resolveModel(DefaultModelResolver.java:158)
    at org.apache.maven.repository.internal.DefaultModelResolver.resolveModel(DefaultModelResolver.java:204)
    at org.apache.maven.model.building.DefaultModelBuilder.readParentExternally(DefaultModelBuilder.java:1035)
    at org.apache.maven.model.building.DefaultModelBuilder.readParent(DefaultModelBuilder.java:827)
    at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:336)
    at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:248)
    at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:293)
    at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:183)
    at org.eclipse.aether.internal.impl.DefaultRepositorySystem.readArtifactDescriptor(DefaultRepositorySystem.java:269)
    at clojure.tools.deps.extensions.maven$read_descriptor.invokeStatic(maven.clj:117)
    at clojure.tools.deps.extensions.maven$read_descriptor.invoke(maven.clj:108)
    at clojure.tools.deps.extensions.maven$eval61086$fn__61087.invoke(maven.clj:148)

How to access non-jar artifacts (aar, zip) downloaded via tools.deps in build.clj?

Wed, 24 Dec 2025 08:55:46 +0000

I want to access an aar/zip artifact in build.clj that's already downloaded via deps.edn:

{:deps
 {com.newrelic.agent.android/android-agent {:local/root "build/jars/android-agent.jar"
                                            :doc        "extracted from aar"}
  com.newrelic.agent.java/newrelic-api     {:local/root "build/jars/newrelic-api.jar"
                                            :doc        "extracted from newrelic-java.zip"}}
 :aliases
 {:build {:deps       {io.github.clojure/tools.build            {:mvn/version "0.10.11"}
                       com.newrelic.agent.java/newrelic-java    {:mvn/version "8.25.1"
                                                                 :extension   "zip"}
                       com.newrelic.agent.android/android-agent {:mvn/version "7.6.14"
                                                                 :extension   "aar"}}
          :ns-default build}}}

In build.clj I need to unpack this zip and extract a jar from it.
And I don't have to hardcode .m2 paths or duplicate the version.

How can it be done?

See https://github.com/clojure/tools.deps/blob/46661e929ab702d0ee532188edb58d06b39cef6c/src/main/clojure/clojure/tools/deps/extensions/maven.clj#L172

Prepping - cache invalidation

Tue, 16 Sep 2025 10:31:56 +0000

Following the existing example:

https://clojure.org/reference/clojure_cli#deps_prep

{:paths ["src" "target/classes"]
 :deps/prep-lib
 {:ensure "target/classes"
  :alias :build
  :fn compile-java}}

How does one control cache invalidation? i.e. if any .java file has been added or modified, a compilation should be re-triggered.

The linked doc says:

:ensure - directory existence determines whether prep is needed

It seems to me that only the existence of a directory may not be sufficient to determine if a lib is 'prepped?' One might want to add a custom function, for instance.

This use case is particularly painful in monorepos, where a given sub-project may depend on a sibling project's java classes.

There's the option of wrapping things with babashka, which worked well for me, however it is surprising that standalone tools.deps appeared to be insufficient for this use case.

tools.deps.cli.api/mvn-install docstring typo

Sat, 09 Aug 2025 15:57:48 +0000

> clojure -X:deps help/doc :fn mvn-install
-------------------------
clojure.tools.deps.cli.api/mvn-install
([{:keys [jar pom lib version classifier local-repo]}])
  Install a jar and pom to the Maven local cache.
  The pom file must either be supplied, or generated based
  on provided lib/version/classifier, or provided inside the jar.
  The group/artifact/version coordinate will be pulled from the
  pom source as above.

  Required:
    :jar (reqired) - path to jar file (embedded pom used by default)

Should be :jar (required).

In addition, further on in the docstring:

  Execute ad-hoc:
    clj -X:deps mvn/install :jar '"foo-1.2.3.jar"'

That should be mvn-install, not mvn/install.

clojure -X:deps find-versions doesn't sort tags correctly?

Sat, 09 Aug 2025 14:04:34 +0000

When Maven versions include pre-releases, find-versions seems to sort correctly:

> clojure -X:deps find-versions :lib hiccup/hiccup
Downloading: hiccup/hiccup/maven-metadata.xml from clojars
{:mvn/version "2.0.0-alpha1"}
{:mvn/version "2.0.0-alpha2"}
{:mvn/version "2.0.0-RC1"}
{:mvn/version "2.0.0-RC2"}
{:mvn/version "2.0.0-RC3"}
{:mvn/version "2.0.0-RC4"}
{:mvn/version "2.0.0-RC5"}
{:mvn/version "2.0.0"}

However, when a library is git-only with tags and pre-releases exist, they seem to be alpha-sorted instead:

> clojure -X:deps find-versions :lib io.github.seancorfield/deps-new
{:git/tag "v0.7.1-rc", :git/sha "5a6168d"}
{:git/tag "v0.8.0", :git/sha "2f96530"}
{:git/tag "v0.8.1", :git/sha "2859baf"}
{:git/tag "v0.9.0", :git/sha "da2f764"}
{:git/tag "v0.9.0-rc", :git/sha "b32e0f2"}
{:git/tag "v0.10.0", :git/sha "20e82f8"}
{:git/tag "v0.10.1", :git/sha "a90029c"}
{:git/tag "v0.10.1-rc", :git/sha "35159b5"}

Those -rc versions should sort before the full release.

`create-basis` silently accepts missing aliases

Tue, 10 Jun 2025 00:57:37 +0000

In this scenario, the alias for create-basis was incorrect, preventing security deps overrides from being applied. For 2 years clojars was deployed with these vulnerable deps.

In this case, I proposed to inline the alias into :deps. However there may be other scenarios where a tools.deps change might be helpful to catch these mistakes earlier, for example a warning or error.

Confusing error message on deps.edn wrong git/url dependency

Thu, 05 Dec 2024 10:17:00 +0000

When you misspell the :git/url of a git dependency (or the dep name used to infer a url) the error says :

Cloning: https://github.com/MISSPELLED-ORG-OR-REPO-PATH
Error building classpath. Unable to clone /home/user/.gitlibs/_repos/https/github.com/MISSPELLED-ORG-OR-REPO-PATH
fatal: could not read Username for 'https://github.com': terminal prompts disabled

I find the the 3 sentences confusing since they mention classpath, a local folder, username, and terminal prompts disabled.

It would be great if it more clearly can express that no repo could be found with the provided URL, or at least remove the last sentence fatal: could not read Username for 'https://github.com': terminal prompts disabled which I find misleading.

Maybe the Unable to clone could be followed by the url instead of the local folder also?

Does tools.deps treat a version (e.g. 1.2.3) as a range (i.e. [1.2.3,))?

Wed, 05 Jun 2024 16:53:59 +0000

Does tools.deps treat a version (e.g. 1.2.3) as a range (i.e. [1.2.3,)) like Maven and Gradle or like a fixed version (i.e. [1.2.3])? In addition, Maven has the concept of "soft" versions where the specified version is "preferred". Does tools.deps have a similar concept?

Option in `clojure` CLI command to print the basis

Thu, 15 Feb 2024 13:57:44 +0000

It would be nice that clojure could print the basis for a given combination of aliases (or none).

A common use case is obtaining the computed :paths for a combination of aliases.

Improve error message for tools.deps when m2 permissions do not allow write

Mon, 29 Jan 2024 15:34:07 +0000

I've encountered this error and it took a while to figure out.

For some reason downloading eclipse artifacts was failing.
It was very frustrating.
I found the issue - to be caused by wrong permissions for directories under $HOME/.m2/repository.

Downloading deps failed to write any files to local Maven repo and that triggered a fail in deps resolving for my project.
The error was related to FileNotFound - and not "I can't write to this directory" .
It was more confusing since I had some dependencies in that directory (probably from an previous download).

I suspect the directory permissions where changed by mounting the local maven repo inside a docker container.

I reported this on slack and created this question as suggested by Alex Miller.
He mentioned this might be caused by maven libraries and out of control of tools.deps.

IMO I find it strange that maven libs don't error out when trying to create the local file and we get an exception when trying to read the file by tools.deps.

Steps to reproduce:

rm -rf ~/.m2/repository/org/eclipse 
mkdir -p ~/.m2/repository/org/eclipse/
chown root:root ~/.m2/repository/org/eclipse
clojure -Sdeps '{:deps {info.sunng/ring-jetty9-adapter {:mvn/version "0.22.0"}}}'

Exception

clojure -Sdeps '{:deps {info.sunng/ring-jetty9-adapter {:mvn/version "0.22.0"}}}'
Warning: failed to load the S3TransporterFactory class
Warning: failed to load the S3TransporterFactory class
Warning: failed to load the S3TransporterFactory class
Warning: failed to load the S3TransporterFactory class
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Warning: failed to load the S3TransporterFactory class
Warning: failed to load the S3TransporterFactory class
Warning: failed to load the S3TransporterFactory class
Warning: failed to load the S3TransporterFactory class
Error building classpath. Failed to read artifact descriptor for org.eclipse.jetty.websocket:websocket-servlet:jar:11.0.15
org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for org.eclipse.jetty.websocket:websocket-servlet:jar:11.0.15
    at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(Unknown Source)
    at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(Unknown Source)
    at org.eclipse.aether.internal.impl.DefaultRepositorySystem.readArtifactDescriptor(Unknown Source)
    at clojure.tools.deps.extensions.maven$read_descriptor.invokeStatic(maven.clj:115)
    at clojure.tools.deps.extensions.maven$read_descriptor.invoke(maven.clj:106)
    at clojure.tools.deps.extensions.maven$fn__825.invokeStatic(maven.clj:146)
    at clojure.tools.deps.extensions.maven$fn__825.invoke(maven.clj:143)
    at clojure.lang.MultiFn.invoke(MultiFn.java:244)
    at clojure.tools.deps$expand_deps$children_task__463$fn__465$fn__466.invoke(deps.clj:407)
    at clojure.lang.AFn.applyToHelper(AFn.java:152)
    at clojure.lang.AFn.applyTo(AFn.java:144)
    at clojure.core$apply.invokeStatic(core.clj:667)
    at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1990)
    at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1990)
    at clojure.lang.RestFn.invoke(RestFn.java:425)
    at clojure.lang.AFn.applyToHelper(AFn.java:156)
    at clojure.lang.RestFn.applyTo(RestFn.java:132)
    at clojure.core$apply.invokeStatic(core.clj:671)
    at clojure.core$bound_fn_STAR_$fn__7194.doInvoke(core.clj:2020)
    at clojure.lang.RestFn.invoke(RestFn.java:397)
    at clojure.lang.AFn.call(AFn.java:18)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact org.eclipse.jetty.websocket:websocket-servlet:pom:11.0.15 from/to central (https://repo1.maven.org/maven2/): /home/ieugen/.m2/repository/org/eclipse/jetty/websocket/websocket-servlet/11.0.15/websocket-servlet-11.0.15.pom.part.lock (No such file or directory)
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(Unknown Source)
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(Unknown Source)
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(Unknown Source)
    ... 25 more
Caused by: org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact org.eclipse.jetty.websocket:websocket-servlet:pom:11.0.15 from/to central (https://repo1.maven.org/maven2/): /home/ieugen/.m2/repository/org/eclipse/jetty/websocket/websocket-servlet/11.0.15/websocket-servlet-11.0.15.pom.part.lock (No such file or directory)
    at org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed(Unknown Source)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(Unknown Source)
    at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(Unknown Source)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector$DirectExecutor.execute(Unknown Source)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector.get(Unknown Source)
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(Unknown Source)
    ... 28 more
Caused by: java.io.FileNotFoundException: /home/ieugen/.m2/repository/org/eclipse/jetty/websocket/websocket-servlet/11.0.15/websocket-servlet-11.0.15.pom.part.lock (No such file or directory)

Does the clojure cli have an option to include new extensions for dependency resolution?

Mon, 20 Nov 2023 13:31:36 +0000

From the source of tools.deps, I see that git and Maven is implemented via extensions and multimethods. Is there a way for someone to write a third-party extension and tell the clojure CLI tool to use it? e.g. an extension for SVN dependencies via :svn

My company has an internal build system that is not git or Maven based. It should be able to integrate with the current extensions framework within deps. However, I can't seem to find a way to tell clojure to load this extension file. It would be nice if we could configure via deps.edn or a flag when calling clojure to include other extension implementations.

In tools.deps, get the source of dependencies without downloading

Thu, 09 Nov 2023 17:01:03 +0000

When using the tools.deps API, I would like to resolve dependencies without downloading all of the jars, and I would like the return value to tell me the original URL of the dependency, rather than the path on disk.

The reason is I use bazel + https://github.com/griffinbank/rules_clojure, and if I have the URL, I can immutably cache artifacts in bazel, and have that cache be shared with all other developers at work.

how to convert deps,edn file to pom.xml file i am using leiningen

Mon, 09 Oct 2023 23:34:24 +0000

how to convert deps,edn file to pom.xml file i am using leiningen

install-latest with a name already in use by another tool results in cryptic error message

Mon, 09 Oct 2023 12:14:59 +0000

Repro:

$ clojure -Ttools install-latest :lib io.github.seancorfield/clj-new :as new
new: Installed io.github.seancorfield/clj-new v1.2.404 {:git/tag "v1.2.404", :git/sha "d4a6508"}

$ clojure -Ttools install-latest :lib io.github.seancorfield/deps-new :as new
Execution error (ExceptionInfo) at clojure.tools.deps.extensions.git/coord-err (git.clj:46).
Library io.github.seancorfield/deps-new has invalid tag: v1.2.404

Abstract env var reading in tools.deps

Sat, 16 Sep 2023 02:35:57 +0000

I have a Cursive user wanting to use the CLOJURE_CLI_ALLOW_HTTP_REPO env var. Since Cursive invokes tools.deps in-process, there's no way for me to set an env var in the current process. deps.clj has a nice abstraction for this - would it be possible to use this or something similar for tools.deps too? I assume this problem will affect any process using tools.deps as a library.

clojure.tools.deps/calc-basis can fail with "Unable to resolve" due to a version range

Sat, 26 Aug 2023 06:26:56 +0000

The following code:

https://gist.github.com/vemv/0254e4f594626b090520f1da323f9476

(I used a gist due to this site's 12000 character limit)

...will fail with:

Unable to resolve org.clojure/clojure version: [1.2.1],[1.3.0]
{:lib org.clojure/clojure, :coord #:mvn{:version "[1.2.1],[1.3.0]"}}

Perhaps it's worth noting that [org.clojure/clojure #:mvn{:version "1.11.1"}] is already explicitly specified.

I'm using org.clojure/tools.deps "0.18.1354".

Thanks - V

make-classpath2 in tools.deps cannot be run from a directory other than the project one

Fri, 14 Jul 2023 13:49:38 +0000

In Cursive's new deps integration, I invoke make-classpath2 from within IntelliJ, which means that the CWD is not the project directory. I bind clojure.tools.deps.util.dir/*the-dir* when I do this, and most things work correctly. However, there are some exceptions which don't work:

:mvn/local-repo
:local/root deps inside :override-deps
:classpath-overrides

It would be nice if they could be made to work too. If there's interest in a patch for this I could have a go at it.

Could the CLI check for the latest stable version?

Mon, 19 Jun 2023 18:49:58 +0000

Given the number of people (mostly beginners) who seem to run into problems with the CLI because they have an older version and are following either newer instructions or trying to use libraries that rely on newer features, I wonder whether it would be worth adding a check to the script that would output a warning if the current stable version is more recent (from the download site)?

I know that won't help people who already have an old version installed but it would at least help future people...

...or is the command-line "API" stable enough at this point that it wouldn't be worth it?

Examples include the changing semantics for both -M and -T over the years, support for :git/sha (vs :sha), and several other changes, both in tools.deps itself and the CLI script.

Using deps/root with local/root

Thu, 04 May 2023 12:45:13 +0000

When working with monorepos, specifically depending on libs defined in monorepos, it is common to see those pulled in via :git/url,tag,sha and :deps/root like:

; clojure -Sdeps '{:deps {polylith-kaocha/kaocha-wrapper
  {:git/url "https://github.com/imrekoszo/polylith-kaocha"
   :git/sha "d1fcd406cf0813b836419e39706ae61929a391a2"
   :deps/root "projects/kaocha-wrapper"}}}' -Stree | grep 'lambdaisland/kaocha'
    . lambdaisland/kaocha 1.68.1059

There are cases when it is beneficial for development or debugging to clone the monorepo and reference its libs using :local/root. However, simply replacing the git attributes and point to a local folder does not currently work (Clojure CLI version 1.11.1.1273):

; clojure -Sdeps '{:deps {polylith-kaocha/kaocha-wrapper
  {:local/root "/Users/ikoszo/imre/oss/polylith-kaocha"
   :deps/root "projects/kaocha-wrapper"}}}' -Stree | grep 'lambdaisland/kaocha'

(notice that lambdaisland/kaocha is not printed indicating it isn't in the dep tree)

As tools.deps currently appears to ignore :deps/root for :local/root dependencies, developers currently have to combine the two into :local/root like so:

; clojure -Sdeps '{:deps {polylith-kaocha/kaocha-wrapper
  {:local/root "/Users/ikoszo/imre/oss/polylith-kaocha/projects/kaocha-wrapper"}}}' -Stree | grep 'lambdaisland/kaocha'
    . lambdaisland/kaocha /Users/ikoszo/imre/oss/kaocha
    . lambdaisland/kaocha /Users/ikoszo/imre/oss/kaocha

At https://clojure.org/reference/deps_and_cli the following are currently stated:

Common coordinate attributes (all optional):
...
:deps/root - relative directory path within a file based dep to search for the manifest file

and

Local project attributes:
:local/root (required) - directory path (will auto detect whether a deps or pom project, also see :deps/manifest)

Reading the above, :local/root can qualify as a file based dep, which to me implies that :deps/root should work as it isn't stated otherwise.

Could support be added for this? This would make life easier switching between local and git deps when developing a library or tool.

If yes, this might also be a way to help solve at least a subset of the problem reported in https://clojure.atlassian.net/browse/TDEPS-132

(Originally asked and discussed in this slack thread: https://clojurians.slack.com/archives/C6QH853H8/p1682712720554439)

"Execution error at java.lang.ProcessImpl/forkAndExec Argument list too long" when using add-lib

Fri, 14 Apr 2023 15:55:25 +0000

Slack Discussion

This error was being thrown for me on Linux running:
clojure 1.12.0-alpha2
clojure CLI: 1.11.1.1273

The deps.edn file in the directory I was running clj included four non-clojure dependencies which expanded to 222 transitive dependencies. Additionally I had nine aliases in my ~/.clojure/deps.edn which added additional dependencies to the tools.deps basis.

Alex Miller mentioned on Slack that large basis data could cause this exception and that he would try to address the problem.

Can clojure.tools.deps.script.make-classpath2/run-core be considered a public API?

Wed, 22 Feb 2023 03:10:38 +0000

If not, could it (or something like it) be? This would be useful for allowing projects which embed tools.deps, like Cursive, to get classpath info from a project without having to muck around creating a bunch of files.

If -main is going to be the only officially-sanctioned public API to this functionality, it would be good to abstract calls to System/exit in such a way that embedding tools can catch process exits without being killed. deps.clj has an example of one way to do this here: https://github.com/borkdude/deps.clj/blob/master/deps.clj#L34-L39.

Support `:mvn/repos` inside an alias

Tue, 08 Nov 2022 19:08:24 +0000

This is similar to https://ask.clojure.org/index.php/11711/set-mvn-local-repo-via-alias, sorry if I should have just added a note there instead of opening another question.

:mvn/repos does not seem to work inside of an alias either -- only at the top level of deps.edn. If this is intentional, it would be a nicer experience if CLI logged some sort of ":mvn/repos is not supported inside of aliases" warning on launch.

Add retries when downloading dependencies?

Tue, 18 Oct 2022 00:42:44 +0000

Our CI runs failed because of a temporary Clojars blip:

Error building classpath. Could not transfer artifact
rewrite-clj:rewrite-clj:jar:1.1.45 from/to clojars
(https://repo.clojars.org/): Connect to repo.clojars.org:443
[repo.clojars.org/146.75.29.128] failed: Connect timed out 401 Error:
Process completed with exit code 1.

Does it seem like it could be good to add some retries to tools.deps when downloading dependencies to handle temporary failures?

Add example of clojure -Sdeps to Deps and CLI guide or reference

Tue, 04 Oct 2022 18:38:54 +0000

Testing libraries before adding them to a deps.edn file by using -Sdeps '{...}' is a frequent recommendation, but it's only mentioned in a single line in the Reference page.

I think it would be helpful if it (and the other flags) had examples with more detail about how they're used and why. Maybe something like:

-Sdeps - pass the config data on the command line

For example, instead of creating a new deps.edn file to experiment with a new library, you can use -Sdeps to pass in the :deps map to clojure:

$ clojure -Sdeps '{:deps {org.clojure/core.match {:mvn/version "1.0.0"}}}'
Clojure 1.11.1
user=> (require '[clojure.core.match :refer [match]])
nil

create-basis uses the wrong local maven repository

Sat, 24 Sep 2022 08:29:08 +0000

I am attempting to build a Clojure project with numerous dependencies and contain everything into a single folder. The reason for doing this is because I will eventually build this software in an environment where I am not permitted to modify the home folder.

The following command is being used
GITLIBS=/var/build/software/.gitlibs clojure -Srepro -Sdeps '{:mvn/local-repo "/var/build/software/.m2/repository"}' -T:build uber-aot

tools.deps is able to download all dependencies into the specified local repository. Once it's time for tools.build to run, I seem to get an error in the middle of a call to b/create-basis. Here is what the error looks like.

Execution error (FileNotFoundException) at java.io.RandomAccessFile/open0 (RandomAccessFile.java:-2). /home/user/.m2/repository/org/clojure/clojure/1.11.1/clojure-1.11.1.pom.part.lock (No such file or directory)

Is there any way to pass in the local Maven repository to tools.build? Preferably in the command line.

clojure tools invalid certificate exception when downloading dependencies behind a firewall

Fri, 09 Sep 2022 17:17:02 +0000

Hi,

I'm getting a PKIX certificate exception when trying to download project dependnecies with the clojure tool behind a strict firewall on MS-Windows:

Could not transfer artifact ... from/to central
(https://repo1.maven.org/maven2/):
sun.security.validator.ValidatorException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable
to find valid certification path to requested target

I believe this is due to self-signed root certificate utilized by the Firewall to control & monitor the traffic.

Analysis follows and apologies for the long read.

It is a standard practice in big organisations for workstations to sit behind a firewall that controls and monitors all traffic to and from the Internet. It is also common for the company to create their own self-signed root certificate and replace with it the certificates found in the https traffic so as to decrypt and analyze the data flows, utilizing in effect the man-in-the-middle attack to their advantage.

For this to work, the user workstation where the connection originates from needs to have the self-signed certificates installed in their trusted certificate store. On Windows, this will be the trusted root authority certification keystore.

Java comes with its own certificate store, the java KeyStore, which is separate from the general keystore available on the workstation. Thus, any java https connections to the Internet through the firewall is most likely to fail with an unknown certificate error, since the self-signed certs are not installed on the java keystore.

This adversely affects the clojure/clj command line tools when downloading libs because it can't find the self-signed cert in the java keystore. For example, an exception is thrown when trying to access the Internet from such computers:

 > clojure -P
 
 Error building classpath ... 
 org.eclipse.aether.resolution.ArtifactDescriptorException:   Failed to
 read artifact descriptor for ....  ...  Caused by:
 org.eclipse.aether.resolution.ArtifactResolutionException:   Could not
 transfer artifact ... from/to central
 (https://repo1.maven.org/maven2/):   
 sun.security.validator.ValidatorException: PKIX path building failed:
      sun.security.provider.certpath.SunCertPathBuilderException:
        unable to find valid certification path to requested target  ...
  ...

I can think of at least two solution to circumvent this problem:
1. Install the missing certificates in the java keystore, or
2. Instruct the java instance to look into the general keystore.

There does not seem to be a straightforward way to discover the missing certificates and would probably require some user effort to install them into the java keystore using the java keytool (See Working with Certificates and SSL at https://docs.oracle.com/cd/E19830-01/819-4712/ablqw/index.html).

Instead, it seems easier to instruct java to look at the general or alternative keystores using the java.next.ssl.* properties (https://stackoverflow.com/questions/5871279/ssl-and-cert-keystore).

On Windows, which I assume most of users with such restrictive firewalls are sitting on, it is a just matter of setting a simple java property to instruct java to use the general windows keystore (https://stackoverflow.com/questions/41257366/import-windows-certificates-to-java):

javax.net.ssl.trustStoreType=Windows-ROOT

Given this, I was expecting the following to work on windows

clojure -J-D'javax.net.ssl.trustStoreType=Windows-ROOT' -P

but it doesn't. It turns out the $JvmOpts variable is not passed onto the prep command by the powershell script here https://github.com/clojure/brew-install/blob/b91fb78e321b5e39bedda594f5d578579d448d19/src/main/resources/clojure/install/ClojureTools.psm1#L388 :

& $JavaCmd -classpath $ToolsCp clojure.main -m clojure.tools.deps.alpha.script.make-classpath2 --config-user $ConfigUser --config-project $ConfigProject --basis-file $BasisFile --libs-file $LibsFile --cp-file $CpFile --jvm-file $JvmFile --main-file $MainFile --manifest-file $ManifestFile @ToolsArgs

Some options I can think of to utilise java properties to solve the general issue, from most intrusive/specific to the general case:

Have the clojure.tools.deps.alpha.script.make-classpath2 fn which is responsible for downloading the dependencies set javax.net.ssl.trustStoreType=Windows-ROOT at runtime, if running on Windows,
- .e.g. (System/setProperty "javax.net.ssl.trustStoreType" "Windows-ROOT").
- cons:
  - This only works on windows.
  - forces to use of the general keystore, which might not be what the user always want.
    - This can be mitigated by having a new script option (say -W) that is passed onto clojure.tools.deps.alpha as an option. The drawback is that users/tooling has to always pass this flag on invocation.
  - setting the javax.net.ssl.tustStoreType property at runtime with System/setProperty could have no effect if any SSL connection was previously made from other parts of the code.
Same as previous, but with a new script option (say -Sssl EDN) whose EDN are the javax.net.ssl.* properties, and will be passed as such to clojure.tools.deps.alpha for setting at runtime.
- e.g. clojure -Sssl {:trustStoreType "Windows-ROOT"} -P
- cons:
  - Same disadvantages as in #1, and moreover is more cumbersome for the user to type an edn map each time it invokes clojure.
Invent a new deps.edn key (e.g. :clojure.tools.deps.alpha/ssl) whose pairs shall be java.net.ssl.* properties, that can be set in the user config deps.edn and parsed by clojure.tools.deps.alpha s make-classpath2 fn setting the properties using System/setPropertyat runtime.
- e.g. example clj user config {:clojure.tools.deps.alpha/ssl {:trustStore "Windows-ROOT"}}
- cons
  - setting the javax.net.ssl.* properties at runtime with System/setProperty might have no effect if any SSL connection was previously made from other parts of the code.
Update script so that -J jvm options are passed onto the java invocation of the -P command (I have tested this to work).
- e.g. clojure -J-D'javax.net.ssl.trustStoreType=Windows-ROOT' -P
- cons:
  - User/tooling still have to provide the -J-Djavax.net.ssl.* options at each clojure invocation.
Same as previous, but introduce a CLJ_JVMOPTS environment variable whose value is inserted in the script's $JvmOpts variable.
- e.g. set CLJ_JVMOPTS=-D'javax.net.ssl.trustStoreType=Windows-ROOT', and then clojure -P

I'm more in favor of #5, since this does not seem to have any disadvantages and can be set once and be applied on all clojure invocations, creating a better experience for the user/tooling.

Let me know your thoughts. Happy to look after the changes in the scripts and/or tools.deps.alpha lib.

Thanks

PS: I have also tried to set the properties in the MAVEN_OPTS variable as pe r https://maven.apache.org/guides/mini/guide-repository-ssl.html, but it had no effect. I suspect this only take effect when invoking the mvn command line tool directly.

Add validation for lib groupId and name coordinate of git deps in tools.deps

Mon, 05 Sep 2022 02:56:53 +0000

In tools.deps, when using git deps, the following declared dependencies:

{io.github.xadecimal/riddley {...}}

{com.github.xadecimal/riddley {...}}

{com.xadecimal/riddley {:git/url "https://github.com/xadecimal/riddley.git" ...}}

{com.xadecimal/riddley {:mvn/version ...}}

{garbage/whatever {:git/url "https://github.com/xadecimal/riddley.git" ...}}

will all resolve, but they will not conflict against one another, even though they are all the same lib.

As it stands, the consumer that declares the dependency can choose from all these equivalent declarations, but if it chooses differently than any of its transitive dependencies have chosen, a hidden conflict is created, and more than one version of the lib will be pulled into the classpath, giving non-deterministic behavior to what version of the code will actually be used at runtime.

This problem could be solved by letting the lib author choose the canonical lib group/name that users must use always when declaring a dependency on it.

This can be achieved by adding a new key to deps.edn, such as :lib which takes a fully qualified symbol which will be validated against the symbol declared in the :deps map of the consuming libs.

deps.edn of the library being depended upon:

{:lib com.xadecimal/riddley ;; <-- This new key is added to deps.edn, declaring the canonical lib name
 :paths ["src"]
 :deps {...}
 ...}

deps.edn of the libraries that declare a dependency on the above lib:

{:paths ["src"]
 :deps {com.xadecimal/riddley {...}}
 ...}

When tools.deps resolves the above, it will check if the pulled lib has :lib key declared in its deps.edn, if it does, it will assert that the symbol used here matches the declared :lib symbol, if it does not, it will error to the user, making this an illegal dependency declaration. In the above it is correct, but in the following it is wrong:

deps.edn of a library that badly declared an illegal dependency on the above lib:

{:paths ["src"]
 :deps {com.github.xadecimal/riddley {...}}
 ...}

This would error, because even if the library is found at github.com/xadecimal/riddley.git and thus the above resolves, the library at github.com/xadecimal/riddley.git declared that its canonical lib name was com.xadecimal/riddley and not com.github.xadecimal/riddley. Therefore, in this case the only valid git dependency declaration is:

{:paths ["src"]
 :deps {com.xadecimal/riddley {:git/url "https://github.com/xadecimal/riddley.git" ...}}
 ...}

This validation would follow a similar ideology to asserting that the git/tag matches the git/sha, by asserting that the lib name used matches what the lib expects to uniquely be identified by.

Doing this allows conflict resolution to work as expected, and avoids the possibility that transitive dependencies use a different name, causing hidden runtime version conflicts not detected and resolved by tools.deps.

tools.deps integration with Sourcehut

Wed, 22 Jun 2022 15:06:24 +0000

The implementation of tools.deps currently allows developers to specify git dependencies for their projects in ways that allow them to omit a git url if they are using specific git forges.

For example, io.github.username/example would resolve to https://github.com/username/example.git

I would like to request that sourcehut be added to the list of git forges that are supported by tools.deps.

Sourcehut git urls are of the form https://git.sr.ht/~username/example
I would like to suggest that the artifact name take on the form ht.sr.username/example, at the discretion of the maintainers.

Why does tools install-latest not always find the latest version?

Sun, 19 Jun 2022 20:08:24 +0000

As seen on Slack: https://clojurians.slack.com/archives/C03S1KBA2/p1655612767836199

I have confirmed that:

clojure -Ttools install-latest :lib io.github.seancorfield/deps-new :as new

Installs v0.4.9 despite three releases having been made since then.

And if you have an earlier version of deps-new installed and try to update it:

clojure -Ttools install-latest :tool new

you get v0.4.9.

How is it comparing the versions? A lexical sort won't work because v0.4.10, v0.4.11, and v0.4.12 will all sort "earlier" than v0.4.9 but they are later versions.

relative directories in classpath-roots

Sun, 19 Jun 2022 16:50:53 +0000

In the context of a monorepo containing several modules I have noticed an issue with clojure.tools.build.api/compile-clj which is making difficult to compile files when the basis project root is not in the current directory.

Here is a minimal scenario demonstrating the problem.

I have a top-level monorepo deps.edn using tools.build to prepare/package its modules

 {:deps {monorepo/module-a {:local/root "modules/a"}}
 :aliases {:build {:deps {io.github.clojure/tools.build {:git/tag "v0.8.2"
                                                         :git/sha "ba1a2bf"}}
                   :ns-default build}}}

Module A contains some clojure source files. Here is modules/a/deps.edn

{:paths ["src"]}

Those files require a compilation step. Here is the top-level build.clj file expected to proccess those files.

(ns build
  (:require [clojure.tools.build.api :as b]))

(def module-a (binding [b/*project-root* "modules/a"]
                (b/create-basis {:project "deps.edn"})))

(defn compile-a [_]
  (b/compile-clj
   {:basis module-a
    :src-dirs ["modules/a/src"]
    :class-dir "target/module-a/classes"}))

However when invoking clojure -T:build compile-a from top-level directory I encounter the following error

Execution error (FileNotFoundException) at user/eval136$fn (compile.clj:5).
Could not locate monorepo/module/a__init.class, monorepo/module/a.clj or monorepo/module/a.cljc on classpath.

The problem is that the "src" relative directory is not expanded in the :classpath-roots which lead compile-clj to search source files in the wrong location. It is possible to work around the issue by binding the *project-root* again and adapting :src-dirs and class-dir relative to module A root directory like this

(defn compile-a [_]
  (binding [b/*project-root* "modules/a"]
    (b/compile-clj
     {:basis module-a
      :src-dirs ["src"]
      :class-dir "../../target/module-a/classes"})))

However IMO it would be better if the computed classpath in the basis was fully expanded by transforming relative paths into absolute ones to avoid this extra ceremony.

What do you think?
Is there a reason why relative source path directories are currently kept relative ?

Regression in tools.deps.alpha - can't use find-versions with non-git coordinates

Thu, 16 Jun 2022 08:11:23 +0000

This commit seems to be the culprit: https://github.com/clojure/tools.deps.alpha/commit/a4ad9c23250270a173101aee6bb2eceeaee53d92

Before it, using a non-git library name would just result in nil returned from find-vesions, which would allow find-all-versions to try out other coord types.

$ clj --version
Clojure CLI version 1.11.1.1129

$ clj -X:deps find-versions :lib org.clojure/clojure
Execution error (ExceptionInfo) at clojure.tools.deps.alpha.extensions.git/coord-err (git.clj:44).
Failed to determine git url for: org.clojure/clojure

Full report at:
/tmp/clojure-5949355979496977609.edn

$ clj -X:deps find-versions :lib org.clojure/tools.deps.alpha                                    
Execution error (ExceptionInfo) at clojure.tools.deps.alpha.extensions.git/coord-err (git.clj:44).
Failed to determine git url for: org.clojure/tools.deps.alpha

Full report at:
/tmp/clojure-5130519385828631638.edn

$ clj -X:deps find-versions :lib io.github.clojure/tools.build
Cloning: https://github.com/clojure/tools.build.git
{:git/tag "0.0.1"}
{:git/tag "0.0.2"}
[...]

Not entirely related, but page at https://clojure.org/reference/deps_and_cli#find-versions says:

To search for available versions of a git lib use the find-versions tool provided with the built-in :deps alias

So the docs make it sound as if find-versions is only for Git-based coordinates. All while it used to work with other things as well.

-X:deps prep not handling nested :local/root

Sat, 11 Jun 2022 23:28:53 +0000

While setting up a monorepo using tools.deps I have noticed that clojure -X:deps prep was not handling the case of nested :local/root dependency. Here is a minimal example that should help understand the issue.

I have a top-level deps.edn serving as a development basis depending two modules.

{:deps {monopero/module-a {:local/root "modules/a"}
        monopero/module-b {:local/root "modules/b"}}}

Here is module/a/deps.edn which is a module without dependency

{:paths ["src"]}

And here is module/b/deps.edn which defines a prep only dependency on module A

{:paths ["target/resources"]
 :deps/prep-lib {:alias :resources
                 :fn monorepo.b.core/generate
                 :ensure "target/resources"}
 :aliases {:resources {:paths ["src"]
                       :deps {monopero/module-a {:local/root "../a"}}}}}

From the top-level development basis I would expect to be able to run clojure -X:deps prep to trigger the preparation of module B. However when doing so I encounter the following error

~/src/scratch/recur-prep $ clojure -X:deps prep
Prepping monopero/module-b in /home/mthl/src/scratch/recur-prep/modules/b
Execution error (ExceptionInfo) at clojure.tools.deps.alpha.extensions/throw-bad-manifest (extensions.clj:158).
Manifest type not detected when finding deps for monopero/module-a in coordinate #:local{:root "/home/mthl/src/scratch/a"}

Full report at:
/tmp/clojure-5135322536447899533.edn

And here is the content of /tmp/clojure-5135322536447899533.edn

{:clojure.main/message
 "Execution error (ExceptionInfo) at clojure.tools.deps.alpha.extensions/throw-bad-manifest (extensions.clj:158).\nManifest type not detected when finding deps for monopero/module-a in coordinate #:local{:root \"/home/mthl/src/scratch/a\"}\n",
 :clojure.main/triage
 {:clojure.error/class clojure.lang.ExceptionInfo,
  :clojure.error/line 158,
  :clojure.error/cause
  "Manifest type not detected when finding deps for monopero/module-a in coordinate #:local{:root \"/home/mthl/src/scratch/a\"}",
  :clojure.error/symbol
  clojure.tools.deps.alpha.extensions/throw-bad-manifest,
  :clojure.error/source "extensions.clj",
  :clojure.error/phase :execution},
 :clojure.main/trace
 {:via
  [{:type clojure.lang.ExceptionInfo,
    :message
    "Manifest type not detected when finding deps for monopero/module-a in coordinate #:local{:root \"/home/mthl/src/scratch/a\"}",
    :data
    {:lib monopero/module-a,
     :coord {:local/root "/home/mthl/src/scratch/a"}},
    :at
    [clojure.tools.deps.alpha.extensions$throw_bad_manifest
     invokeStatic
     "extensions.clj"
     158]}],
  :trace
  [[clojure.tools.deps.alpha.extensions$throw_bad_manifest
    invokeStatic
    "extensions.clj"
    158]
   [clojure.tools.deps.alpha.extensions$throw_bad_manifest
    invoke
    "extensions.clj"
    153]
   [clojure.tools.deps.alpha.extensions$eval443$fn__444
    invoke
    "extensions.clj"
    166]
   [clojure.lang.MultiFn invoke "MultiFn.java" 244]
   [clojure.tools.deps.alpha$expand_deps$children_task__611$fn__613$fn__614
    invoke
    "alpha.clj"
    405]
   [clojure.tools.deps.alpha.util.concurrent$submit_task$task__247
    invoke
    "concurrent.clj"
    35]
   [clojure.lang.AFn call "AFn.java" 18]
   [java.util.concurrent.FutureTask run "FutureTask.java" 264]
   [java.util.concurrent.ThreadPoolExecutor
    runWorker
    "ThreadPoolExecutor.java"
    1128]
   [java.util.concurrent.ThreadPoolExecutor$Worker
    run
    "ThreadPoolExecutor.java"
    628]
   [java.lang.Thread run "Thread.java" 829]],
  :cause
  "Manifest type not detected when finding deps for monopero/module-a in coordinate #:local{:root \"/home/mthl/src/scratch/a\"}",
  :data
  {:lib monopero/module-a,
   :coord {:local/root "/home/mthl/src/scratch/a"}}}}

As I understand it the issue is that the "../a" local reference in module B is resolved from top-level directory instead of module B directory. While looking at the code I noticed that clojure.tools.deps.alpha/prep-libs! is not changing the current directory when creating the basis. My intuition is that it should do so with something like the following snippet

(let [root-dir (jio/file root)
      basis (dir/with-dir root-dir
              (create-basis
               {:extra {:aliases {:deps/TOOL {:replace-deps {} :replace-paths ["."]}}}
                :aliases [:deps/TOOL alias]}))
      ,,,]
  ,,,)

In case it matters I am using Clojure CLI version 1.11.1.1113

Memory leak in clojure.tools.deps.alpha/resolve-deps ?

Thu, 02 Jun 2022 14:21:52 +0000

Hi Clojure team,

At Liftoff, we are big users of Clojure (400k LOC across 250+ projects). We have built some tooling that automatically generates deps.edn files for projects, based on static analysis of the project's code.

Some of our tools and test code repeatedly calls clojure.tools.deps.alpha/resolve-deps. As of version v0.14.1178, we have noticed the following:

resolve-deps creates a new java.util.concurrent.ExecutorService on each call, but doesn't shut it down before returning (code). This lead our test code to spawn 20k+ threads.
clojure.tools.deps.alpha.util.concurrent/submit-task pushes thread bindings onto a fixed size threadpool, but doesn't pop them (code). As threads are re-used, the frame stack keeps growing.

Internally, we are now using a patched version of the tools that unconditionally shuts down the ExecutorService on each call to resolve-deps and pops the thread bindings for each task. This solves our problem, but we don't know if this is right solution.

How is the core Clojure team thinking about concurrency in the context of the CLI tools? Can we expect this behavior to change in future releases of the CLI tools?

Thanks!

tools.deps not working when downloading a big jar on a small machine

Fri, 29 Apr 2022 06:54:04 +0000

I started an ec2 t3.nano instance and created a clojure deps project. The only dependency I had was a private jar file which weighted around ~100mb.
clj command could not download the file and was misleading in its error :
Error building classpath. Could not find artifact name/of/artifact

By curiosity I looked at tools.deps code and tried to run the s3 call :
`
(aws/invoke s3-client {:op :GetObject

                     :request {:Bucket "bucketname"
                               :Key "/path/to/jar"}})

and then we had a clear error (see https://clojurians.slack.com/files/U2N9GDB1U/F03E16TB10Q/image.png)

It looks like cognitect http is keeping the file in memory, on a really small instance or a container it does not work (it worked with mvn)
Quick fix: have a better error message
Thanks a lot

Possible concurrency issue with git deps

Thu, 24 Mar 2022 07:23:41 +0000

When launching clojure in our CI pipeline, we're occasionally seeing the problem in the logs below (edited for brevity). Clojure is trying to clone one of our git dependencies multiple times at once and ends up failing with the git error message fatal: destination path '...' already exists and is not an empty directory.. The said dependency appears multiple times in our dependency tree.

$ clojure -A:build:dev:test -P
Cloning: ssh://git@gitlab.com/xxx/yyy/www/our-repo.git
Cloning: ssh://git@gitlab.com/xxx/yyy/www/our-repo.git
Cloning: ssh://git@gitlab.com/xxx/yyy/www/our-repo.git
Downloading: org/clojure/clojure/1.10.3/clojure-1.10.3.pom from central
Error building classpath. Unable to clone /builds/xxx/yyy/zzz/.gitlibs/_repos/ssh/gitlab.com/xxx/yyy/www/our-repo
fatal: destination path '/builds/xxx/yyy/zzz/.gitlibs/_repos/ssh/gitlab.com/xxx/yyy/www/our-repo' already exists and is not an empty directory.

Could this be a bug in tools.deps.alpha or in tools.gitlibs? This is different from TDEPS-180 in that in our case, we're launching only one clojure process.

We're using Clojure CLI version 1.10.3.1040. We're upgrading to 1.10.3.1087, but since this problem happens intermittently, it will take a while to tell if it helps.

Allow configuring behavior for -X when it's invoked as entry point

Mon, 21 Feb 2022 19:51:06 +0000

Problem

It is useful to add custom behavior for program entry points. Examples of such behavior:
- calling (shutdown-agents) to ensure graceful shutdown;
- calling (Platform/exit) for JavaFX applications to ensure graceful shutdown;
- rebinding System/out and System/err in test runners to collect all output.

The -main function for -M-style entry points is a natural place to put these behaviors.

One property of these behaviors is that they might be very useful for entry points, but they affect JVM in such a way that these programs might be impossible to integrate into longer-running programs.

The situation with -X-invokable functions is where this property is problematic because these are supposed to be invokable both as an entry point and as a part of a longer-running program called by the user of the dependency, but there is no way to tell those apart from inside the function.

Considered solutions

Do nothing. Library authors can add optional arguments to their -X-invokable functions that disable special entry-point behavior, although this might be annoying to users if their programs break in unexpected ways when invoked from user code. Library authors can add optional args to enable special entry-point behavior instead of disabling it, but it makes the library more inconvenient to use at the command line. Library authors can create separate -X-invokable functions for use at the command line and at the user code, but it decreases the usefulness of the whole idea of having the same API in code and at the CLI with -X.
-X-style invocations can be given an extra argument by clj-exec that hints that it is invoked as an entry point, e.g. clj -X clojure.core/prn :a 1 will print {:entry-point true :a 1}. I think this is behavior is unexpected and might be confusing or even annoying.
-X-invokable vars can define metadata that will be added to the arg when invoked as an entry point, e.g.:
(defn ^{:default-clj-exec-argmap {:rebind-system-err true}} do-stuff [argmap] ...)
Then, running clj -X my.ns/do-stuff :other :args will use {:rebind-system-err true} as a default argmap that will then be merged with other args.

Will tools.build be released as a jar?

Wed, 05 Jan 2022 15:02:43 +0000

I am developing a library with a dependency on tools.build and I would like to publish it as a jar. Whilst tools.build is only available as a git dep, I cannot include it as a dependency in my pom file. A wider discussion on this topic was held on this slack thread, where @alexmiller requested that I raise this topic here.

tools.deps.graph uses an outdated version of tools.deps.alpha that doesn't support :git/sha coordinates

Fri, 10 Dec 2021 12:54:46 +0000

Repro deps.edn:

{:deps {io.github.clojure/spec-alpha2 {:git/sha "99456b1856a6fd934e2c30b17920bd790dd81775"}}}

Running clj -X:project/graph-deps, where project/graph-deps is the following alias in ~/.clojure/deps.edn:

:project/graph-deps
{:replace-paths []
 :replace-deps  {org.clojure/tools.deps.graph {:mvn/version "1.0.63"}}
 :ns-default    clojure.tools.deps.graph
 :exec-fn       graph
 :exec-args     {:output "project-dependencies-graph.png"
                 :size   true}}

Results in:

Library io.github.clojure/spec-alpha2 has missing :sha in coordinate.

Bumping the version of tools.deps.alpha to 0.12.1090 seems to fix the issue

Is it possible to build an uberwar using deps.edn using tools.build or other tool?

Thu, 02 Dec 2021 13:22:01 +0000

We have a small Clojure & ClojureScript web app running in Tomcat. The project managed to end up with a deps.edn for use with Shadow-cljs & lein for the backend to essentially build an uberwar. Is there a way to build an uberwar with tools-build or in some other way using deps.edn?

Thanks for any pointers

Cheers
Daniel Sunnerek

tools.deps - How do I list dependencies and their versions from the REPL?

Thu, 11 Nov 2021 11:47:04 +0000

I want to check which version of some dependencies I have loaded in the project from the REPL. Can it be done? How do I do this, if so?

The project I am working with is started via a rather involved script, making it a bit hard to know what version of certain dependencies are eventually loaded. I am thinking that once I have a REPL, then I have the truth.

It's a tools.deps project

Allow absolute paths for file-based Git URLS

Fri, 05 Nov 2021 19:24:15 +0000

Type: Enhancement
Summary: Expand tools.gitlibs support for local Git URLs
Description:
When specifying a local file-based URL for a Git repo containing a dependency, I encounter the following error when using an absolute URL:

java.lang.IllegalArgumentException: /path/to/my/repo is not a relative path

    at clojure.java.io$as_relative_path.invokeStatic(io.clj:414)
    at clojure.java.io$file.invokeStatic(io.clj:426)
    at clojure.java.io$file.invoke(io.clj:418)
    at clojure.lang.ArraySeq.reduce(ArraySeq.java:111)
    at clojure.core$reduce.invokeStatic(core.clj:6829)
    at clojure.java.io$file.invokeStatic(io.clj:418)
    at clojure.tools.gitlibs.impl$git_dir.invokeStatic(impl.clj:59)
    at clojure.tools.gitlibs.impl$ensure_git_dir.invokeStatic(impl.clj:80)
    at clojure.tools.gitlibs$procure.invokeStatic(gitlibs.clj:61)
    at clojure.tools.deps.alpha.extensions.git$fn__1317.invokeStatic(git.clj:98)
    at clojure.tools.deps.alpha.extensions.git$fn__1317.invoke(git.clj:96)
    at clojure.lang.MultiFn.invoke(MultiFn.java:239)
    at clojure.tools.deps.alpha$expand_deps.invokeStatic(alpha.clj:425)
    at clojure.tools.deps.alpha$resolve_deps.invokeStatic(alpha.clj:495)
    at clojure.tools.deps.alpha$calc_basis.invokeStatic(alpha.clj:738)
    ...

The requirement of relative URLs makes specifying dependencies more difficult in certain contexts - the specific motivating use case for me is a WSL environment where the Clojure process is running in Ubuntu/WSL but the source code is in the Windows filesystem.

Alex Miller mentioned on the Clojurians Slack when I mentioned this problem that "that's really a subset of the problem - the code that translates git urls to cache dirs and libs doesn't handle either raw paths or relative urls."

Using a git SHA for a local git dependency?

Fri, 05 Nov 2021 19:12:28 +0000

Type: Enhancement

Summary: Use git SHAs + tags for local deps

Description:

Currently, when working with local dependencies, I cannot use a git tag to refer to a specific revision of a project. :local/root only respects the current version of the working tree, which can make using local deps under active development a bit trickier.

Support for SHAs and tags for local projects which are specified via file-based Git URLs would allow projects consuming local deps to specify the state of those projects unambiguously.

I previously asked a version of this question on the Clojurians slack and was referred here by Alex Miller.

:deps in dependency not (always) working when there is no :path specified in dependency

Tue, 26 Oct 2021 08:42:30 +0000

I am not sure how to reliably reproduce this, but I had a problem with dependencies not being picked up from a :local/root dependency, and my colleague found the linked question, and it helped me find a workaround.

In my case my full deps in the dependency project was:

{:paths ["src"]}

Changing it to:

{:paths ["src"]
 :deps {}}

made things work.

I do have another project with the same dependency, and it was happy with the no-deps version.

This was in a dependency of a dependency. So I have both these cases:

Not working: project-a -> -dependency-1 -> dependency-2-path-only
Working: project-b -> -dependency-1 -> dependency-2-path-only

(”Not working/Working” as in dependency-2-path-only not being on the classpath of project-a, but on the classpath of project-b.)

Further... dependency-1 is in a sub directory of dependency-2-path-only. So in dependency-2-path-only/dependency-1/deps.edn:

{:paths ["src"]
 :deps {d2/dependency-2-path-only {:local/root ".."}}}

I realize now, after having described all this that I could have made a repro instead. Let me know if that is needed.

I'm currently on 1.10.3.967, btw.

tools.build uber task fails when the dependency tree contains a dep which has no jar

Tue, 12 Oct 2021 14:00:42 +0000

The clojure.tools.build.api/uber task fails when the dependency tree contains a dep which has no jar.

A relatively minimal failing test case is the following tools.build task:

(ns build 
  (:require [clojure.tools.build.api :as b]))

(defn failing-uber [_]
    (b/uber {:class-dir "classes"
     :uber-file "foo.jar"
     :basis (b/create-basis {:project {:deps {'org.apache.jena/apache-jena-libs {:mvn/version "4.2.0" :extension "pom"} }}})})

(comment 
   (failing-uber {})
 )

Evaluating the above function results in the following exception:

1. Unhandled clojure.lang.ExceptionInfo
   Unexpected lib file:
   /Users/rick/.m2/repository/org/apache/jena/apache-jena-libs/4.2.0/apache-jena-libs-4.2.0.pom
   {}
                  uber.clj:  182  clojure.tools.build.tasks.uber/explode
                  uber.clj:  141  clojure.tools.build.tasks.uber/explode
                  uber.clj:  257  clojure.tools.build.tasks.uber/uber/fn/fn
     PersistentVector.java:  343  clojure.lang.PersistentVector/reduce
                  core.clj: 6829  clojure.core/reduce
                  core.clj: 6812  clojure.core/reduce
                  uber.clj:  256  clojure.tools.build.tasks.uber/uber/fn
             protocols.clj:   49  clojure.core.protocols/iter-reduce
             protocols.clj:   75  clojure.core.protocols/fn
             protocols.clj:   75  clojure.core.protocols/fn
             protocols.clj:   13  clojure.core.protocols/fn/G
                  core.clj: 6830  clojure.core/reduce
                  core.clj: 6812  clojure.core/reduce
                  uber.clj:  254  clojure.tools.build.tasks.uber/uber
                  uber.clj:  241  clojure.tools.build.tasks.uber/uber
                  Var.java:  384  clojure.lang.Var/invoke
                   api.clj:  394  clojure.tools.build.api/uber
                   api.clj:  339  clojure.tools.build.api/uber
                      REPL:   21  build/eval25043
                      REPL:   21  build/eval25043
             Compiler.java: 7181  clojure.lang.Compiler/eval
             Compiler.java: 7136  clojure.lang.Compiler/eval
                  core.clj: 3202  clojure.core/eval
                  core.clj: 3198  clojure.core/eval
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn/fn
                  AFn.java:  152  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj: 1977  clojure.core/with-bindings*
                  core.clj: 1977  clojure.core/with-bindings*
               RestFn.java:  425  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn
                  main.clj:  437  clojure.main/repl/read-eval-print/fn
                  main.clj:  437  clojure.main/repl/read-eval-print
                  main.clj:  458  clojure.main/repl/fn
                  main.clj:  458  clojure.main/repl
                  main.clj:  368  clojure.main/repl
               RestFn.java:  137  clojure.lang.RestFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj:  662  clojure.core/apply
                regrow.clj:   20  refactor-nrepl.ns.slam.hound.regrow/wrap-clojure-repl/fn
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   84  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   56  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  152  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  202  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  201  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  825  java.lang.Thread/run

I believe this is because the above dependency has no corresponding jar file, and only contains a pom.xml which groups up a set of dependencies.

Resolve git deps via artifactory

Wed, 15 Sep 2021 13:07:15 +0000

As a follow up to https://ask.clojure.org/index.php/8866/using-git-deps-in-a-restricted-environment?show=8866#q8866

Given the increasing usage of git deps (eg tools.build) it is increasingly painful to use in corporate environments without direct github access. The majority of such corporates use tools like artifactory which, other than proxy things like maven, also allow proxying GitHub.

Specifically it allows access, via an API, that will download a specific sha or tag from github as a zip/tar archive.

In a general sense, could tools.deps be augmented to support this use case? I can see a few scenarios:

Provide a remote/root or remote/archive that works similar to a local/root, but can source from an arbitrary zip/tar provided by a url
Similarly allow git deps to resolve from some archive provided from a url

Since the url will have the tag/sha in it, presumably tools.deps would not need to download it again unless the URL was changed.

Should tools.build functions respect with-dir (from tools.deps.alpha)?

Wed, 08 Sep 2021 22:02:10 +0000

I ran across this issue with depstar but observed that tools.build behaves the same, possibly incorrect, way.

See https://github.com/seancorfield/with-dir-repro

If you wrap (b/create-basis ..) in with-dir from tools.deps.alpha, it calculates the basis from the deps.edn file in the context of the with-dir location.

However, if you use that basis with tools.build functions, they do not behave correctly -- even if they are also wrapped in the same with-dir context.

The dependencies come from the project deps.edn (Clojure 1.9.0 in this example) but the src used is from the root of the repo, not from the project folder.

This is counter-intuitive (and took me a while to track down with depstar -- where I definitely consider it to be a bug).

Hope tools.deps.alpha add feature for listing available aliases

Fri, 03 Sep 2021 07:54:40 +0000

I think add an option for listing out available aliases will be helpful when there are many alaises.

Here is my try. https://github.com/seancorfield/dot-clojure/issues/7

Why tools-deps downloads all deps again when i run the tests (locally and on CI)

Thu, 02 Sep 2021 13:50:27 +0000

Hi,

I am trying to download all deps so tests and other steps on CI should not do it.

I run this command to download all deps
clojure -Sdeps '{:mvn/local-repo ".m2"}' -A:dev:test:prod:frontend -Spath > /dev/null
I can confirm .m2 and .cpcache dirs are there.

But when I run
clojure -Sdeps '{:mvn/local-repo ".m2"}' -M:dev:test
All deps are being downloaded again.

It reproduces locally and on ci.

Sdescribe
{:version "1.10.3.967" :config-files ["/usr/local/Cellar/clojure/1.10.3.967/deps.edn" "/Users/kirill/.clojure/deps.edn" "deps.edn" ] :config-user "/Users/kirill/.clojure/deps.edn" :config-project "deps.edn" :install-dir "/usr/local/Cellar/clojure/1.10.3.967" :config-dir "/Users/kirill/.clojure" :cache-dir ".cpcache" :force false :repro false :main-aliases "" :repl-aliases ""}

getting "Function not found" when invoking clojure with -X:alias

Wed, 18 Aug 2021 16:07:57 +0000

I have two aliases in my deps.edn:

{
 :deps
 {
  org.clojure/spec.alpha {:mvn/version "0.2.194"}
  aleph/aleph {:mvn/version "0.4.7-alpha7"}
  hiccup/hiccup {:mvn/version "2.0.0-alpha2"}
  }

 :aliases 
 { 
  :web-server {
               :exec-fn web.network-service/run 
               :exec-args {:port 8080} 
               }
  :test-web-server { 
                    :exec-fn test.tnetwork-service/run 
                    :exec-args {:port 9090}
                    }
  }
 }

When invoking

clojure -X:web-server

I get

Function not found: web.network-service/run

Everything is honky dory when invoking

clojure -X:test-web-server

I don't see why would that be, the exec-funs are basically identical. Take a look at full code here: https://github.com/sidesteps/network-grid
I have :paths ["." "src"] in my global deps.edn

Did TDEPS-187 break add-libs?

Thu, 12 Aug 2021 14:02:38 +0000

TDEPS-187 changed clojure.tools.deps.alpha/calc-basis, which now merges argmaps after {:libs libs} - diff.

I'm running on tools deps at this commit, and I'm unable to add libs.

clojure.tools.deps.alpha.repl/add-libs calls calc-basis with initial-basis as the second argument, which means the :libs in initial basis replace those calculated by calc-basis.

Switching the merge order to:

`clojure
(merge master-edn cp argmaps {:libs libs})``

enables me to add libs to the classpath.

Is this a regression, or am I holding it wrong?