tools.deps shares one Maven DefaultModelValidator across threads, and dependency expansion fails at random

Question

Dependency expansion sometimes fails with this error. We use Clojure CLI 1.12.5.1664 (tools.deps 0.31.1642):

java.lang.ClassCastException: class java.util.HashMap$Node cannot be cast to class java.util.HashMap$TreeNode (java.util.HashMap$Node and java.util.HashMap$TreeNode are in module java.base of loader 'bootstrap')
    at java.base/java.util.HashMap$TreeNode.moveRootToFront(HashMap.java:1994)
    at java.base/java.util.HashMap$TreeNode.treeify(HashMap.java:2110)
    at java.base/java.util.HashMap.treeifyBin(HashMap.java:778)
    at java.base/java.util.HashMap.putVal(HashMap.java:650)
    at java.base/java.util.HashMap.put(HashMap.java:618)
    at java.base/java.util.HashSet.add(HashSet.java:229)
    at org.apache.maven.model.validation.DefaultModelValidator.validateId(DefaultModelValidator.java:1153)
    at org.apache.maven.model.validation.DefaultModelValidator.validateEffectiveDependency(DefaultModelValidator.java:888)
    at org.apache.maven.model.validation.DefaultModelValidator.validateEffectiveDependencies(DefaultModelValidator.java:776)
    at org.apache.maven.model.validation.DefaultModelValidator.validateEffectiveModel(DefaultModelValidator.java:487)
    at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:548)
    ...
    at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:183)
    at org.eclipse.aether.internal.impl.DefaultRepositorySystem.readArtifactDescriptor(DefaultRepositorySystem.java:269)
    at clojure.tools.deps.extensions.maven$read_descriptor.invokeStatic(maven.clj:117)
    ...
    at clojure.tools.deps$expand_deps$children_task__976$fn__978$fn__979.invoke(deps.clj:303)
    ...
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)

This problem is probably the same as https://ask.clojure.org/index.php/15192/concurrency-bug-in-tools-deps-use-of-aether-maven-libs. The error at the end of that report is the same HashMap$TreeNode error.

Cause

In Maven, DefaultModelValidator is a singleton. It keeps valid ids in a plain HashSet with the name validIds. validateId reads and writes this set without a lock. If two threads write to the set at the same time, the internal state of the HashMap breaks. The ClassCastException above is one result of this race condition. Maven 3.8.8 and Maven 3.9.16 both have this HashSet.

All expansion threads of tools.deps use the same validator:

• In 0.31.1638 and later, read-descriptor uses session/retrieve for :mvn/context, :mvn/system, and :mvn/session. Thus all threads share one context, one RepositorySystem, and one validator.
• In 0.29.1598, each thread calls session/retrieve-local to get its own RepositorySystem. But make-system gets the RepositorySystem from one shared DefaultServiceLocator, and the locator keeps one instance of each service. Because of this, all threads also get the same RepositorySystem, the same model builder, and the same validator.

I did not examine versions 0.31.1619 to 0.31.1629, which use MIMA with session/retrieve-local.

Upstream Maven status

Maven fixed the HashSet on master only: https://github.com/apache/maven/issues/11618 and https://github.com/apache/maven/pull/11734. The maven-3.9.x branch still has the plain HashSet. Thus no Maven 3.9.x release has the fix.

Reproduction

The failure does not occur on each run. It occurs only when the classpath is not in the cache. The dependency graph must also be large, so that several expansion threads read POMs at the same time.

To make a failure more probable, use a project with many Maven dependencies. Run this command many times:

clojure -Sforce -Spath

Workaround

With one expansion thread, the race condition cannot occur:

• For the Clojure CLI, add -Sthreads 1.
• For clojure.tools.build.api/create-basis, add :args {:threads 1}. For example: (b/create-basis {:project "deps.edn" :args {:threads 1}})

Versions

• Clojure CLI 1.12.5.1664 (tools.deps 0.31.1642, maven-model-builder 3.9.16, maven-resolver-provider 3.9.16)
• tools.build 0.10.13 (tools.deps 0.29.1598, maven-model-builder 3.8.8)
• The changelog of tools.deps 0.31.1646 shows no fix for this problem.
• macOS 26.6.2

Possible fixes

• Give each expansion thread its own RepositorySystem. Make each RepositorySystem from its own service locator or MIMA context.
• When a Maven release keeps validIds in a concurrent set, use that release.

Answer 1

There are open and reviewed PRs to backport the Maven 4.x fix back to 3.9 and 3.10. I am watching those and waiting for one of them to be applied, will then update to use it.

Maven 4.x is a big jump, do not want to go there yet.

For now, a workaround is to use -Sthreads 1 to use only 1 expansion thread.