Welcome! Please see the About page for a little more info on how this works.

+1 vote
in tools.gitlibs by

If two Clojure projects are started concurrently, and both require the same :git/url dependency, then one of the projects will raise an exception. This doesn't appear to be a problem for maven dependencies. We have encountered the issue during local testing and continuous integration.

Minimal repro case:

mkdir gitlibs
mkdir project
echo '{:deps {com.cognitect/test-runner {:git/url "https://github.com/cognitect-labs/test-runner" :sha "b6b3193fcc42659d7e46ecd1884a228993441182"}}}' > project/deps.edn
export GITLIBS=gitlibs
(cd project && clojure -Spath) & (cd project && clojure -Spath)

Running this is likely to raise the following exception:

Cloning: https://github.com/cognitect-labs/test-runner
Cloning: https://github.com/cognitect-labs/test-runner
Error building classpath. Cannot lock /home/sean/tmp/project/gitlibs/_repos/github.com/cognitect-labs/test-runner/config. Ensure that no other process has an open file handle on the lock file /home/sean/tmp/project/gitlibs/_repos/github.com/cognitect-labs/test-runner/config.lock, then you may delete the lock file and retry.
org.eclipse.jgit.api.errors.JGitInternalException: Cannot lock /home/sean/tmp/project/gitlibs/_repos/github.com/cognitect-labs/test-runner/config. Ensure that no other process has an open file handle on the lock file /home/sean/tmp/project/gitlibs/_repos/github.com/cognitect-labs/test-runner/config.lock, then you may delete the lock file and retry.
at org.eclipse.jgit.api.CloneCommand.call(CloneCommand.java:206)
at org.eclipse.jgit.api.CloneCommand.call(CloneCommand.java:89)
at clojure.tools.gitlibs.impl$call_with_auth.invokeStatic(impl.clj:50)
at clojure.tools.gitlibs.impl$call_with_auth.invoke(impl.clj:41)
at clojure.tools.gitlibs.impl$git_clone_bare.invokeStatic(impl.clj:71)
at clojure.tools.gitlibs.impl$git_clone_bare.invoke(impl.clj:68)
at clojure.tools.gitlibs.impl$ensure_git_dir.invokeStatic(impl.clj:110)
at clojure.tools.gitlibs.impl$ensure_git_dir.invoke(impl.clj:100)
at clojure.tools.gitlibs$resolve.invokeStatic(gitlibs.clj:33)
at clojure.tools.gitlibs$resolve.invoke(gitlibs.clj:29)
at clojure.tools.gitlibs$procure.invokeStatic(gitlibs.clj:47)
at clojure.tools.gitlibs$procure.invoke(gitlibs.clj:41)
at clojure.tools.deps.alpha.extensions.git$eval966$fn__968.invoke(git.clj:41)
at clojure.lang.MultiFn.invoke(MultiFn.java:239)
at clojure.tools.deps.alpha$expand_deps.invokeStatic(alpha.clj:181)
at clojure.tools.deps.alpha$expand_deps.invoke(alpha.clj:164)
at clojure.tools.deps.alpha$resolve_deps.invokeStatic(alpha.clj:231)
at clojure.tools.deps.alpha$resolve_deps.invoke(alpha.clj:213)
at clojure.tools.deps.alpha.script.make_classpath$create_classpath.invokeStatic(make_classpath.clj:61)
at clojure.tools.deps.alpha.script.make_classpath$create_classpath.invoke(make_classpath.clj:53)
at clojure.tools.deps.alpha.script.make_classpath$run.invokeStatic(make_classpath.clj:79)
at clojure.tools.deps.alpha.script.make_classpath$run.invoke(make_classpath.clj:72)
at clojure.tools.deps.alpha.script.make_classpath$_main.invokeStatic(make_classpath.clj:118)
at clojure.tools.deps.alpha.script.make_classpath$_main.doInvoke(make_classpath.clj:93)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.core$apply.invokeStatic(core.clj:665)
at clojure.main$main_opt.invokeStatic(main.clj:514)
at clojure.main$main_opt.invoke(main.clj:510)
at clojure.main$main.invokeStatic(main.clj:664)
at clojure.main$main.doInvoke(main.clj:616)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.main.main(main.java:40)

Caused by: org.eclipse.jgit.errors.LockFailedException: Cannot lock /home/sean/tmp/project/gitlibs/_repos/github.com/cognitect-labs/test-runner/config. Ensure that no other process has an open file handle on the lock file /home/sean/tmp/project/gitlibs/_repos/github.com/cognitect-labs/test-runner/config.lock, then you may delete the lock file and retry.
at org.eclipse.jgit.storage.file.FileBasedConfig.save(FileBasedConfig.java:209)
at org.eclipse.jgit.api.CloneCommand.fetch(CloneCommand.java:294)
at org.eclipse.jgit.api.CloneCommand.call(CloneCommand.java:200)
... 33 more

I'm not sure what the best fix is. A straightforward approach would be to acquire a blocking file lock inside clojure.tools.gitlibs.impl/call-with-auth.

For example, when cloning or checking-out $GITLIBS/_repos/github.com/cognitect-labs/test-runner, clojure.tools.gitlibs could acquire a file lock on $GITLIBS/_repos/github.com/cognitect-labs/test-runner.lock.

I'm happy to propose a patch along these lines.

1 Answer

+1 vote
by

Yes, I'm aware of this issue and we have an issue tracking it in the context of concurrent downloads in tools.deps. I think there is work to do before getting to file locking by looking at upgrading to newer jgit and understanding more about how jgit works. (Or alternately, looking at the options we have around shelling out to git instead, which might have a totally different set of needs and options.) This is an issue that we will ultimately get to but I think it interacts with these other things and can't be considered completely independently.

As a workaround, you can use tools.deps / clj with -Sthreads 1 to prevent concurrent downloads for now.

...