Race condition in .cpcache directory creation

Question

As described in https://github.com/clojure/brew-install/pull/8, in our CI we need to run several clojure tasks for the same project in parallel. Which sometimes fails with

Refreshing classpath
Error building classpath. Can't create directory: /path/to/project/.cpcache

My understanding is that it is happening in https://github.com/clojure/tools.deps/blob/master/src/main/clojure/clojure/tools/deps/util/io.clj#L53-56 where we first check whether a directory exists and then create it.

(when-not (.exists parent)
  (when-not (.mkdirs parent)
    (let [parent-name (.getCanonicalPath parent)]
      (throw (ex-info (str "Can't create directory: " parent-name) {:dir parent-name})))))

If another process manages to create it after we checked for its existence, .mkdirs returns false and this process fails. A possible solution would be to check for the directory existence again like this:

(when-not (.exists parent)
  (when-not (.mkdirs parent)
    (when-not (.exists parent)
      (let [parent-name (.getCanonicalPath parent)]
        (throw (ex-info (str "Can't create directory: " parent-name) {:dir parent-name}))))))

Answer 1

We have a ticket for this in https://clojure.atlassian.net/browse/TDEPS-250

Comments

I investigated this a bit, and it seems to me the real problem here is that Java's mkdirs does not explain why the directory could not be made. Typically the way to overcome a race for a mkdir is to check for EEXIST and if that's the error, then continue with no error, because we know that the other process won the race.

java.nio.path.createDirectories will throw FileAlreadyExistsException when it loses the race.

I see that other code in that repo already uses java.nio, so I guess converting from mkdirs to createDirectories would be possible.

Hope this is helpful!