s3 repo + certain deps appear to cause infinite resolution loop

Question

Hi, I was playing around with getting the AWS CDK working with a Datomic Cloud project and ran into this issue (CLI 1.10.3.855). It seems that if an S3 repo (e.g. the datomic repo) is defined and certain deps are defined, CLI/deps seems to go into an infinite resolution loop. after a certain point I see the same set of "Downloading: from <repo-alias>" messages until it craps out with "Too many open files" exception

Here's a stripped down version of the deps.edn that causes the problem. Note that commenting out the :mvn/repos eliminates the problem. Further the iam, and core CDK deps on their own (with the :mvn/repos defined)don't seem to cause it either.

{:paths   ["src" "resources"]
 :deps    {org.clojure/clojure {:mvn/version "1.10.3"}
           ;com.datomic/ion                  {:mvn/version "0.9.50"},
           ;com.datomic/client-cloud         {:mvn/version "0.8.113"},
           }
 :mvn/repos
          {"datomic-cloud" {:url "s3://datomic-releases-1fc2183a/maven/releases"}}
 :aliases {:ion-dev {:extra-deps {com.datomic/ion-dev {:mvn/version "0.9.290"}}}
           :cdk     {:extra-paths ["cdk"]
                     :extra-deps  {
                                   software.amazon.awscdk/codebuild {:mvn/version "1.115.0"}
                                   ;software.amazon.awscdk/core      {:mvn/version "1.115.0"}
                                   ;software.amazon.awscdk/iam       {:mvn/version "1.115.0"}
                                   }}}}

Answer 1

Logged as https://clojure.atlassian.net/browse/TDEPS-199

I was able to reproduce this. There's some parallelism happening that can make the errors confusing, so using -Sthreads 1 helped a little there. I'm a little suspicious that this has more to do with sockets, and not with actual files. But, it needs more investigation.

Comments

Great thanks!   yeah not sure what was going on when i was investigating it earlier, I was subsequently able to get it to fail in a REPL for the tools.deps  project itself.

---

Just as an FYI, yeah.. sockets lol.  Couldn't attach the pics, but after tweaking the  JVM to allow more file handles for my REPL session it ran much longer, and ended up opening 25k ports before it failed

---

sounds suspicious, thx. I will look at in more depth at some point, but might be a while

---

No problem.  But, I think I've fixed it ;)  I just got rid of the code that explicitly passes the http-client to the aws/client.  I have a patch, but don't have a JIRA acct

---

To provide a patch, I would need you to follow the contributor process (sign a Contributor Agreement, request access to jira, attach a patch to the jira). See https://clojure.org/dev/dev#_becoming_a_contributor

Or if you just want to point at some code, that's fine too.

---

Filled out the contrib agreement, but in the meantime, it's on my fork: https://github.com/eoliphan/tools.deps.alpha

Answer 2

A bit more info (and weirdness). So, I decided to take a look at it myself, stripped down my project and pulled in 0.12.985 directly. Running the following in the REPL gives me the same issue:

(require '[clojure.tools.deps.alpha :as deps])
  (require '[clojure.tools.deps.alpha.util.maven :as mvn])
  (deps/resolve-deps {:paths ["src" "resources"]
                      :deps  '{org.clojure/clojure              {:mvn/version "1.10.3"}
                               com.datomic/ion                  {:mvn/version "0.9.50"},
                               software.amazon.awscdk/codebuild {:mvn/version "1.115.0"}}
                      :mvn/repos
                             (merge mvn/standard-repos {"datomic-cloud" {:url "s3://datomic-releases-1fc2183a/maven/releases"}})} {})

But.. I then pulled the tools.deps.alpha code, and ran the same thing in the Rich Comment Block, and it works fine.

 (print-tree
    (resolve-deps {:paths ["src" "resources"]
                   :deps  '{org.clojure/clojure              {:mvn/version "1.10.3"}
                            com.datomic/ion                  {:mvn/version "0.9.50"},
                            software.amazon.awscdk/codebuild {:mvn/version "1.115.0"}}
                   :mvn/repos
                          (merge mvn/standard-repos {"datomic-cloud" {:url "s3://datomic-releases-1fc2183a/maven/releases"}})}
      {}))

I first assumed that it'd been fixed on master, but even checking out the 0.12.985 tag, and rerunning it seems to be fine. In my test project, making tools.deps a :local/root or :git/url based dep results in it working as expected.

So I'm assuming that maybe there's some weirdness in the CLI.jar itself. I tried upgrading to the latest 1.10.3.929, but the issue persists via the CLI