Why tools-deps downloads all deps again when i run the tests (locally and on CI)

Question

Hi,

I am trying to download all deps so tests and other steps on CI should not do it.

I run this command to download all deps
clojure -Sdeps '{:mvn/local-repo ".m2"}' -A:dev:test:prod:frontend -Spath > /dev/null
I can confirm .m2 and .cpcache dirs are there.

But when I run
clojure -Sdeps '{:mvn/local-repo ".m2"}' -M:dev:test
All deps are being downloaded again.

It reproduces locally and on ci.

Sdescribe
{:version "1.10.3.967" :config-files ["/usr/local/Cellar/clojure/1.10.3.967/deps.edn" "/Users/kirill/.clojure/deps.edn" "deps.edn" ] :config-user "/Users/kirill/.clojure/deps.edn" :config-project "deps.edn" :install-dir "/usr/local/Cellar/clojure/1.10.3.967" :config-dir "/Users/kirill/.clojure" :cache-dir ".cpcache" :force false :repro false :main-aliases "" :repl-aliases ""}

Answer 1

Please share a deps.edn that reproduces this.

Comments

```
  {:deps {;; Platform and system logic
        aero/aero                           {:mvn/version "1.1.6"}
        buddy/buddy-sign                    {:mvn/version "3.2.0"}
        ch.qos.logback/logback-classic      {:mvn/version "1.2.3"}
        clj-htmltopdf/clj-htmltopdf         {:mvn/version "0.1-alpha7"}
        clojure.java-time/clojure.java-time {:mvn/version "0.3.2"}
        com.novemberain/pantomime           {:mvn/version "2.11.0"}
        cryptohash-clj/cryptohash-clj       {:mvn/version "0.1.6"}
        expound/expound                     {:mvn/version "0.8.7"}
        haversine/haversine                 {:mvn/version "0.1.1"}
        integrant/repl                      {:mvn/version "0.3.2"}
        integrant/integrant                 {:mvn/version "0.8.0"}
        metosin/spec-tools                  {:mvn/version "0.10.4"}
        org.apache.pdfbox/pdfbox            {:mvn/version "2.0.22"}
        org.apache.pdfbox/pdfbox-tools      {:mvn/version "2.0.22"}
        org.bouncycastle/bcprov-jdk15on     {:mvn/version "1.67"}
        org.clojure/clojure                 {:mvn/version "1.10.1"}
        org.clojure/core.async              {:mvn/version "1.3.610"}
        org.clojure/core.cache              {:mvn/version "1.0.207"}
        org.clojure/data.csv                {:mvn/version "1.0.0"}
        org.clojure/tools.cli               {:mvn/version "1.0.194"}

        ;; Database and SQL
        hikari-cp/hikari-cp       {:mvn/version "2.13.0"}
        honeysql/honeysql         {:mvn/version "1.0.444"}
        lambdaisland/uri          {:mvn/version "1.4.54"}
        migratus/migratus         {:mvn/version "1.3.3"}
        nilenso/honeysql-postgres {:mvn/version "0.4.112"}
        org.clojure/java.jdbc     {:mvn/version "0.7.11"}
        org.postgresql/postgresql {:mvn/version "42.2.18"}

        ;; HTTP and web
        buddy/buddy-auth      {:mvn/version "2.2.0"}
        clj-http/clj-http     {:mvn/version "3.10.3"}
        hiccup/hiccup         {:mvn/version "1.0.5"}
        metosin/reitit        {:mvn/version "0.5.10"}
        org.clojure/data.json {:mvn/version "1.0.0"}
        ring/ring-defaults    {:mvn/version "0.3.2"}
        ring/ring-json        {:mvn/version "0.5.0"}
        ring/ring             {:mvn/version "1.8.2"}

        ;; Integration with 3rd parties
        circleci/rollcage       {:mvn/version "1.0.208"}
        clj-commons/iapetos     {:mvn/version "0.1.11"}
        io.prometheus/simpleclient_hotspot {:mvn/version "0.11.0"}
        clj-commons/clj-ssh     {:mvn/version "0.5.15"}
        com.sun.mail/javax.mail {:mvn/version "1.6.2"}
        ovotech/clj-gcp         {:mvn/version "0.6.11"}}

 :paths ["src/clj" "src/cljc" "resources"]

 :aliases
 {:dev {:extra-paths ["env/dev/clj"]
        :extra-deps  {clj-commons/clj-yaml        {:mvn/version "0.7.0"}
                      criterium/criterium         {:mvn/version "0.4.6"}
                      pogonos/pogonos             {:mvn/version "0.1.1"}
                      ubergraph/ubergraph         {:mvn/version "0.8.2"}
                      vlaaad/reveal               {:mvn/version "1.3.196"}

                      ;; to be deleted?
                      binaryage/devtools {:mvn/version "0.9.10"}}}

  :prod {:extra-paths ["env/prod/clj"]}

  :repl {:extra-paths ["env/dev/resources"]
         :ns-default  cockpit.repl
         :jvm-opts    ["-Dvlaaad.reveal.prefs={:font-size 30 :font-family \"Roboto\"}"]}

  :test {:extra-paths ["test" "test-resources"]
         :extra-deps {clj-http-fake/clj-http-fake {:mvn/version "1.0.3"}
                      nubank/matcher-combinators  {:mvn/version "3.1.4"}
                      cloverage/cloverage           {:mvn/version "1.2.2"}
                      lambdaisland/kaocha {:mvn/version "1.0.887"}
                      lambdaisland/kaocha-cloverage {:mvn/version "1.0.75"}
                      lambdaisland/kaocha-junit-xml {:mvn/version "0.0.76"}}
         :main-opts  ["-m" "kaocha.runner"]}

  :uberjar {:replace-deps {com.github.seancorfield/depstar {:mvn/version "2.1.278"}}
            :exec-fn      hf.depstar/uberjar
            :exec-args    {:jar             "app.jar"
                           :compile-aliases [:prod]
                           :aot             true
                           :main-class      app.server}}

  :frontend {:replace-paths ["src/cljs" "src/cljc"]
             :deps  {thheller/shadow-cljs        {:mvn/version "2.9.10"}
                     camel-snake-kebab           {:mvn/version "0.4.1"}
                     cljs-ajax                   {:mvn/version "0.8.0"}
                     cljs-http                   {:mvn/version "0.1.45"}
                     com.andrewmcveigh/cljs-time {:mvn/version "0.5.2"}
                     day8.re-frame/http-fx       {:mvn/version "0.1.6"}
                     metosin/reitit              {:mvn/version "0.3.10"}
                     re-frame                    {:mvn/version "1.2.0"}
                     reagent                     {:mvn/version "1.0.0"}
                     tongue                      {:mvn/version "0.3.0"}
                     fork                        {:mvn/version "2.2.4"}
                     phrase                      {:mvn/version "0.3-alpha4"}
                     binaryage/oops              {:mvn/version "0.7.0"}}
             :main-opts  ["-m" "shadow.cljs.devtools.cli"]}}}
```

first command which I expect should fetch all deps
```
clojure -Sdeps '{:mvn/local-repo ".m2"}' -A:dev:test:prod:frontend -Spath
```

When I run tests it is expected no deps to be fetched
```
clojure -Sdeps '{:mvn/local-repo ".m2"}' -M:dev:test --profile :ci
```
```

---

I haven't not looked at this in depth yet, but by specifying different aliases, you are getting different sets of deps, so it is downloading new versions of deps.

For example, the first one does not use honeysql and the second one does so it has to be downloaded. If there are version differences between the two calculations, then the second might download a new version of a lib it already has.

Rerunning either individual command you list does not download new libs, the problem is these commands are different dep sets so your assumption that no downloads should occur is incorrect.

---

ah, right, `frontedn` excludes a lot of deps. makes sense.

thanks for the check

Answer 2

When you use -Sdeps, the cache is never used (because deps on the command line is not part of the cache key calculation).

Comments

thanks for the answer!
is there a way to avoid redundant deps download if deps already fetched? (when using project local `.m2` directory)?

---

To circle back for clarity, the classpath cache is not being used, but the jars in the repo should be and you should not be seeing it download deps that already exist in the local repo. I can't reproduce that with something similar to your example (although I don't have your deps.edn so can't really tell for sure).