Duplicate source files passed to Closure Compiler causes ModuleLoader#resolvePaths to throw "Duplicate module path after

Question

I have a small (not minimal) repo here: https://github.com/au-phiware/boot-parsets

The repo includes a es6 module and a cljs file that both depend on a node module, which produces the following error.


Writing main.cljs.edn...
Compiling ClojureScript...
• main.js
                                       java.lang.Thread.run                  Thread.java:  748
         java.util.concurrent.ThreadPoolExecutor$Worker.run      ThreadPoolExecutor.java:  617
          java.util.concurrent.ThreadPoolExecutor.runWorker      ThreadPoolExecutor.java: 1142
                        java.util.concurrent.FutureTask.run              FutureTask.java:  266
                                                        ...                                   
                        clojure.core/binding-conveyor-fn/fn                     core.clj: 2022
                              adzerk.boot-cljs/compile-1/fn                boot_cljs.clj:  160
                                   adzerk.boot-cljs/compile                boot_cljs.clj:   72
                                          boot.pod/call-in*                      pod.clj:  413
                                                        ...                                   
org.projectodd.shimdandy.impl.ClojureRuntimeShimImpl.invoke  ClojureRuntimeShimImpl.java:  102
org.projectodd.shimdandy.impl.ClojureRuntimeShimImpl.invoke  ClojureRuntimeShimImpl.java:  109
                                                        ...                                   
                                          boot.pod/call-in*                      pod.clj:  410
                                      boot.pod/eval-fn-call                      pod.clj:  359
                                         clojure.core/apply                     core.clj:  657
                                                        ...                                   
                         adzerk.boot-cljs.impl/compile-cljs                     impl.clj:  151
                                       cljs.build.api/build                      api.clj:  205
                                         cljs.closure/build                  closure.clj: 2595
                             cljs.closure/handle-js-modules                  closure.clj: 2496
                            cljs.closure/process-js-modules                  closure.clj: 2389
                            cljs.closure/convert-js-modules                  closure.clj: 1680
                com.google.javascript.jscomp.Compiler.parse                Compiler.java:  995
          com.google.javascript.jscomp.Compiler.parseInputs                Compiler.java: 1731
      com.google.javascript.jscomp.deps.ModuleLoader.<init>            ModuleLoader.java:   92
com.google.javascript.jscomp.deps.ModuleLoader.resolvePaths            ModuleLoader.java:  276
java.lang.IllegalArgumentException: Duplicate module path after resolving: /home/corin/Projects/Demos/boot-parsets/node_modules/d3/d3.js
        clojure.lang.ExceptionInfo: Duplicate module path after resolving: /home/corin/Projects/Demos/boot-parsets/node_modules/d3/d3.js
    from: :boot-cljs
        clojure.lang.ExceptionInfo: Duplicate module path after resolving: /home/corin/Projects/Demos/boot-parsets/node_modules/d3/d3.js
    line: 33


Run `boot cljs` to reproduce the issue.

The patch attach removes duplicates from the set of input source files before they are preprocessed. With this patch the repo compiles correctly.

Answer 1

Comment made by: mfikes

Hi Corin,

1. Have you signed the CA? (Your name doesn't appear on https://clojure.org/community/contributors)
2. Can you provide a minimal repro that doesn't employ downstream tooling? (Bug-filing details are at https://clojurescript.org/community/reporting-issues)

Answer 2

Comment made by: phiware

1. I have signed the CA (I did it after filing this bug).
2. Not a problem, I'll get on to that later today. Is it okay to link to the project on GitHub, or should I upload a tarball?

Answer 3

Comment made by: mfikes

Hi Corin, it is not OK to link to GitHub, and any repro should not make use of any downstream tooling (Leiningen, Boot, etc.)—this means that a repro would ideally depend only on the shipping {{cljs.jar}}, executable like the examples at Quick Start https://clojurescript.org/guides/quick-start

Answer 4

Comment made by: mfikes

Hey Corin, you may want to submit the patch using the instructions at https://clojurescript.org/community/patches (your current patch won't apply using {{git am}}, which is what I suspect David uses in the end.

Answer 5

_Comment made by: mfikes_

{{lein test}} is failing when applying {{patch}}


FAIL in (commonjs-module-processing) (module_processing_tests.clj:54)
processed modules are added to :libs
expected: (= {:foreign-libs [], :ups-foreign-libs [], :libs [(test/platform-path "out/src/test/cljs/reactJS.js") (test/platform-path "out/src/test/cljs/Circle.js")], :closure-warnings {:non-standard-jsdoc :off}} (env/with-compiler-env cenv (closure/process-js-modules {:foreign-libs [{:file "src/test/cljs/reactJS.js", :provides ["React"], :module-type :commonjs} {:file "src/test/cljs/Circle.js", :provides ["Circle"], :module-type :commonjs, :preprocess :jsx}], :closure-warnings {:non-standard-jsdoc :off}})))
  actual: (not (= {:foreign-libs [], :ups-foreign-libs [], :libs ["out/src/test/cljs/reactJS.js" "out/src/test/cljs/Circle.js"], :closure-warnings {:non-standard-jsdoc :off}} {:foreign-libs [], :closure-warnings {:non-standard-jsdoc :off}, :libs ["out/src/test/cljs/Circle.js" "out/src/test/cljs/reactJS.js"], :ups-foreign-libs []}))

Answer 6

Comment made by: phiware

Thanks Mike,

I will make an effort follow all those instructions, but I do not see how I should provide the repro (no links, no attachments)... should it be inline code blocks?

Also, I shall be sure to run {{lein test}} before submitting the next patch. I note that the difference is only the order of the items in {{:libs}} vector, can you advise if order is important?

Answer 7

Comment made by: mfikes

Hi Corin,

Yes, inline code blocks are great. Anything minimal that doesn't depend on more than the shipping {{cljs.jar}} to demonstrate the issue (either directly in its REPL or via compiling using the build API). Here is a recent example using the build API: https://dev.clojure.org/jira/browse/CLJS-2397?focusedCommentId=47278&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-47278

There are actually other tests as well beyond {{lein test}}. See https://clojurescript.org/community/running-tests

I haven't looked into the details of this issue, so can't speak to to whether order of items is important.

Answer 8

_Comment made by: phiware_

h2. Steps to reproduce the problem.

Consider the following three source files:

{code:title=src/distinct_inputs/core.cljs}
(ns distinct-inputs.core
  (:require [d3]
            [circle :refer [circle]]))

(-> d3
    (.select "body")
    (.append "svg")
    (.attr "width" 200)
    (.attr "height" 200)
    (.call circle "steelblue"))


{code:title=es6/circle.js}
import * as d3 from 'd3';

export function circle(sel, color) {
  return sel
    .append("circle")
    .attr("cx", "100px")
    .attr("cy", "100px")
    .attr("r",  "100px")
    .attr("fill", color);
}


{code:title=build.clj}
(require 'cljs.build.api)

(cljs.build.api/build
  "src"
  {:main 'distinct-inputs.core
   :output-to "out/main.js"
   :install-deps true
   :foreign-libs [{:file "es6"
                   :module-type :es6}]
   :npm-deps     {:d3 "3.5.16"}})


Execute {{cljs}}:

{code:none}
java -cp cljs.jar:src clojure.main build.clj


h2. Expected outcome

{{cljs}} should produce nothing to the standard output, exit cleanly and write the following files (approximately).

{code:title=out/main.js}
var CLOSURE_UNCOMPILED_DEFINES = {};
var CLOSURE_NO_DEPS = true;
if(typeof goog == "undefined") document.write('<script src="out/goog/base.js"></script>');
document.write('<script src="out/goog/deps.js"></script>');
document.write('<script src="out/cljs_deps.js"></script>');
document.write('<script>if (typeof goog == "undefined") console.warn("ClojureScript could not load :main, did you forget to specify :asset-path?");</script>');
document.write('<script>goog.require("process.env");</script>');
document.write('<script>goog.require("distinct_inputs.core");</script>');


{code:title=out/distinct_inputs/core.js}
goog.provide('distinct_inputs.core');
goog.require('cljs.core');
goog.require('module$distinct_inputs$node_modules$d3$d3');
goog.require('module$distinct_inputs$es6$circle');
module$distinct_inputs$node_modules$d3$d3.select("body").append("svg").attr("width",(200)).attr("height",(200)).call(module$distinct_inputs$es6$circle.circle,"steelblue");

//# sourceMappingURL=core.js.map


{code:title=out/es6/circle.js}
goog.provide("module$distinct_inputs$es6$circle");goog.require("module$distinct_inputs$node_modules$d3$d3");function circle$$module$distinct_inputs$es6$circle(sel,color){return sel.append("circle").attr("cx","100px").attr("cy","100px").attr("r","100px").attr("fill",color)}module$distinct_inputs$es6$circle.circle=circle$$module$distinct_inputs$es6$circle


h2. Actual outcome.

{{cljs}} exits with exit code 1 and produces the following standard out.

{code:none|title=stdout}
Exception in thread "main" java.lang.IllegalArgumentException: Duplicate module path after resolving: /distinct_inputs/node_modules/d3/d3.js, compiling:(/distinct_inputs/build.clj:3:1)
    at clojure.lang.Compiler.load(Compiler.java:7391)
    at clojure.lang.Compiler.loadFile(Compiler.java:7317)
    at clojure.main$load_script.invokeStatic(main.clj:275)
    at clojure.main$script_opt.invokeStatic(main.clj:335)
    at clojure.main$script_opt.invoke(main.clj:330)
    at clojure.main$main.invokeStatic(main.clj:421)
    at clojure.main$main.doInvoke(main.clj:384)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at clojure.lang.Var.invoke(Var.java:379)
    at clojure.lang.AFn.applyToHelper(AFn.java:154)
    at clojure.lang.Var.applyTo(Var.java:700)
    at clojure.main.main(main.java:37)
Caused by: java.lang.IllegalArgumentException: Duplicate module path after resolving: /distinct_inputs/node_modules/d3/d3.js
    at com.google.javascript.jscomp.deps.ModuleLoader.resolvePaths(ModuleLoader.java:276)
    at com.google.javascript.jscomp.deps.ModuleLoader.<init>(ModuleLoader.java:92)
    at com.google.javascript.jscomp.Compiler.parseInputs(Compiler.java:1731)
    at com.google.javascript.jscomp.Compiler.parse(Compiler.java:995)
    at cljs.closure$convert_js_modules.invokeStatic(closure.clj:1680)
    at cljs.closure$process_js_modules.invokeStatic(closure.clj:2371)
    at cljs.closure$handle_js_modules.invokeStatic(closure.clj:2495)
    at cljs.closure$build.invokeStatic(closure.clj:2592)
    at cljs.build.api$build.invokeStatic(api.clj:204)
    at cljs.build.api$build.invoke(api.clj:189)
    at cljs.build.api$build.invokeStatic(api.clj:192)
    at cljs.build.api$build.invoke(api.clj:189)
    at user$eval24.invokeStatic(build.clj:3)
    at user$eval24.invoke(build.clj:3)
    at clojure.lang.Compiler.eval(Compiler.java:6927)
    at clojure.lang.Compiler.load(Compiler.java:7379)
    ... 11 more


None of the aforementioned expected files are produced.

h2. Cause of the exception.

The exception emitted by {{ModuleLoader#resolvePaths}} is a result of the same input file (i.e. {{node_modules/d3/d3.js}}) having been specified more than once to {{Compiler#initModules}}. There happens to be this note in {{Compiler#getAllInputsFromModules}}:

{code:title=src/com/google/javascript/jscomp/Compiler.java}
        // NOTE(nicksantos): If an input is in more than one module,
        // it will show up twice in the inputs list, and then we
        // will get an error down the line.


{{cljs.closure/process-js-modules}} is provided a {{:foreign-libs}} vector which contains a repeated entry for {{node_modules/d3/d3.js}} (and also it's {{package.json}}). That vector is a result of multiple invocations of {{cljs.closure/node-inputs}}; once for {{out/cljs$node_modules.js}} (which is presumably a result of the dependency in {{distinct_inputs/core}}) and again for {{es6/circle.js}}.

In short, the dependency on D3 is pulled in by both ClojureScript source files and JavaScript module source files.

h2. Proposed solution.

In this scenario the {{:foreign-libs}} vector contains repeated entries dispite the use of {{distinct}} within {{cljs.closure/node-inputs}}. A possible solution would to remove the use of {{distinct}} within {{cljs.closure/node-inputs}} and require the caller of {{cljs.closure/node-inputs}} to use {{distinct}}.

{code:title=Solution A}
From 063e35080c14d35189ab7827f25f071e958ab5b4 Mon Sep 17 00:00:00 2001
From: Corin Lawson <corin@responsight.com>
Date: Tue, 21 Nov 2017 01:31:53 +1100
Subject: [PATCH] CLJS-2402: Ensure :foreign-libs vector contains distinct
 entries.

---
 src/main/clojure/cljs/closure.clj | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)

diff --git a/src/main/clojure/cljs/closure.clj b/src/main/clojure/cljs/closure.clj
index a686f878..74a0cc86 100644
--- a/src/main/clojure/cljs/closure.clj
+++ b/src/main/clojure/cljs/closure.clj
@@ -2219,7 +2219,7 @@
      (when env/*compiler*
        (:options @env/*compiler*))))
   ([entries opts]
-   (into [] (distinct (mapcat #(node-module-deps % opts) entries)))))
+   (into [] (mapcat #(node-module-deps % opts) entries))))
 
 (defn index-node-modules
   ([modules]
@@ -2480,14 +2480,15 @@
         output-dir (util/output-directory opts)
         opts (update opts :foreign-libs
                (fn [libs]
-                 (into (if (= target :nodejs)
-                         []
-                         (index-node-modules node-required))
-                   (into expanded-libs
-                     (node-inputs (filter (fn [{:keys [module-type]}]
-                                            (and (some? module-type)
-                                              (not= module-type :amd)))
-                                    expanded-libs))))))
+                 (distinct
+                   (into (if (= target :nodejs)
+                           []
+                           (index-node-modules node-required))
+                         (into expanded-libs
+                               (node-inputs (filter (fn [{:keys [module-type]}]
+                                                      (and (some? module-type)
+                                                           (not= module-type :amd)))
+                                                    expanded-libs)))))))
         opts (if (some
                    (fn [ijs]
                      (let [dest (io/file output-dir (rel-output-path (assoc ijs :foreign true) opts))]
--
2.13.0



A more general solution is {{cljs.closure/process-js-modules}} must ensure the set of input files (i.e. {{js-modules}}) is distinct. This patch would be simpler (i.e. doesn't mess with code I don't understand) and closer to the call to Google Closure Compiler.

{code:title=Solution B}
From 6bf11a24b93642e118e6d29c5af8a137fa01ea94 Mon Sep 17 00:00:00 2001
From: Corin Lawson <corin@responsight.com>
Date: Sun, 19 Nov 2017 20:25:31 +1100
Subject: [PATCH] CLJS-2402: Ensure input source files are distinct.

---
 src/main/clojure/cljs/closure.clj | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/src/main/clojure/cljs/closure.clj b/src/main/clojure/cljs/closure.clj
index a686f878..24421bde 100644
--- a/src/main/clojure/cljs/closure.clj
+++ b/src/main/clojure/cljs/closure.clj
@@ -2364,7 +2364,8 @@
   (let [;; Modules from both :foreign-libs (compiler options) and :ups-foreign-libs (deps.cljs)
         ;; are processed together, so that files from both sources can depend on each other.
         ;; e.g. commonjs module in :foreign-libs can depend on commonjs module from :ups-foreign-libs.
-        js-modules (filter :module-type (concat (:foreign-libs opts) (:ups-foreign-libs opts)))]
+        js-modules (filter :module-type (concat (:foreign-libs opts) (:ups-foreign-libs opts)))
+        js-modules (distinct js-modules)]
     (if (seq js-modules)
       (util/measure (:compiler-stats opts)
         "Process JS modules"
--
2.13.0



FWIW: I prefer Solution B.

Answer 9

Comment made by: phiware

Attached proposed Solution B

Answer 10

Comment made by: phiware

Hi Mike,

I hope this is to your's (and BDFL's) satisfaction now; I ran {{lein test}} for both proposed solutions and I do not receive any failures. I do receive errors, however, that do not occur in assertions. I assume that the cause is something peculiar (or lack thereof) in my setup. Let me know if you require anything else from me.

Cheers,
Corin.

Answer 11

Comment made by: mfikes

Thanks Corin. The entire test suite passes for me with your latest patch.

Answer 12

Comment made by: mfikes

CLJS-2402.patch added to Patch Tender (i)

Answer 13

Reference: https://clojure.atlassian.net/browse/CLJS-2402 (reported by phiware)