Enhance CLJS module build times by skipping recompilation / regeneration of modules with no source changes

Question

I am trying to improve my auto build times for an use case were we cannot use } (it is a chrome extension and chrome's security policy prohibits dynamic loading JS for the cases where it is injected into web pages). So when using the next quickest optimization } it was taking a lot of time (~30sec for even small change) as the app pulls in a lot of dependencies. I tried to use the new released modules feature to split the output into 2 files one with all the dependencies and the other with my own code and observed that clojurescript compiler does not skip re-generation of modules where the source files have not changed at all.

Digging into the code it looks like it is a minor change in the closure.clj in the function in the {{optimize-modules}} function where we could skip emitting the module whose all the source files are not changed at all.

Answer 1

Comment made by: dvdreddy

Hi,
I was unsure about the community's best practices around issues and hence created a JIRA, if you want me to do a preliminary discussion about this in (link: https://groups.google.com/forum/#!forum/clojurescript text: clojurescript) google group, I will move this to the google group.

Thank you

Answer 2

Comment made by: thheller

:modules via the Closure Compiler only work when used on the entire program. It is not possible to feed in partial code and re-use files from another compile run. The compiler will otherwise inject a conflicting goog/base.js and maybe other things.

It is however easily possible to combine the output without the Closure Compiler at all. Just concatenate all sources from a given module into a single file without any processing at all. Unfortunately there is no easy hook for this in CLJS.

Answer 3

Comment made by: dvdreddy

(link: ~thheller) Yeah I agree, what I observed is from the timestamps of the files generated is that, the closure compiler is not the step that is taking time, rather generating the source-mappings is, if we disable the source-mappings we cut down the time to ~ 5 sec (can't do without them during development)and observing the timestamps of the generated files for the cljs-base module after the js file is generated the js.map file generation was taking around more than 15 sec of the overall 30 sec compilation time. It is this part I wanted to see if we can skip

Answer 4

Comment made by: dvdreddy

(link: ~thheller) any comments on the above ?

Answer 5

Comment made by: thheller

Source maps should not take that much time. What kind of hardware/OS are you on? Are you maybe writing to a network mounted drive or something similar? Source maps can get quite big so it may be slow IO?

Answer 6

Comment made by: dvdreddy

This is a normal 2013 Macbook pro, not NAS involved. The size of the generated source map is around 6MB, not too big for a SSD, let me see if I can profile a build and get you something that is the more concrete about the bottlenecks.

Answer 7

Reference: https://clojure.atlassian.net/browse/CLJS-2481 (reported by alex+import)