Background
While working on CLJS-3079 I noticed some Windows support for bash/sh scripts in {{script/aot_core}}. This made me think that I should verify my bash/sh script changes on Windows.
Windows support was added for {{script/aot_core}} in CLJS-1797 to support running under Git Bash. Git Bash is part of (link: https://gitforwindows.org/ text: Git for Windows).
While testing with Git Bash, I discovered support was specific to {{script/aot_core}} and lacking in other scripts. I asked on (link: https://clojurians.slack.com/messages/C07UQ678E text: Slack #cls-dev) if there is interest in upping support for bash/sh scripts on Windows and got an affirmative, hence this JIRA issue.
Actual behavior
Some sh/bash scripts currently fail on Window under Git Bash.
Desired behavior
Flesh out support for sh/bash script on Windows under Git Bash.
Related
CLJS-3075 - changes overlap - will need to be tweaked to support graaljs under Windows.
Windows Environment
Windows 10 Home v1903 64-bit with 6gb of RAM.
|tool|install via|
| :-- | :-- | :-- |
|Git for Windows v2.21.0|manual download and install from https://gitforwindows.org/
This gives us Git Bash in which I did all my bash/sh testing|
|java|{{scoop add bucket java}}
{{scoop install corretto8}}|
|JavaScript Engines|Installed via (link: https://github.com/GoogleChromeLabs/jsvu text: jsvu)|
|maven|{{scoop install maven}}|
|leiningen|{{scoop install leiningen}}|
|node|{{scoop install nodejs}}|
|google chrome web browser|manual download and install and set as default browser|
|clj|as per https://github.com/clojure/tools.deps.alpha/wiki/clj-on-Windows#install|
Scope
There are a couple of Windows PowerShell scripts under {{script/}}. This work is only for the bash/sh scripts. I won't touch the ps1 scripts.
See also: Observations section below.
Analysis
classpath format
Most script failures are due to java classpath formatting differences between Windows and Linux/macOS.
default encoding
Some tests fail because the default java file.encoding is not utf-8 when running under Git Bash on Windows. (Note that I found this also to be true when testing on a Docker ubuntu image).
tooling
unzip behaves a bit differently on Windows
script/test-cli
More failures here, but all due to minor differences in linux/macos vs windows, in addition to above:
1. embedded double quotes in command line args need to be escaped in Windows
1. test results need to account for line ending differences on Windows
1. windows paths need to be escaped for JavaScript (JavaScript eats the backslashes)
1. graaljs {{js}} is {{js.cmd}} on windows
Approach
classpath format
Solve classpath formatting issues with cygpath which is included with Git Bash. Do this in a dedicated script and call from other scripts that require it.
default encoding
Explicitly specify encoding at utf-8 where necessary.
tooling
The unzip utility requires } to deep match files (we were using ). This is compatible with unix/macos.
script/test-cli.
Address issues uncovered during analysis. We'll have to touch main graaljs and nahsorn clj repl code in addition to bash scripts and test-cli util code.
Implementation notes
- Java classpath conversion support is in the new {{bin/classpath_conv}}. I would have put it under {{script}} but {{bin/cljsc}} makes use of it, so thought better to place under {{bin}}.
- Only one test needed to be changed: {{src/test/cljs_cli/cljs_cli/test}} {{eval-test}} was passing in {{nil}} as a command line argument, this was translated to empty string, which on Windows, does not work. I assume the test really wanted to actually pass in nil so the argument has been changed to {{"nil"}}. Test otherwise remains the same.
- I added {{script/verify-scripts}} to run all scripts. Forces trace of commands (set -x) for all scripts called and saves output in a way that makes comparison with other patches/versions easy. Example usage is in attached {{verify.sh}}.
- I explicitly set {{file.encoding}} to {{UTF-8}} in {{bin/cljsc}}. I think this is appropriate but I'm sure someone will let me know if it is not.
- {{script/closure-library-release/closure-library-release.sh}} - updated github url to use https: instead of git: to reduce setup burden on developers.
- {{script/test-compile}} required a tweak not directly related to this issue to function. It was specifying a stale location for the closure jar.
- all scripts switched over to from
#\!/bin/sh to #\!/usr/bin/env bash to allow me and other maintainers to confidently use bash features. This also allowed my custom verification scripts to force echo-tracing of commands which helped in verification.
Testing notes
Resolved environmental issues
- {{script/benchmark}} failed for Nashorn on Windows with
{{OutOfMemoryError: Java heap space}}
at
{{;; primitive array reduce 1000000 many ops}}
Increasing RAM on my Windows VM from 4 to 6Gb resolved the issue.
- {{script/benchmark}} failed for Nashorn on Linux with
{{OutOfMemoryError: GC overhead limit exceeded}}
at
{{;;; vector equality}}
Increasing Docker->Preferences->Advanced->Memory from 2 to 6Gb resolved the issue.
Testing across OSes
Testing was done on macOS development box. Windows testing was via ssh to Windows 10 VM. Linux testing was done via Docker to ubuntu based container ({{Dockerfile}} attached).
Scripted tests and resulting logs are attached in cljs-3098...zip file. These scripts are specific to my setup but do include Dockerfile used for linux. Entry-point is {{verify.sh}}. Tested across the following variations:
- macos unpatched
- linux unpatched
- macos patched
- linux patched
- windows patched
I ran unpatched variations to compare results with patched variations.
Particular attention was paid to verifying that classpaths remained equivalent across OSes.
Note that there were some failures in linux-unpatched due to file.encoding not being utf-8 in the version of linux I used. This patch explicitly specifies utf-8 in {{cljsc}} which fixes this issue.
Pre-existing issues
Pre-existing issues found when running on macOS (out of scope for this issue):
1. {{script/browser-repl}} cljs-logo is broken in browser - note that
{{java -cp target/cljs.jar cljs.main -re browser}}
displays the cljs-logo just fine.
1. {{script/compile}} fails. Script obsolete? Left as is.
1. {{script/test-simple}} has an error that is to be fixed in CLJS-3075. Skipped in testing.
1. {{ast-ref/buildref.sh}} - unsure what this is, did not run sucessfully on macOS. Left as is.
