Welcome! Please see the About page for a little more info on how this works.

+1 vote
in IO by

The clojure.java.shell/sh function has this construct at the end:

(with-open [stdout (.getInputStream proc)
            stderr (.getErrorStream proc)]
  (let [out (future (stream-to-enc stdout out-enc))
        err (future (stream-to-string stderr))
        exit-code (.waitFor proc)]
    {:exit exit-code :out @out :err @err}))

So the outputs of a process are read in two futures, while the current thread awaits for process to end (and generate exit code). If an exception happens during an execution of a future, that exception is stored and thrown (wrapped in ExecutionException), when futures are derefed (in this case this is after process ends).

I've had a case where I was calling git with sh and the generated output was large enough to produce OutOfMemoryError. And here's where you get a "deadlock" (it's not one technically).

  • Git was shoveling data into pipe, while Java process was reading it in the future, main thread was awaiting process end
  • Future reading out encounters an OutOfMemory exception, it is stored and out reader loop terminates
  • Pipe buffer between processes becomes full, as git is writing but nobody is reading anymore
  • Git stalls indefinitely trying to write to pipe
  • Java process main thread awaits git process end indefinitely
  • The future is never derefed so no exception stacktrace or message is ever produced

This can happen with any type of exception during one of those 2 future's reads. Another realistic scenario would be if I specify an encoding and underlying process returns some bytes that cannot be parsed in that encoding.

This "bug" was quite frustrating because there is no error message and very little indication of what is wrong. In my case it happened sporadically in production and it was quite hard to track down.

I don't know what the remedy is, but it might be smart to do some sort of mitigation for this.

1 Answer

+1 vote
Currently I have wrote my own sh, where the code in the futures is wrapped in try catch and in the catch the input stream is read into a very small buffer in a loop (and buffer is never used), until it's exhausted. With OOM it might be smarter to read into an int on the stack.
One workaround for this is to use the new (in Clojure 1.12) clojure.java.process api:

    (require '[clojure.java.process :as process])
    (def result (process/exec "git" "whatever")))

If that hits OOME, you’ll get an exception like:

    Execution error at clojure.java.process/exec (process.clj:171).
    Process failed with exit=141

I believe 141 usually indicates SIGPIPE so it would at least not hang.